Scene enhancement method and device based on three-dimensional Gaussian, equipment and storage medium

Abstract

The invention provides a scene enhancement method and device based on three-dimensional Gaussian, equipment and a storage medium, and the method comprises the steps: generating an initial Gaussian scene based on point cloud data, and determining the gradient information of each region based on the Gaussian distribution information of a plurality of regions in the initial Gaussian scene; and adjusting the Gaussian distribution information based on the gradient information to obtain a reconstructed Gaussian scene of the point cloud data. According to the method and the device, the scene is divided into the plurality of areas, and the gradient information in each area is determined based on the Gaussian distribution information in each area so as to determine whether the area is an information dense area, so that the Gaussian distribution information is adjusted based on the gradient information of each area so as to refine the geometric representation of the dense area, and the accuracy of the scene is improved. And the display effect of the high-detail area is effectively improved while the computing resources are effectively saved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional Gaussian-based scene enhancement method, device, equipment and storage medium. Background Art

[0002] New view synthesis is a key task in computer vision and graphics. It aims to render new views of 3D scenes based on a set of input images. It has wide applications in many fields, such as virtual reality, augmented reality, robotics, and game production. Currently, the more common ways to achieve 3D scene reconstruction are structured light motion (Structure from Motion, SfM), multiple view stereo (Multiple View Stereo MVS) and neural radiance fields (NeRF). However, structured light motion and multiple view stereo often rely on rasterization or ray tracing rendering. In scenes with complex textures, occlusions or transparent materials, the reconstruction quality is poor, and incompleteness or artifacts often occur. Neural radiance fields perform multiple sampling through ray marching to generate high-quality view synthesis, but dense sampling brings huge computational overhead, which limits the real-time performance of neural radiance fields. Even if a variety of anti-aliasing strategies are used to reduce high-frequency components and improve rendering efficiency, there are still problems of long training time and high computational complexity.

[0003] The 3D Gaussian Splatting (3DGS) technology encodes the geometry and appearance of the scene by optimizing the parameters of the Gaussian distribution (position, covariance matrix, color, etc.), providing a more flexible way to represent the scene, effectively saving the amount of calculation while ensuring the basic reconstruction effect. However, the scenes currently reconstructed based on 3D Gaussian Splatting do not distinguish between the degree of detail or the distance. As a result, the reconstruction effect is not fine enough in some high-detail areas or close-range scenes. Summary of the invention

[0004] Based on this, it is necessary to provide a three-dimensional Gaussian-based scene enhancement method, device, equipment and storage medium to at least partially solve the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a scene enhancement method based on three-dimensional Gaussian, comprising:

[0006] Generate an initial Gaussian scene based on the point cloud data, wherein the initial Gaussian scene includes Gaussian distribution information of multiple positions;

[0007] Based on Gaussian distribution information of multiple regions in the initial Gaussian scene, determining gradient information in each of the regions;

[0008] The Gaussian distribution information is adjusted based on the gradient information to obtain a reconstructed Gaussian scene of the point cloud data.

[0009] As a feasible embodiment of the present application, the adjusting the Gaussian distribution information based on the gradient information to obtain the reconstructed Gaussian scene of the point cloud data includes:

[0010] Based on the gradient information of each position in the region, determining the number of positions in the region where the gradient information is greater than a preset gradient threshold;

[0011] If the number is greater than a preset number threshold, the Gaussian distribution information in the area is adjusted to obtain a reconstructed Gaussian scene of the point cloud data.

[0012] As a feasible embodiment of the present application, the adjusting the Gaussian distribution information in the area includes:

[0013] determining whether the region is uniformly distributed based on gradient information within the region;

[0014] In the case where the region is a non-uniformly distributed region, assigning Gaussian distribution information of multiple positions in the region to neighboring positions to obtain adjusted Gaussian distribution information; and / or

[0015] In the case that the region is a uniformly distributed region, the Gaussian distribution information in the region is adjusted based on a loss value between the Gaussian distribution information in the region and an average Gaussian distribution to obtain adjusted Gaussian distribution information.

[0016] As a feasible embodiment of the present application, the generating of the initial Gaussian scene based on the point cloud data includes:

[0017] Input the point cloud data into the trained target model to determine multiple locations and determine the Gaussian distribution information of each location to generate an initial Gaussian scene;

[0018] The target model is trained through the following steps:

[0019] Acquire sample point cloud data and data labels of the sample point cloud data, wherein the data labels include label positions and label normal vectors of the label positions;

[0020] Inputting the sample point cloud data into an initial model to obtain a predicted Gaussian scene of the sample point cloud data, wherein the predicted Gaussian scene includes Gaussian distribution information of a plurality of predicted positions;

[0021] Determine a model loss value of the initial model based on a first loss value between the predicted position and the label position, and a second loss value between the direction vector in the Gaussian distribution information and the label normal vector;

[0022] The initial model is trained according to the model loss value until a target model is obtained.

[0023] As a feasible embodiment of the present application, the method further includes:

[0024] Obtaining Gaussian distribution information of multiple reference positions within the grid where the predicted position is located;

[0025] determining a third loss value based on a difference between the Gaussian distribution information of the predicted position and Gaussian distribution information of a plurality of reference positions;

[0026] The determining of the model loss value of the initial model based on the first loss value between the predicted position and the label position, and the second loss value between the direction vector in the Gaussian distribution information and the label normal vector, comprises:

[0027] Based on the first loss value between the predicted position and the label position, the second loss value between the direction vector in the Gaussian distribution information and the label normal vector, and the third loss value, the model loss value of the initial model is determined.

[0028] As a feasible embodiment of the present application, the determining of the model loss value of the initial model based on the first loss value between the predicted position and the label position, the second loss value between the direction vector in the Gaussian distribution information and the label normal vector, and the third loss value includes:

[0029] Determining weights of multiple loss values ​​based on the number of training times of the initial model;

[0030] The first loss value, the second loss value and the third loss value are weighted according to the weight to obtain a model loss value of the initial model.

[0031] As a feasible embodiment of the present application, the method further includes:

[0032] extracting scene feature points from a plurality of scene images of a target scene;

[0033] The point cloud data of the target scene is determined based on the overlapping information of the scene feature points, wherein the point cloud density of the point cloud data is lower than a preset density threshold.

[0034] In a second aspect, the present application provides a scene enhancement device based on three-dimensional Gaussian, comprising:

[0035] A generating module, configured to generate an initial Gaussian scene based on the point cloud data, wherein the initial Gaussian scene includes Gaussian distribution information of multiple locations;

[0036] A determination module, configured to determine the gradient information in each of the regions based on the Gaussian distribution information of the multiple regions in the initial Gaussian scene;

[0037] A reconstruction module is used to adjust the Gaussian distribution information based on the gradient information to obtain a reconstructed Gaussian scene of the point cloud data.

[0038] In a third aspect, the present application further provides a computer device, the computer device comprising:

[0039] one or more processors;

[0040] Memory; and

[0041] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the three-dimensional Gaussian-based scene enhancement method provided above.

[0042] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, and the computer program is loaded by a processor to execute the above-mentioned three-dimensional Gaussian-based scene enhancement method.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned three-dimensional Gaussian-based scene enhancement generation method.

[0044] The three-dimensional Gaussian-based scene enhancement method provided in the present application, after relying on point cloud data to generate an initial Gaussian scene composed of Gaussian distribution information of multiple positions, divides the scene into multiple areas, and determines the gradient information in each area based on the Gaussian distribution information in each area to determine whether the area is an information-intensive area, thereby adjusting the Gaussian distribution information based on the gradient information of each area to refine the geometric representation of the dense area, effectively saving computing resources while effectively improving the display effect of the high-detail area. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 A schematic diagram of the steps of a three-dimensional Gaussian scene enhancement method provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a process flow for adjusting Gaussian distribution information based on gradient information provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a flow chart of steps for adjusting Gaussian distribution information based on uniform distribution of regions provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a step flow of training a target model provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of another step flow for calculating the model loss value provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the structure of a three-dimensional Gaussian scene enhancement provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0054] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0055] In the description of the present application, the word "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0056] In order to clearly understand the method provided in the embodiment of the present application, the following first specifically describes the relevant implementation background of the three-dimensional Gaussian-based scene enhancement method provided in the embodiment of the present application. The solution provided in this application mainly involves three-dimensional scene rendering in various fields such as virtual reality and augmented reality, and aims to generate a view of a three-dimensional scene by rendering a set of input images. Among them, the traditional implementation of three-dimensional scene reconstruction, such as structured light motion, multi-view stereo dependent rasterization or ray tracing rendering, has poor reconstruction quality in some scenes with complex textures, occlusions or transparent materials, and often appears incomplete or artifacts. With the introduction of neural network algorithms, neural networks have gradually been applied in such scenarios to reduce the artifacts generated by rendering. Among them, the neural radiation field uses a multilayer perceptron (MLP) to implicitly encode the three-dimensional scene and performs multiple sampling through ray marching to generate high-quality perspective synthesis. However, the method of volume rendering by querying dense points along the rays in the above process is computationally intensive, making it difficult to apply in real-time environments. The rendering speed is usually less than 1fps (Frame Per Second). Even if anti-aliasing strategies such as Mip-NeRF, Tri-MipNeRF and Zip-NeRF (several improved neural radiation field technologies) are used to reduce high-frequency components through multi-scale position encoding and voxel grids, thereby improving rendering efficiency, there are still problems of long training time and relatively high computational complexity.

[0057] On the basis of the several 3D scene reconstruction technologies provided above, a point-based rendering method, namely 3D Gaussian sputtering technology, was proposed. This method encodes the geometry and appearance of the scene by optimizing the parameters of the Gaussian distribution (position, covariance matrix, color, etc.), providing a more flexible scene reconstruction method. Since the 3D Gaussian sputtering technology uses the displayed Gaussian distribution to represent the scene, compared with the implicit encoding of the 3D scene in the neural radiation field, these explicit features are more suitable for real-time rendering, and high-quality real-time new view synthesis can be achieved at a resolution of 1080p, with a frame rate of more than 90fps.

[0058] Specifically, 3DGS starts with a set of sparse point clouds, and creates a set of 3D Gaussians consisting of positions, covariance matrices, and opacity α as primitives for scene representation through these point clouds / and the normal vector model formed by the point clouds. They inherit the characteristics of differentiable volume representation and are structureless and explicit, allowing fast rendering. Rendering is achieved by projecting 3D Gaussians into 2D, a process that involves converting the 3D covariance matrix into a 2D covariance matrix in the camera coordinate system. Optimization is performed by iterative rendering and comparing the resulting image with the training views in the captured dataset. Stochastic gradient descent technology is used for optimization, making full use of the standard graphics processor (Compute Unified Device Architecture, GPU) acceleration framework, and adding customized unified computing device architecture (Compute Unified Device Architecture, CUDA) kernels to improve efficiency. During the optimization process, the density of Gaussian points is controlled by adding or removing them to better represent the scene. After the warm-up phase of the optimization, the density of Gaussian points is increased every certain number of iterations, and those Gaussian points that are almost transparent are removed. In addition, a tile-based rasterizer is designed for Gaussian drawing, which allows alpha blending at different positions and can be quickly sorted, thereby accelerating training and real-time rendering. Once the optimization is completed, the optimized 3D Gaussian and camera pose can be used to render new views or perform 3D reconstruction.

[0059] In order to make resource utilization more efficient, Adaptive Density Control (ADC) is usually used in the process of optimizing Gaussian density. It introduces a transparency-based deletion mechanism to treat Gaussians that are closer to the camera and have higher transparency as redundant and delete them. By deleting these Gaussians with lower transparency, foreground interference can be reduced, memory utilization can be optimized, and rendering clarity can be improved without losing key visual information. Although the above adjustment can optimize the distribution of Gaussians, it uses consistent treatment for all areas, ignoring the spatial differences in detail requirements. The original ADC mechanism fails to have the spatial awareness to recognize local complexity in the scene, which may lead to uneven Gaussian density, resulting in visual artifacts or reduced computational efficiency in some areas.

[0060] In order to solve the above technical problems, this application proposes an improved adaptive density control in the process of 3D scene reconstruction based on 3D Gaussian sputtering technology, which aims to introduce regional segmentation and selective Gaussian adjustment for different gradients in the region, which can adapt to different levels of complexity in the scene, ensure that the area with rich high-frequency details has appropriate Gaussian density, and the area with less details maintains a lower density. And finally maintain high-resolution rendering with balanced resource usage.

[0061] Specifically, the following content will explain the above scheme in detail, please refer to Figure 1 , Figure 1 A schematic flow chart of a three-dimensional Gaussian scene enhancement method provided in an embodiment of the present application, specifically, comprising steps S110 to S140:

[0062] S110, generating an initial Gaussian scene based on the point cloud data.

[0063] In the embodiment of the present application, point cloud data can be understood as a set of vectors in a three-dimensional coordinate system, recorded in the form of points, each point contains three-dimensional coordinates, and can further contain attribute information such as color information or reflection intensity information. Among them, the source of point cloud data can be obtained by scanning multiple scenes, of course, it can also be obtained by processing multiple images. The embodiment of the present application does not limit the method of obtaining point cloud data. All point cloud data that can be used to describe the scene are within the scope of protection required by this application. However, in order to more clearly understand the method of obtaining point cloud data provided in the embodiment of the present application, it will be described in detail in the subsequent further embodiments.

[0064] On the basis of the point cloud data obtained above, relying on the three-dimensional Gaussian injection technology provided above, the point cloud data can be converted into an initial Gaussian scene, wherein the initial Gaussian scene is usually represented by multiple positions (which can be understood as the center position of the Gaussian distribution), that is, the Gaussian distribution information of multiple Gaussian points, such as covariance matrix and opacity. To simplify the description, the scene described in the above form will be referred to as a Gaussian scene. For example, in a common implementation scheme, several target positions in the scene can be randomly initialized based on the point cloud data, and the Gaussian distribution information of each position can be initialized. Then, during the optimization process, the density of Gaussian points is continuously controlled by adding or removing Gaussian points, so as to finally generate an initial Gaussian scene that accurately represents the scene through a set of accurate and appropriate Gaussian distribution information. Of course, it is also feasible to implement a scheme based on other improved three-dimensional Gaussian injection technologies to reasonably generate Gaussian scenes through point cloud data, and the embodiments of the present application are not limited here.

[0065] Furthermore, considering that in the aforementioned initial Gaussian generation scenario, different randomly initialized target positions, that is, Gaussian positions are one of the key steps to achieve high-quality three-dimensional reconstruction, this process is not only related to the accuracy of reconstruction, but also directly affects the efficiency and accuracy of subsequent processing. Therefore, as a feasible implementation scheme of the present application, an enhanced Gaussian initialization process is also provided, which no longer relies on randomly generated parameters. In this way, a more accurate starting point can be provided for Gaussian initialization, thereby significantly improving the convergence speed and the final reconstruction accuracy. The specific implementation scheme will be described in detail in the subsequent embodiments.

[0066] S120, determining gradient information in each of the regions based on Gaussian distribution information of the multiple regions in the initial Gaussian scene.

[0067] Different from the adaptive density control provided by the related art to optimize Gaussian density, an improved and enhanced adaptive density control is proposed in the embodiment of the present application. Specifically, the solution provided in the embodiment of the present application first divides the initial Gaussian scene into multiple regions according to certain division rules, and then uses the Gaussian distribution information in each region to determine the gradient information in each region.

[0068] Specifically, there may be many ways to divide the initial Gaussian scene into multiple regions. For example, the initial Gaussian scene may be divided according to a preset size or ratio. Of course, as another optional implementation scheme of the present application, the initial Gaussian scene may be image recognized to identify the subject information and background area therein, thereby dividing the initial Gaussian scene based on the subject and background in the scene, thereby further improving the effect of dividing the multiple regions in the initial Gaussian scene. The embodiment of the present application does not limit the division method of multiple regions in the initial Gaussian scene.

[0069] On the basis of the multiple regions obtained above, the gradient information in the region can be obtained by individually evaluating the Gaussian distribution of each region. For example, the gradient information of the region can be described by using high gradient or low gradient, etc. Of course, the number of Gaussians with high gradient in the region can be further used as the gradient information of the region. The embodiment of the present application does not limit the specific meaning of the gradient information of the region. Any method that can describe the overall gradient distribution in the region to a certain extent can be understood as the gradient information in the present application. For example, in a feasible implementation scheme, the set Gk of Gaussians with high gradient in the region can be used as the gradient information of the region, as follows:

[0070]

[0071] in, is the area divided in the initial Gaussian scene, x is the Gaussian position in the area, is the gradient corresponding to the position, which can usually be obtained based on the Gaussian distribution information between the position and the remaining neighboring positions, for example, it can be obtained by Gaussian gradient operator operation, δ g is a preset gradient threshold, that is, the Gaussian positions in the region whose gradient information exceeds the preset gradient threshold can be added to the set Gk, and the region is represented by the set Gk. The gradient information.

[0072] By performing similar processing on each region, the gradient information of each region in the initial Gaussian scene can be obtained, and described by a Gaussian set with high gradients in each region. The above gradient information will be used as differentiation information to implement different strategies for adjusting the Gaussian distribution information of different regions, thereby completing the reconstruction optimization of the initial Gaussian scene. For details, please refer to the relevant description of the subsequent step S130.

[0073] S130, adjusting the Gaussian distribution information based on the gradient information to obtain a reconstructed Gaussian scene of the point cloud data.

[0074] In an embodiment of the present application, based on the aforementioned division based on different regions and individual evaluation of the Gaussian distribution of different regions to determine the gradient information of each region, the corresponding Gaussian distribution information in each region is adjusted based on the strategy corresponding to the different gradient information. It is possible to give priority to processing gradient-dense regions, thereby achieving a finer Gaussian distribution and making the fidelity of detail regions higher, while avoiding redundant Gaussian arrangements in simple regions, thereby effectively improving the effect of the reconstructed Gaussian scene.

[0075] Specifically, for ease of understanding, the region is represented by the set Gk provided above. For example, in a feasible implementation, the number of locations where the gradient information in the region is greater than a preset gradient threshold can be used to determine which region's Gaussian distribution information needs to be adjusted. For details, see Figure 2 , Figure 2 A schematic flow chart of a step of adjusting Gaussian distribution information based on gradient information provided in an embodiment of the present application, specifically, comprising steps S210 to S220:

[0076] S210: Determine the number of positions in the region whose gradient information is greater than a preset gradient threshold based on the gradient information of each position in the region.

[0077] In the embodiment of the present application, combined with the foregoing relevant description, it can be seen that since the Gaussians whose gradients in the region are higher than a certain gradient threshold are added to the set Gk, that is, the number of elements (Gaussians) in the set Gk can be used to describe whether the number of Gaussians with high gradients in a certain region is significant, that is, whether the region is a significant high-gradient region, and thus more details are required to support the scene representation of the region. Therefore, the number of elements in the set Gk can be counted to determine the number of positions in the region where the gradient information is greater than the preset gradient threshold, so as to subsequently determine whether the Gaussian distribution information of the region needs to be adjusted.

[0078] S220: If the number is greater than a preset number threshold, the Gaussian distribution information in the area is adjusted to obtain a reconstructed Gaussian scene of the point cloud data.

[0079] In the embodiment of the present application, specifically, if the number is greater than a certain number threshold, the region is a significant high-gradient region, and more details are needed to support the scene representation of the region. Therefore, the Gaussian distribution information in the region can be adjusted to achieve detailed representation of the region. Conversely, for regions with low gradient significance, conventional adjustment strategies can be considered, such as a transparency-based deletion mechanism to remove Gaussians with low transparency in these regions, so as to reduce foreground interference and optimize memory utilization while not losing key visual information as much as possible, thereby improving the rendering representation effect of high-gradient regions.

[0080] Specifically, there are many implementation schemes for adjusting the Gaussian distribution information in the high gradient area. For example, the Gaussian distribution information of multiple positions in the high gradient area can be directly copied, or other strategies can be used to increase the Gaussian density in the area. In order to further improve the performance of supporting the details of the corresponding area in the scene through the Gaussian density in the area, as a further optional implementation scheme of the present application, it is also possible to determine whether the Gaussian distribution in the area is uniformly distributed based on the gradient information in the area, so as to use different adjustment strategies to adjust the Gaussian distribution information in the area. For details, please refer to Figure 3 , Figure 3 A schematic flow chart of the steps for adjusting Gaussian distribution information based on uniform distribution of regions provided in an embodiment of the present application, specifically, including steps S310 to S330:

[0081] S310: Determine whether the region is evenly distributed based on gradient information within the region.

[0082] In the embodiment of the present application, the standard deviation of the high-gradient Gaussian in the region can be used to determine whether the region is uniformly distributed. For example, as an implementation of the present application, the standard deviation σ k The calculation formula is as follows:

[0083]

[0084] in, is the average value of the Gaussian with high gradient in the region, that is:

[0085]

[0086] That is, through the above formula, the standard deviation σ of the Gaussian with high gradient in the region is calculated by using the Gaussian with high gradient in the region and its corresponding average value. k Then, based on the standard deviation σ k The relationship between the standard deviation threshold and the area distribution can be determined. For example, kIf the standard deviation is greater than the preset threshold value δu, the area can be considered as a non-uniform distribution area, and if the standard deviation is σ k When the value is less than or equal to the preset standard deviation threshold δu, the area can be considered as a uniformly distributed area.

[0087] S320, when the region is a non-uniformly distributed region, assign Gaussian distribution information of multiple positions in the region to neighboring positions to obtain adjusted Gaussian distribution information.

[0088] In the embodiment of the present application, when the region is a non-uniformly distributed region, the current Gaussian can be directly copied to cover all necessary points in the region, that is, the Gaussian distribution information of multiple positions in the region, such as the Gaussian position with high gradient mentioned above, is assigned to the neighboring position, where the position of the new Gaussian x′ is the position after adding a small disturbance ∈ to the original Gaussian position, that is:

[0089] x′=x+∈

[0090] Here, ∈ may be a randomly generated disturbance, and its disturbance value may specifically obey a Gaussian distribution with a mean of 0.

[0091] S330, when the region is a uniformly distributed region, the Gaussian distribution information in the region is adjusted based on a loss value between the Gaussian distribution information in the region and an average Gaussian distribution to obtain adjusted Gaussian distribution information.

[0092] In the case where the area is a uniformly distributed area, in order to avoid the conventional replication strategy causing clustering in the uniformly distributed area and thus causing an imbalance in the distribution, in an embodiment of the present application, another strategy for adjusting the Gaussian distribution information is proposed for the uniformly distributed area, which is detailed as follows.

[0093] In the embodiment of the present application, by introducing the dispersion term loss function To encourage a more uniform distribution, the dispersion loss function The calculation formula is as follows:

[0094]

[0095] That is, the dispersion loss function It is associated with the standard deviation of the Gaussian with high gradient in the region, and on this basis, the dispersion loss function is The reconstruction loss used to adjust the Gaussian distribution information in the conventional 3D Gaussian injection technique By performing weighted summation, we can get the final composite loss value Specifically, the composite loss value The calculation formula is as follows:

[0096]

[0097] Where N represents the N regions obtained by dividing the three-dimensional Gaussian scene. That is the kth region. Reconstruction loss It can be understood as the difference between the Gaussian distribution information in the region before adjustment and the Gaussian distribution information in the region after adjustment. The specific method can be implemented in a variety of ways, such as based on distance or similarity. In this embodiment of the present application, the reconstruction loss There is no restriction on the calculation process. Dispersion loss function The meaning of λ refers to the above description, and will not be repeated in this embodiment. k It can be understood as the weight coefficient of the scattered term loss function corresponding to the area. Specifically, λ k The value is usually between 0 and 1, which can be a preset value, but in order to better achieve the uniform adjustment of the Gaussian distribution information of the uniform distribution area, its value can usually be related to the uniform distribution of the area. The higher the uniform distribution of the area, the higher the value of k The value of can also be set higher.

[0098] Through the above-mentioned scheme, the reconstruction loss can avoid the large difference between the Gaussian distribution information after adjustment and the Gaussian distribution information before adjustment. At the same time, the introduced dispersion loss function It also effectively avoids the clustering phenomenon of the adjusted Gaussian distribution information and causes distribution imbalance, thereby effectively improving the reconstruction effect in the uniform area. The replication of Gaussians in the non-uniform area can simply and effectively increase the density without reinitializing all Gaussians. While saving memory and computing resources, it ensures that the non-uniform area with rich high-frequency details also has appropriate Gaussian density, thereby maintaining high-resolution rendering while balancing resource usage.

[0099] Through the above scheme, the improved adaptive density control significantly improves the visual details and computational efficiency of 3D Gaussian jets through a mathematically driven region-sensitive control strategy. Through region segmentation and gradient-sensitive adjustment, Gaussians are selectively added to achieve a fine balance between image quality and computational efficiency, so that the improved adaptive density can still show strong adaptability in high-fidelity rendering of complex scenes.

[0100] The three-dimensional Gaussian-based scene enhancement generation method provided in the present application, after relying on point cloud data to generate an initial Gaussian scene composed of Gaussian distribution information of multiple positions, divides the scene into multiple areas, and determines the gradient information in each area based on the Gaussian distribution information in each area to determine whether the area is an information-intensive area, thereby adjusting the Gaussian distribution information based on the gradient information of each area to refine the geometric representation of the dense area, effectively saving computing resources while effectively improving the display effect of the high-detail area.

[0101] Furthermore, combined with the above-mentioned related explanations, it can be known that the relevant technology is to reconstruct the three-dimensional Gaussian scene by randomly initializing Gaussians and continuously iteratively optimizing (such as adding, updating, and deleting) Gaussians. In the actual process, it is found that accurately initializing the Gaussian representation is one of the key steps to achieve high-quality three-dimensional reconstruction. This process is not only related to the accuracy of reconstruction, but also directly affects the efficiency and accuracy of subsequent processing. If a randomly initialized Gaussian is used, the convergence speed and the final reconstruction accuracy often cannot achieve the ideal effect.

[0102] In order to further improve the reconstruction effect of the three-dimensional Gaussian scene, based on the solution provided in the embodiment of the present application, the present application further provides an implementation solution for fitting and generating better initialization Gaussian parameters based on the model. It is particularly noted that the solution provided in the embodiment of the present application is particularly suitable for COLMAP data (an open source software that supports sparse reconstruction and dense reconstruction). The key components contained in the COLMAP data set, such as calibrated camera parameters and sparse point clouds, are of great significance for understanding the spatial relationship between various elements in the scene. Using these structured data, the errors caused by random initialization can be further effectively reduced. Therefore, in the subsequent embodiments, COLMAP data will be used as an example for explanation, but it should not be understood as a limitation to the embodiments of the present application.

[0103] Specifically, in order to initialize the Gaussian parameters, an initial Gaussian scene is generated based on the point cloud data, which includes:

[0104] The point cloud data is input into the trained target model to determine multiple locations and determine the Gaussian distribution information of each location to generate an initial Gaussian scene.

[0105] That is to say, in the solution provided in the embodiment of the present application, by inputting the point cloud data into the trained model, the model can recover a number of suitable initialization Gaussians based on the solution of the point cloud data, that is, the positions of multiple initialization Gaussians and the Gaussian distribution information initialized at each position, thereby further generating an initial Gaussian scene.

[0106] It is understandable that the training method of the target model will affect the reconstruction effect of the final output of the initialization Gaussian. Therefore, in order to more accurately initialize the Gaussian representation, as an implementation scheme of this application, a solution for training the target model based on position and normal vector is also proposed. For details, please refer to Figure 4 , Figure 4 A schematic flow chart of a training target model provided in an embodiment of the present application, specifically, including steps S410 to S440:

[0107] S410, obtaining sample point cloud data and data labels of the sample point cloud data.

[0108] In the embodiment of the present application, taking COLMAP data as an example, the sample point cloud data can generally be a three-dimensional point set extracted from multiple overlapping images contained in the COLMAP data. Usually, the sample point cloud data can be sparse point cloud data. The embodiment of the present application again does not limit the acquisition of the sample point cloud data. In addition to the sample point cloud data, the COLMAP data generally also includes other key components, such as calibrated camera parameters for understanding the spatial relationship between various elements in the scene. Therefore, during the training process, the above data can also be used to input into the initial model to train the model.

[0109] Specifically, the camera parameters include camera intrinsic parameters and camera extrinsic parameters, where the camera intrinsic parameters include focal length, principal point coordinates (c x , c y ) and parameters such as the slope s, which together define how a 3D point is projected into a 2D image. Accurate camera intrinsic parameters are crucial for correctly reconstructing the scene because they directly determine the relationship between each point in 3D space and the pixel position in the image. Specifically, the intrinsic parameter matrix K can be formalized as:

[0110]

[0111] Such a matrix structure is the basis for realizing the projection of three-dimensional points onto two-dimensional images.

[0112] Secondly, the camera extrinsic parameters also play a vital role. The extrinsic parameters consist of the rotation matrix R and the displacement vector T. These parameters define the position and orientation of the camera in the world coordinate system. The three-dimensional point P can be transformed into the camera coordinate system through the following formula:

[0113] P camera =R·P+T

[0114] Among them, P camera Represents a 3D point in the camera coordinate system.

[0115] Of course, in order to realize the training of the model, the sample point cloud data can usually be pre-labeled with data labels through manual labeling or other methods. The specific data labels usually include the label position and the label normal vector of the label position, where the label position can be understood as the Gaussian position formed by the sample point cloud data that is more suitable as the initialization Gaussian, and the label normal vector can be understood as the normal vector at the label position.

[0116] S420: Input the sample point cloud data into an initial model to obtain a predicted Gaussian scene of the sample point cloud data.

[0117] In the embodiment of the present application, the specific structure of the initial model can generally adopt a common neural network model structure, such as a deep model or a convolutional model, etc. Of course, in order to better process the structured data including point clouds and camera parameters provided by the aforementioned COLMAP to better and efficiently initialize Gaussian, in the embodiment of the present application, a multi-layer perceptron network architecture can be specifically selected, wherein the architecture is composed of multiple layers, and each layer plays an important role in converting input data into Gaussian parameters.

[0118] Specifically, at the input layer, the network receives various features from the COLMAP data. These features are the 3D point coordinates in the sparse point cloud. Specifically, the input can be expressed as:

[0119]

[0120] The interpretation of various parameters in the input can refer to the above related descriptions, where M is the input point cloud data.

[0121] The hidden layer consists of one or more fully connected layers that use nonlinear activation functions (such as ReLU) to capture the complex relationship between input features. In a hidden layer with several neurons, the output can be defined as:

[0122] H = ReLU(W · X + b)

[0123] In this formula, W represents the weight matrix to be trained, X is the input vector, and b is the bias vector to be trained.

[0124] The output layer is responsible for generating the required Gaussian parameters, including the xyz coordinates μ of each Gaussian i , covariance matrix∑ i and the transparency value α i (opacity), that is, these outputs can be expressed as:

[0125]

[0126] Here, the covariance matrix is ​​used to define the shape and spread of the Gaussian, usually taking a diagonal matrix to indicate isotropic behavior.

[0127] In general, the sample point cloud data is input into a model with trained parameters to output accurate Gaussian parameters for reconstructing three-dimensional Gaussian scenes. However, in general, the parameters in the model need to be trained. Therefore, the sample point cloud data is input into the initial model with initialized parameters. The output predicted Gaussian parameters and the ideal Gaussian parameters, that is, the aforementioned label data, often differ. By iteratively optimizing the parameters in the model based on the difference, the model can be updated until the final target model is obtained.

[0128] It can be understood that, combined with the above description, the moral prediction Gaussian scenario input into the initial model includes Gaussian distribution information of multiple prediction positions, where the prediction position will usually be different from the label data.

[0129] S430, determining a model loss value of the initial model based on a first loss value between the predicted position and the label position, and a second loss value between the direction vector in the Gaussian distribution information and the label normal vector.

[0130] In order to better optimize the model, in the embodiment of the present application, in addition to considering the loss value between the predicted position and the label position, the Gaussian normal vector is also considered to encourage the alignment of the Gaussian and the surface normal vector. That is to say, in one embodiment, the calculation formula of the model loss value Total Loss of the initial model is as follows:

[0131] Total Loss=Distance Loss+λ·Orientation oss

[0132] Among them, Distance Loss is the distance loss, which quantifies the deviation between the predicted Gaussian position and its target position, that is, the label position. Its specific calculation formula is as follows:

[0133]

[0134] Where N is the predicted number of Gaussians, is the i-th predicted Gaussian position, is the corresponding label position. λ is a hyperparameter that controls the relative importance of the direction loss in the optimization process.

[0135] The Orientation Loss is a directional loss that encourages the alignment of the Gaussian with the surface normal vector. It is described by the angle difference between the direction vector in the Gaussian distribution information and the label normal vector. The calculation formula of the Orientation Loss is as follows:

[0136]

[0137] Among them, θ i Calculated by the following formula:

[0138]

[0139] Among them, U is the direction vector in the Gaussian distribution information, that is, the predicted direction vector, which can usually be obtained by extracting the normal vector of the grid point where the corresponding predicted Gaussian position is located, and the Gaussian can be aligned with the geometric direction of the surface, and U i is the corresponding label normal vector.

[0140] S440, training the initial model according to the model loss value until a target model is obtained.

[0141] After determining the model loss value through the method provided above, since the model loss value fully considers the deviation of position and direction, after training the initial model based on the model loss value, the final target model can fully guarantee the readiness of the predicted position and direction of the initial Gaussian, thereby improving the effect of subsequent three-dimensional Gaussian scene reconstruction.

[0142] In order to monitor the convergence of the optimization process, a threshold can usually be defined based on the total loss so that training can be stopped when the total loss is lower than a preset small threshold, and the model obtained by the current training can be used as the target model.

[0143] Furthermore, the training of the initial model can be based on the idea of ​​back propagation to optimize the parameters. Specifically, for the MLP model, the gradient of the loss function relative to the weight can be calculated by using the chain rule, where a certain learning rate is used to determine the speed at which each Gaussian moves toward the target position. In addition, after the training is completed, the generalization ability of the model can be ensured by using a validation set different from the training data. Performance evaluation can use indicators such as mean square error (MSE). In addition, training is stopped when the validation loss fails to improve within a specified round to prevent overfitting.

[0144] It is understandable that the above description of training the initial model is only a feasible implementation scheme and should not be understood as a limitation on the embodiments of the present application. In fact, training the model through other model structures or methods to utilize position and direction deviations should be within the scope of protection required by this application.

[0145] Of course, in addition to the aforementioned position and direction-based deviations to achieve the measurement of model loss, as a further feasible implementation of this application, in order to ensure that the Gaussian generated by initialization can accurately reflect the surface characteristics of objects in the scene, the surface fitting loss can also be considered in the total loss value. For details, please refer to Figure 5 , Figure 5 A schematic flow chart of another step of calculating the model loss value provided in an embodiment of the present application, specifically, comprising steps S510 to S530:

[0146] S510, obtaining Gaussian distribution information of multiple reference positions in the grid where the predicted position is located.

[0147] In order to realize the calculation of surface fitting loss, in an embodiment of the present application, other reference positions within the grid where the predicted Gaussian position is located and whose distance from the predicted Gaussian position is less than a certain distance are further obtained, that is, the neighboring grid points surrounding the Gaussian position, which can be specifically represented by the set N(i).

[0148] S520: Determine a third loss value based on the difference between the Gaussian distribution information of the predicted position and the Gaussian distribution information of multiple reference positions.

[0149] In the embodiment of the present application, based on the above, the calculation formula of the third loss value, that is, the surface fitting loss SurfaceFitting Loss, is as follows:

[0150]

[0151] Among them, the parameters in the above formula have been specifically explained above and will not be repeated here in the embodiments of the present application.

[0152] S530, determining the model loss value of the initial model based on the first loss value between the predicted position and the label position, the second loss value between the direction vector in the Gaussian distribution information and the label normal vector, and the third loss value.

[0153] In an embodiment of the present application, on the basis of the above, the first loss value (distance loss), the second loss value (direction loss value) and the third loss value (surface fitting loss) mentioned above are combined as the model loss value. During the model training process, the trained model can predict the position and direction that are more accurate and close to the surface of the object, and the initialization Gaussian that can better reflect the surface characteristics of the mesh, thereby further improving the effect of subsequent three-dimensional Gaussian scene reconstruction.

[0154] Specifically, the model loss value is determined based on the three loss values ​​mentioned above. Usually, the model loss value can be obtained by adding the three loss values. Of course, the model loss value can also be obtained by weighted addition of the three loss values ​​according to a certain weight coefficient, for example:

[0155] Total Loss=Distance Loss+λ·Orientation Loss+γ·Surface FittingLoss

[0156] Wherein, λ and γ are hyperparameters for controlling the relative importance of each loss component in the optimization process. In a feasible embodiment, λ and γ can be set based on the number of training times of the initial model, that is, based on the first loss value between the predicted position and the label position, the second loss value between the direction vector in the Gaussian distribution information and the label normal vector, and the third loss value, the model loss value of the initial model is determined, including:

[0157] Determining weights of multiple loss values ​​based on the number of training times of the initial model;

[0158] The first loss value, the second loss value and the third loss value are weighted according to the weight to obtain a model loss value of the initial model.

[0159] Through the solution provided in the embodiment of the present application, the initialization process can improve accuracy by training the MLP to minimize the error between the predicted value and the convergence center. This enhanced accuracy is crucial for the subsequent Gaussian optimization stage and lays a solid foundation for further optimizing the Gaussian representation.

[0160] In order to clearly understand the implementation effect of the three-dimensional Gaussian scene enhancement method provided in the embodiment of the present application, the following evaluation indicators are used to measure the quality of the experimental results:

[0161] PSNR (Peak Signal-to-Noise Ratio): Measures the quality of an image, with higher values ​​indicating better quality.

[0162] SSIM (Structural Similarity Index): Used to evaluate the perceptual quality of an image. The closer it is to 1, the better the quality.

[0163] LPIPS (Perceptual Patch Similarity): A perceptual metric that evaluates the visual similarity of images, with lower values ​​indicating better quality.

[0164] The method provided by this application can better control the distribution of Gaussian points by dynamically adjusting the resolution of each area, thereby improving the detailed representation of the scene and avoiding wasting computing resources in low-complexity areas. Specific experimental results show that the improved adaptive density control method has significant advantages in improving image quality (PSNR and SSIM) and reducing visual artifacts (LPIPS), especially when processing complex scenes, effectively improving the accuracy of scene representation while maintaining high rendering efficiency.

[0165] The following will describe the scene reconstruction effect improved by the method provided by the present application in combination with specific ablation experimental data, as follows:

[0166] (1) When the improved adaptive density control strategy is cancelled (the corresponding functional module is removed), the system's detail performance in high-complexity areas is significantly reduced, and computing resources are wasted in low-complexity areas. After adding this module, the system can intelligently judge the needs of different areas, significantly improving rendering efficiency and quality.

[0167] (2) In the experiment without grid initialization, the initialization effect of sparse point cloud is poor, resulting in slow convergence speed and reduced rendering quality during training. After using grid initialization, the training speed and rendering quality of the model are improved.

[0168] (3) As for the vertical alignment between the surface normal vector of the object and the plane composed of multiple Gaussian functions, by removing this alignment strategy, the rendering quality is reduced in scenes with complex geometric structures, indicating that this alignment process is crucial to improving detail accuracy.

[0169] On the basis of the above-mentioned three-dimensional Gaussian scene enhancement method, the present application further provides a three-dimensional Gaussian scene enhancement device. For details, please refer to Figure 6 , Figure 6 A schematic diagram of a three-dimensional Gaussian scene enhancement structure provided in an embodiment of the present application specifically includes:

[0170] A generating module 610, configured to generate an initial Gaussian scene based on the point cloud data, wherein the initial Gaussian scene includes Gaussian distribution information of multiple locations;

[0171] A determination module 620, configured to determine gradient information in each of the regions based on Gaussian distribution information of the multiple regions in the initial Gaussian scene;

[0172] The reconstruction module 630 is used to adjust the Gaussian distribution information based on the gradient information to obtain a reconstructed Gaussian scene of the point cloud data.

[0173] In some embodiments of the present application, the reconstruction module 630 is also used to determine the number of positions in the region whose gradient information is greater than a preset gradient threshold based on the gradient information of each position in the region; if the number is greater than the preset number threshold, the Gaussian distribution information in the region is adjusted to obtain a reconstructed Gaussian scene of the point cloud data.

[0174] In some embodiments of the present application, the reconstruction module 630 is also used to determine whether the region is uniformly distributed based on the gradient information in the region; when the region is a non-uniformly distributed region, the Gaussian distribution information of multiple positions in the region is assigned to neighboring positions to obtain adjusted Gaussian distribution information; and / or; when the region is a uniformly distributed region, the Gaussian distribution information in the region is adjusted based on the loss value between the Gaussian distribution information in the region and the average Gaussian distribution to obtain adjusted Gaussian distribution information.

[0175] In some embodiments of the present application, the generation module 610 is used to input the point cloud data into a trained target model to determine multiple positions and determine the Gaussian distribution information of each position to generate an initial Gaussian scene, wherein the target model is trained by the generation module 610 through the following steps: obtaining sample point cloud data and data labels of the sample point cloud data, wherein the data labels include label positions and label normal vectors of the label positions; inputting the sample point cloud data into the initial model to obtain a predicted Gaussian scene of the sample point cloud data, wherein the predicted Gaussian scene includes Gaussian distribution information of multiple predicted positions; determining a model loss value of the initial model based on a first loss value between the predicted position and the label position and a second loss value between the direction vector in the Gaussian distribution information and the label normal vector; and training the initial model according to the model loss value until the target model is obtained.

[0176] In some embodiments of the present application, the generation module 610 is also used to obtain Gaussian distribution information of multiple reference positions within the grid where the predicted position is located; determine a third loss value based on the difference between the Gaussian distribution information of the predicted position and the Gaussian distribution information of multiple reference positions; determine the model loss value of the initial model based on the first loss value between the predicted position and the label position, the second loss value between the direction vector in the Gaussian distribution information and the label normal vector, and the third loss value.

[0177] In some embodiments of the present application, the generation module 610 is also used to determine the weights of multiple loss values ​​based on the number of training times of the initial model; the first loss value, the second loss value and the third loss value are weighted according to the weights to obtain the model loss value of the initial model.

[0178] In some embodiments of the present application, the generation module 610 is also used to extract scene feature points from multiple scene images of the target scene; determine the point cloud data of the target scene based on the overlapping information of the scene feature points, wherein the point cloud density of the point cloud data is lower than a preset density threshold.

[0179] After relying on point cloud data to generate an initial Gaussian scene composed of Gaussian distribution information of multiple locations, this embodiment divides the scene into multiple areas, and determines the gradient information in each area based on the Gaussian distribution information in each area to determine whether the area is an information-intensive area, thereby adjusting the Gaussian distribution information based on the gradient information of each area to refine the geometric representation of the dense area, thereby effectively saving computing resources and effectively improving the display effect of the high-detail area.

[0180] In some embodiments of the present application, the scene enhancement device based on three-dimensional Gaussian can be implemented in the form of a computer program. The computer program can be used in Figure 7 The computer device shown in the figure is run on the computer device. The memory of the computer device can store various program modules constituting the three-dimensional Gaussian-based scene enhancement device, for example, Figure 6 The generating module 610, determining module 620 and reconstructing module 630 are shown. The computer program composed of various program modules enables the processor to execute the steps of the particle animation generating method of each embodiment of the present application described in this specification.

[0181] For example, Figure 7 The computer device shown can be Figure 6 The generation module 610 in the three-dimensional Gaussian-based scene enhancement device shown executes step S110. The computer device can execute step S120 through the determination module 620. The computer device can execute step S130 through the reconstruction module 630. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external computer device through a network connection. When the computer program is executed by the processor, a three-dimensional Gaussian-based scene enhancement method is implemented.

[0182] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0183] In some embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program. The computer program is loaded by a processor, so that the processor performs the following steps:

[0184] Generate an initial Gaussian scene based on the point cloud data, wherein the initial Gaussian scene includes Gaussian distribution information of multiple positions;

[0185] Based on Gaussian distribution information of multiple regions in the initial Gaussian scene, determining gradient information in each of the regions;

[0186] The Gaussian distribution information is adjusted based on the gradient information to obtain a reconstructed Gaussian scene of the point cloud data.

[0187] After relying on point cloud data to generate an initial Gaussian scene composed of Gaussian distribution information of multiple locations, this embodiment divides the scene into multiple areas, and determines the gradient information in each area based on the Gaussian distribution information in each area to determine whether the area is an information-intensive area, thereby adjusting the Gaussian distribution information based on the gradient information of each area to refine the geometric representation of the dense area, thereby effectively saving computing resources and effectively improving the display effect of the high-detail area.

[0188] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Any reference to memory, storage, information library or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0189] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0190] The above is a detailed introduction to a three-dimensional Gaussian-based scene enhancement method, device, equipment and storage medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A scene enhancement method based on three-dimensional Gaussian, characterized in that: include: Generate an initial Gaussian scene based on the point cloud data, wherein the initial Gaussian scene includes Gaussian distribution information of multiple positions; Based on Gaussian distribution information of multiple regions in the initial Gaussian scene, determining gradient information in each of the regions; The Gaussian distribution information is adjusted based on the gradient information to obtain a reconstructed Gaussian scene of the point cloud data.

2. The method according to claim 1, characterized in that The step of adjusting the Gaussian distribution information based on the gradient information to obtain a reconstructed Gaussian scene of the point cloud data includes: Based on the gradient information of each position in the region, determining the number of positions in the region where the gradient information is greater than a preset gradient threshold; If the number is greater than a preset number threshold, the Gaussian distribution information in the area is adjusted to obtain a reconstructed Gaussian scene of the point cloud data.

3. The method according to claim 2, characterized in that The adjusting the Gaussian distribution information in the area includes: determining whether the region is uniformly distributed based on gradient information within the region; In the case where the region is a non-uniformly distributed region, assigning Gaussian distribution information of multiple positions in the region to neighboring positions to obtain adjusted Gaussian distribution information; and / or In the case that the region is a uniformly distributed region, the Gaussian distribution information in the region is adjusted based on a loss value between the Gaussian distribution information in the region and an average Gaussian distribution to obtain adjusted Gaussian distribution information.

4. The method according to claim 1, characterized in that: The generating of an initial Gaussian scene based on point cloud data comprises: Input the point cloud data into the trained target model to determine multiple locations and determine the Gaussian distribution information of each location to generate an initial Gaussian scene; The target model is trained through the following steps: Acquire sample point cloud data and data labels of the sample point cloud data, wherein the data labels include label positions and label normal vectors of the label positions; Inputting the sample point cloud data into an initial model to obtain a predicted Gaussian scene of the sample point cloud data, wherein the predicted Gaussian scene includes Gaussian distribution information of a plurality of predicted positions; Determine a model loss value of the initial model based on a first loss value between the predicted position and the label position, and a second loss value between the direction vector in the Gaussian distribution information and the label normal vector; The initial model is trained according to the model loss value until a target model is obtained.

5. The method according to claim 4, characterized in that The method further comprises: Obtaining Gaussian distribution information of multiple reference positions within the grid where the predicted position is located; determining a third loss value based on a difference between the Gaussian distribution information of the predicted position and Gaussian distribution information of a plurality of reference positions; The determining of the model loss value of the initial model based on the first loss value between the predicted position and the label position, and the second loss value between the direction vector in the Gaussian distribution information and the label normal vector, comprises: Based on the first loss value between the predicted position and the label position, the second loss value between the direction vector in the Gaussian distribution information and the label normal vector, and the third loss value, the model loss value of the initial model is determined.

6. The method according to claim 5, characterized in that The determining of the model loss value of the initial model based on the first loss value between the predicted position and the label position, the second loss value between the direction vector in the Gaussian distribution information and the label normal vector, and the third loss value comprises: Determining weights of multiple loss values ​​based on the number of training times of the initial model; The first loss value, the second loss value and the third loss value are weighted according to the weight to obtain a model loss value of the initial model.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: extracting scene feature points from a plurality of scene images of a target scene; The point cloud data of the target scene is determined based on the overlapping information of the scene feature points, wherein the point cloud density of the point cloud data is lower than a preset density threshold.

8. A scene enhancement device based on three-dimensional Gaussian, characterized in that: include: A generating module, configured to generate an initial Gaussian scene based on the point cloud data, wherein the initial Gaussian scene includes Gaussian distribution information of multiple locations; A determination module, configured to determine the gradient information in each of the regions based on the Gaussian distribution information of the multiple regions in the initial Gaussian scene; A reconstruction module is used to adjust the Gaussian distribution information based on the gradient information to obtain a reconstructed Gaussian scene of the point cloud data.

9. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the three-dimensional Gaussian-based scene enhancement method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded by a processor to execute the three-dimensional Gaussian-based scene enhancement method according to any one of claims 1 to 7.

Images