A volumetric cloud rendering method and system based on 3D Gaussian splashing

Abstract

This invention discloses a volumetric cloud rendering method and system based on 3D Gaussian splashing. The method includes: using the SfM algorithm to perform feature matching and structure reconstruction on the volumetric cloud image to obtain sparse point clouds in the scene, represented as anisotropic Gaussian ellipsoids; decomposing the illumination transmission process of the volumetric cloud into two parts, forward single scattering and internal multiple scattering, based on the radiative transfer equation, and embedding the illumination modeling formula into the volumetric cloud rendering process; determining the contribution of each 3D Gaussian ellipsoid to the rendering result based on its opacity index, and using a delayed deletion strategy to dynamically prune low-contribution 3D Gaussian ellipsoids to reduce redundancy; constructing a comprehensive loss function to focus the 3D Gaussian model on the volumetric cloud region and enhance the ability to restore structural details; and outputting the optimized volumetric cloud rendering result and the 3D Gaussian model. This invention effectively improves the efficiency and quality of volumetric cloud rendering while ensuring real-time rendering.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer graphics and three-dimensional rendering, and particularly relates to a volume cloud rendering method and system based on three-dimensional Gaussian splatter. BACKGROUND

[0002] Volume cloud rendering technology is an important research direction in the field of computer graphics, and is widely used in weather simulation, disaster simulation, virtual reality and other scenes with high requirements for visual realism. Due to the complex reflection, refraction and absorption and other optical phenomena inside the volume cloud, the realization of physically consistent rendering usually depends on the accurate simulation of high-order scattering events, which requires tracking complex light propagation paths, resulting in high computational overhead. The existing resource conditions often cannot meet the requirements of real-time rendering.

[0003] The existing volume cloud rendering method is mostly based on the radiative transfer equation (RTE) as the theoretical basis, and the multiple scattering process of photons in the cloud layer is solved by using numerical methods such as Monte Carlo integration to simulate realistic effects. This kind of method has high physical consistency, can accurately model the multiple scattering process inside the volume medium, and has been widely used in offline rendering and high-quality image generation. However, this kind of method usually has low sampling efficiency and slow convergence speed, and is difficult to meet the performance requirements of real-time or interactive rendering. In order to more efficiently model the light transmission process in the volume cloud, deep learning methods are tried to be introduced, such as introducing diffusion approximation, pre-computing cache or neural network acceleration strategy to improve rendering speed; using radiation prediction neural network combined with Monte Carlo integration on GPU to quickly estimate the irradiance of the cloud, while integrating geometric light path tracing to improve accuracy. Although the above methods have made significant improvements in visual quality, their reasoning still needs several minutes of calculation, which is difficult to apply to real-time scenes, especially in interactive virtual environments.

[0004] In recent years, 3D Gaussian Splatting (3DGS) has gradually become an important technique for 3D scene modeling and rendering due to its combination of high fidelity and real-time performance in novel view synthesis tasks. This method uses a 3D Gaussian ellipsoid to continuously represent the scene and achieves rapid rendering through differentiable rasterization, significantly improving real-time performance while maintaining visual quality. However, existing 3DGS methods are mainly designed for surface scenes. When dealing with complex volumetric media such as volumetric clouds, which have strong viewpoint dependence and anisotropic scattering, it is difficult to accurately model the multiple scattering processes, leading to low rendering quality and artifacts. This method lacks a physically consistent representation of light transmission within volumetric clouds, making it difficult to accurately simulate the multiple anisotropic scattering effects within the volumetric medium, resulting in image distortion, loss of detail, and artifacts, affecting the realism of the rendering. Furthermore, its reliance on a high-density Gaussian distribution, with a large number of low-contribution Gaussians, not only incurs additional computational overhead but also further impacts image quality.

[0005] Therefore, how to achieve physically consistent and detailed rendering of volumetric clouds without sacrificing efficiency has become an important problem to be solved in current volumetric media rendering. Summary of the Invention

[0006] Purpose of the invention: In order to solve the problems existing in the prior art, the present invention provides a volumetric cloud rendering method and system based on three-dimensional Gaussian splashing, which effectively improves the efficiency and quality of volumetric cloud rendering while ensuring real-time rendering.

[0007] This invention adopts the following technical solution: a volumetric cloud rendering method based on three-dimensional Gaussian splashing, the steps of which include:

[0008] S1. Input a set of volumetric cloud scene images, and use the Structure from Motion (SfM) algorithm to perform feature matching and structure reconstruction, and extract the sparse point cloud in the scene and the camera pose corresponding to the image.

[0009] S2. Represent the acquired sparse point cloud as an anisotropic three-dimensional Gaussian ellipsoid. Each three-dimensional Gaussian ellipsoid has the following attributes: position parameter, scaling factor, rotation factor, opacity parameter, and spherical harmonic coefficient.

[0010] S3. The illumination transmission process of volumetric clouds is modeled based on the radiative transfer equation. The illumination transmission process is decomposed into two parts: forward single scattering and internal multiple scattering, and the illumination modeling formula is obtained.

[0011] S4. Embed the forward single scattering component and internal multiple scattering component in the lighting modeling formula into the volumetric cloud rendering process to generate rendered images of the volumetric cloud from different perspectives and obtain the initial three-dimensional Gaussian model.

[0012] S5, determining the contribution of each three-dimensional Gaussian ellipsoid to the rendering result according to the opacity index of the three-dimensional Gaussian ellipsoid, and dynamically pruning low-contribution three-dimensional Gaussian ellipsoids using a delayed deletion strategy to reduce redundancy;

[0013] S6, constructing a comprehensive loss function including a basic reconstruction loss, a local perception loss based on a foreground mask, and an opacity sparsity regularization term, to iteratively optimize the three-dimensional Gaussian model using the input image as a supervision signal, so that the three-dimensional Gaussian model focuses on the volumetric cloud region and enhances the structure detail restoration capability;

[0014] S7, outputting the optimized volumetric cloud rendering result and the three-dimensional Gaussian model to achieve high-quality rendering of the volumetric cloud.

[0015] Preferably, in step S1, a multi-view image sequence of the input volumetric cloud scene is input, and the input image is preprocessed; the SfM algorithm is used for feature matching and structure reconstruction to obtain a sparse point cloud and camera pose, which are used as initialization basis for subsequent Gaussian representation and rendering optimization.

[0016] Preferably, in step S2, the obtained sparse point cloud is represented as an anisotropic three-dimensional Gaussian ellipsoid, including the following sub-steps:

[0017] S21, representing each three-dimensional point in the sparse point cloud as an anisotropic three-dimensional Gaussian ellipsoid , The formula is as follows:

[0018] ;

[0019] Wherein, represents a position vector, represents the center position of the Gaussian, represents a three-dimensional covariance matrix;

[0020] S22, assigning an opacity attribute to each three-dimensional Gaussian ellipsoid ;

[0021] S23, for the color attribute of the three-dimensional Gaussian ellipsoid, using a spherical harmonic function (Spherical Harmonics, SH) with an order of 3 for modeling.

[0022] Preferably, in step S3, the light transmission process in the volumetric cloud is divided into forward single scattering and internal multiple scattering, and the forward single scattering component and the internal multiple scattering component are calculated through the illumination modeling formula.

[0023] For the forward single-scattering part, only direct radiation from the sun direction is considered, and multiple propagation and re-scattering after scattering are ignored; the change of radiation intensity of light rays when passing through the volume cloud medium can be calculated by the differential form of the radiative transfer equation under the single-scattering condition It is described that when the exit point of the volume cloud medium is not illuminated by external incident light , the boundary radiation intensity can be considered as zero, and the integral of the formula at both ends is carried out to obtain the forward single-scattering component ;

[0024] For the internal multiple-scattering part, a constant is introduced to represent the average radiation intensity formed by the internal and surrounding environment radiation field in the multiple-scattering process; under this condition, the change rate of the internal multiple-scattering component of the volume cloud along the light ray propagation path is calculated ; the same condition as the forward single-scattering is adopted, that is, the boundary radiation intensity is zero, and the integral of the formula at both ends is carried out to obtain the internal multiple-scattering component . .

[0025] Preferably, in step S4, the forward single-scattering component and the internal multiple-scattering component in the light modeling formula are embedded into the volume cloud rendering process, including the following sub-steps:

[0026] S41, project the three-dimensional Gaussian ellipsoid to the two-dimensional image plane of the target view angle through the view transformation matrix to obtain the corresponding two-dimensional Gaussian ellipse;

[0027] S42, globally sort the projected two-dimensional Gaussian ellipse according to the depth value, that is, the distance of the center of the three-dimensional Gaussian ellipsoid from the camera, from small to large;

[0028] S43, for each pixel in the target view angle, perform color calculation, and calculate the contribution of the forward single-scattering term and the internal multiple-scattering term to the final color value of the pixel, respectively;

[0029] S44, based on the contribution of the forward single-scattering term and the internal multiple-scattering term to the final color value of the pixel, calculate the final color value of the pixel.

[0030] Preferably, in step S5, the contribution of each three-dimensional Gaussian ellipsoid to the rendering result is determined according to the opacity index of the three-dimensional Gaussian ellipsoid, and a delayed deletion strategy is adopted to dynamically crop low-contribution three-dimensional Gaussian ellipsoids, including the following sub-steps: ​​​

[0031] S51, for each three-dimensional Gaussian ellipsoid, calculate its opacity at the current view angle according to the total extinction coefficient and the path length ; if the opacity of a certain three-dimensional Gaussian ellipsoid continues to be less than a threshold , it means that its contribution to the rendering result is very small and can be regarded as a redundant point;

[0032] S52, use a soft constraint mechanism to delete redundant points, in which first add loss as a regularizer, introduce an opacity-based loss term during the training process to optimize the three-dimensional Gaussian model;

[0033] S53, during the model training process, record the opacity of each three-dimensional Gaussian ellipsoid, and calculate its average opacity in several consecutive training rounds;

[0034] S54, when a certain three-dimensional Gaussian ellipsoid maintains very low opacity in consecutive training rounds, it means that its contribution to the final pixel color is very small, and it is marked as a pruning object;

[0035] S55, adopt a delayed deletion strategy for pruning objects, perform batch pruning every fixed number of steps, and uniformly execute pruning after several training round iterations to achieve precise dynamic pruning of three-dimensional Gaussian ellipsoids.

[0036] Preferably, in step S6, a comprehensive loss function is constructed including the basic reconstruction loss, the local perception loss based on the foreground mask, and the opacity sparsity regularization term, including the following sub-steps:

[0037] S61, based on the pixel difference between the input image and the rendering result, calculate the basic reconstruction loss to guide the three-dimensional Gaussian model to learn finer features in the foreground area and improve the overall rendering quality;

[0038] S62, use the alpha channel mask of the input image to extract the foreground area of the volumetric cloud, and construct a local perception loss in the area to enhance the model's learning of cloud edge and internal details;

[0039] S63, combine the basic reconstruction loss, the local perception loss based on the foreground mask, and the opacity sparsity regularization by weighting to construct a comprehensive loss function ;

[0040] ​S64, based on the above comprehensive loss function, the three-dimensional Gaussian model is iteratively optimized, and each attribute parameter of the three-dimensional Gaussian ellipsoid is continuously adjusted through back propagation, including position parameter, scaling factor, rotation factor, opacity parameter and spherical harmonic function coefficient, so that the authenticity and detail performance of the volume cloud rendering result are gradually improved.

[0041] Preferably, the output of the optimized volume cloud rendering result and the three-dimensional Gaussian model in step S7 realizes high-quality rendering of the volume cloud, including the following sub-steps:

[0042] S71, based on the optimized three-dimensional Gaussian model and the corresponding camera parameters, the volume cloud image of the input image view and the target view is rendered, and the rendering result is output;

[0043] S72, by comparing the rendering result with the input image in terms of lighting consistency and detail restoration, the three-dimensional reconstruction accuracy and the new view synthesis quality of the model are verified;

[0044] S73, output the finally generated volume cloud new view rendering result and the corresponding three-dimensional Gaussian point cloud data, realize high-quality reconstruction and realistic visual display of the volume cloud scene.

[0045] The technical scheme of the present application also provides a volume cloud rendering system based on three-dimensional Gaussian splashing, which is used to implement the above-mentioned method, comprising: a preprocessing module, a Gaussian representation module, a lighting modeling module, a three-dimensional rendering module, an opacity clipping and density regulation module, a loss optimization module and an output module.

[0046] The preprocessing module is used for receiving multi-view images of the volume cloud scene, performing feature matching and structure reconstruction by using the SfM algorithm, and obtaining scene sparse point cloud and corresponding camera pose;

[0047] The Gaussian representation module is used for converting the sparse point cloud into anisotropic three-dimensional Gaussian ellipsoid representation, each three-dimensional Gaussian ellipsoid including position parameter, scaling factor, rotation factor, opacity parameter and spherical harmonic function coefficient;

[0048] The lighting modeling module is used for modeling the light transmission process of the volume cloud, decomposing the light transmission process into forward single scattering component and internal multiple scattering component based on the radiative transfer equation, obtaining a lighting modeling formula which can be embedded in the rendering process, and accurately describing the lighting characteristics of the volume cloud;

[0049] The rendering module is used for projecting the three-dimensional Gaussian ellipsoid to a two-dimensional image plane, and sorting and synthesizing the projection result based on depth information, combining the lighting modeling formula to calculate the pixel color, and obtaining the volume cloud rendering result under different views;

[0050] The opacity pruning and density regulation module is used for pruning and density regulation of low-contribution three-dimensional Gaussian ellipsoids, determines the contribution of the three-dimensional Gaussian ellipsoid to the rendering result according to an opacity index of the three-dimensional Gaussian ellipsoid, and performs delayed deletion and adaptive density adjustment on the low-contribution three-dimensional Gaussian ellipsoid, thereby reducing redundancy and improving rendering efficiency;

[0051] The loss optimization module is used for constructing a comprehensive loss function and optimizing a three-dimensional Gaussian model, constructing a comprehensive loss function composed of a basic reconstruction loss, a local perception loss based on a foreground mask and an opacity sparsity regularization term, and iteratively optimizing based on an input image to enhance the structural details of the volume cloud region;

[0052] The output module is used for outputting the optimized volume cloud rendering result and the three-dimensional Gaussian model, and realizing high-quality rendering of the volume cloud.

[0053] Compared with the prior art, the above technical scheme has the following technical effects:

[0054] 1. The present application physically and consistently models the complex light transmission process in the volume cloud, divides it into a single scattering term and an internal multiple scattering term from the radiative transfer equation, and embeds the modeling and rendering process of 3DGS, thereby realizing efficient and physically consistent volume cloud rendering without changing the original rendering pipeline, and effectively improving the image distortion, detail loss and artifacts that occur when the existing method renders the volume cloud.

[0055] 2. The present application proposes a delayed deletion strategy based on opacity, determines the contribution of the three-dimensional Gaussian ellipsoid to the rendering result based on the opacity index, dynamically perceives the influence of the three-dimensional Gaussian ellipsoid on the rendering result, realizes accurate pruning of low-contribution redundant three-dimensional Gaussian ellipsoids, and compresses the model size and improves the rendering efficiency while ensuring the rendering quality.

[0056] 3. The present application explicitly models the foreground region of the volume cloud, guides the three-dimensional Gaussian model to focus on the cloud region during the training process, constructs a comprehensive loss function, enhances the modeling ability of the model for the structure and hierarchical details of the foreground of the volume cloud, improves the realism of the rendering result, and improves the visual quality at the cloud internal details. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The flow chart of the volume cloud rendering method based on three-dimensional Gaussian splashing of the present application is shown in the figure;

[0058] Figure 2 The overall framework diagram of the algorithm of the present application is shown in the figure;

[0059] Figure 3A comparison chart of rendering effects of the method of the present application and four methods of 3DGS, Mini-Splatting, MVGS and Scaffold-GS;

[0060] Figure 4 A graph of the relationship between the three-dimensional Gaussian ellipsoid number of the present application and image quality. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions of the application will be further described in detail below in combination with the drawings, and the described embodiments are only a part of the embodiments involved in the present application. All non-innovative embodiments of other researchers in the field on the basis of the embodiments belong to the protection scope of the present application. At the same time, the step numbers in the embodiments of the present application are only set for the convenience of description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0062] Embodiment One

[0063] The present embodiment provides a three-dimensional Gaussian splatting-based volume cloud rendering method, as shown in Figure 1 and Figure 2 , specifically comprising the following steps:

[0064] S1, input a set of images of a volume cloud scene, perform feature matching and structure reconstruction by using a motion structure recovery algorithm (Structure from Motion, SfM), obtain a sparse point cloud in the scene and a camera pose corresponding to the images, and use them as an initialization basis for subsequent Gaussian representation and rendering optimization.

[0065] S2, represent the obtained sparse point cloud as an anisotropic three-dimensional Gaussian ellipsoid, each three-dimensional Gaussian ellipsoid contains position parameters, a scaling factor, a rotation factor, an opacity parameter and spherical harmonic function coefficients and other attributes, including the following sub-steps:

[0066] S21, represent each three-dimensional point in the sparse point cloud as an anisotropic three-dimensional Gaussian ellipsoid , the formula is as follows:

[0067] ;

[0068] wherein, represents a position vector, represents the center position of the Gaussian, the superscript represents transposition.

[0069] represents a three-dimensional covariance matrix, which is decomposed into a scale matrix and a rotation matrix , using superscript denotes transpose, The formula is as follows:

[0070] .

[0071] S22, assigning an opacity attribute to each three-dimensional Gaussian ellipsoid .

[0072] S23, using Spherical Harmonics (SH) of order 3 to model the color attribute of the three-dimensional Gaussian ellipsoid.

[0073] S3, based on the radiative transfer equation, the light transmission process of the volume cloud is modeled, which is decomposed into forward single scattering and internal multiple scattering two parts, and the light modeling formula is obtained, including the following sub-steps:

[0074] S31, for the forward single scattering part, only direct radiation from the sun direction is considered, and multiple propagation and re-scattering after scattering are ignored; the change of radiation intensity of light when passing through the volume cloud medium can be described by the differential form of the single scattering condition of the radiative transfer equation, and the differential form equation is as follows:

[0075] ;

[0076] wherein, is the radiation intensity attenuation term caused by absorption and scattering in the propagation process of the light, which reflects the energy attenuation process; denotes the newly added radiation intensity in the medium segment by scattering of sunlight; is the total extinction coefficient, including the absorption coefficient and the scattering coefficient; is the forward single scattering component; denotes the forward single scattering coefficient; is the Henyey-Greenstein phase function, is the angle between the light and the viewing direction; is the radiation intensity of the incident sunlight.

[0077] is the cumulative optical thickness of the light from the incident point to a certain position , used to calculate the energy attenuation ratio of the light in the medium propagation, denotes the path length of the light from to , The formula is as follows:

[0078] .

[0079] When the exit point of the volume cloud medium is not illuminated by external incident light, the boundary radiation intensity can be considered as zero, i.e. Integrating both ends of the formula, the expression of is as follows:

[0080] ;

[0081] S32, for the internal multiple scattering part, a constant is introduced to represent the average radiation intensity formed by the internal and surrounding environment radiation field in the multiple scattering process; under this condition, the change rate of the internal multiple scattering component of the volume cloud along the light propagation path is:

[0082] ;

[0083] wherein, represents the internal multiple scattering coefficient, represents the intensity attenuation caused by absorption and scattering in the propagation process of the light, represents the contribution of the internal radiation field of the medium to the current direction radiation intensity after multiple scattering.

[0084] The same conditions as the forward single scattering are adopted, i.e. Integrating both ends of the formula, the formula of is as follows:

[0085] .

[0086] S4, the forward single scattering component and the internal multiple scattering component in the illumination modeling formula are embedded into the volume cloud rendering process to generate the rendering image of the volume cloud under different viewing angles, and an initial three-dimensional Gaussian model is obtained, including the following sub-steps:

[0087] S41, the three-dimensional Gaussian ellipsoid is projected onto the two-dimensional image plane of the target viewing angle through the view transformation matrix to obtain the corresponding two-dimensional Gaussian ellipse.

[0088] As a preferred, the two-dimensional covariance matrix is calculated according to the following formula:

[0089] ;

[0090] wherein, represents the Jacobian matrix of the perspective projection.

[0091] ​​​S42, the two-dimensional Gaussian ellipses after projection are globally sorted according to the depth value, i.e. the distance from the center of the three-dimensional Gaussian ellipsoid to the camera, from small to large.

[0092] S43, for each pixel in the target view color calculation is performed, respectively calculating the forward single scattering term and the internal multiple scattering term contribution to the final color value of the pixel.

[0093] S431, the forward single scattering term contributes to the final color value of the pixel . which can be expressed as:

[0094] ;

[0095] wherein, is the color of the th three-dimensional Gaussian ellipsoid, is the product of the value of the th three-dimensional Gaussian ellipsoid and its learned opacity , denotes the cumulative transmittance at the th three-dimensional Gaussian ellipsoid, i.e. the probability that a light ray does not get absorbed between the camera and the th three-dimensional Gaussian ellipsoid, denotes the angle between the direction of the incident light at the center of the th three-dimensional Gaussian ellipsoid and the direction of the view.

[0096] As a preferred, the formula is as follows:

[0097] ;

[0098] S432, the internal multiple scattering term contributes to the final color value of the pixel . which can be expressed as:

[0099] .

[0100] S44, the final color value of the pixel can be expressed as:

[0101] ;

[0102] wherein, is the subset of three-dimensional Gaussian ellipsoids involved in the pixel , and the three-dimensional Gaussian ellipsoids in the subset are sorted according to their depth value from small to large.

[0103] S5, determining the contribution of each three-dimensional Gaussian ellipsoid to the rendering result according to the opacity index of the three-dimensional Gaussian ellipsoid, and using a delayed deletion strategy to dynamically prune low-contribution three-dimensional Gaussian ellipsoids to reduce redundancy, including the following sub-steps:

[0104] S51, for each three-dimensional Gaussian ellipsoid, calculating its opacity at the current viewing angle according to the total extinction coefficient and the path length ; if the opacity of a certain three-dimensional Gaussian ellipsoid continues to be less than a threshold , it means that its contribution to the rendering result is very small and can be regarded as a redundant point.

[0105] As a preferred, the calculation formula is as follows:

[0106] ;

[0107] S52, using a soft constraint mechanism to delete redundant points, in which first loss is added as a regularizer, and an opacity-based loss term is introduced in the training process to optimize the three-dimensional Gaussian model.

[0108] The loss term formula is as follows:

[0109] ;

[0110] wherein is a regularization weight coefficient for adjusting the influence degree of the opacity regularization term in the comprehensive loss function to achieve sparse constraint on low-opacity three-dimensional Gaussian ellipsoids.

[0111] S53, during the model training process, recording the opacity of each three-dimensional Gaussian ellipsoid and calculating its average opacity in several consecutive training rounds.

[0112] S54, when a certain three-dimensional Gaussian ellipsoid maintains very low opacity in consecutive training rounds, it means that its contribution to the final pixel color is very small, and it is marked as a pruning object.

[0113] S55, using a delayed deletion strategy for pruning objects, performing batch pruning every fixed number of steps, and uniformly executing pruning after several training round iterations to achieve precise dynamic pruning of three-dimensional Gaussian ellipsoids.

[0114] ​S6, construct a comprehensive loss function including a basic reconstruction loss, a local perception loss based on a foreground mask, and an opacity sparsity regularization term, and iteratively optimize the three-dimensional Gaussian model based on the input image as a supervision signal to focus on the volumetric cloud region and enhance the structure detail restoration capability, including the following sub-steps:

[0115] S61, based on the pixel difference between the input image and the rendering result, calculate the basic reconstruction loss, guide the three-dimensional Gaussian model to learn finer features in the foreground region, and improve the overall rendering quality.

[0116] S62, use the alpha channel mask of the input image to extract the foreground region of the volumetric cloud, and construct a local perception loss in the region to enhance the learning of cloud edge and internal details by the model.

[0117] As a preferred, the foreground perception loss The calculation formula is as follows:

[0118] ;

[0119] Wherein, represents the value of the rendered image at pixel i, represents the value of the corresponding real image at pixel i, is a mask generated based on the alpha channel of the image, which is used to indicate whether pixel i is a cloud region.

[0120] S63, combine the basic reconstruction loss, the local perception loss based on the foreground mask, and the opacity sparsity regularization term by weighting, and construct a comprehensive loss function The formula is as follows:

[0121] ;

[0122] Wherein, , and respectively represent the weight coefficient of the corresponding loss term, represents the pixel-wise L1 loss between the reconstructed image and the real image, represents the structure similarity loss based on the perception quality.

[0123] S64, iteratively optimize the three-dimensional Gaussian model based on the above comprehensive loss function, continuously adjust various attribute parameters of the three-dimensional Gaussian ellipsoid through back propagation, including position parameters, scaling factors, rotation factors, opacity parameters and spherical harmonic function coefficients, and gradually improve the reality and detail performance of the volumetric cloud rendering result.

[0124] S7, output the optimized volumetric cloud rendering result and three-dimensional Gaussian model, realize high-quality rendering of the volumetric cloud, including the following sub-steps:

[0125] S71, based on the optimized three-dimensional Gaussian model and the corresponding camera parameters, render the volumetric cloud image of the input image view and the target view, and output the rendering result;

[0126] S72, verify the three-dimensional reconstruction accuracy and new view synthesis quality of the model by comparing the lighting consistency and detail restoration degree of the rendering result and the input image;

[0127] S73, output the finally generated volumetric cloud new view rendering result and the corresponding three-dimensional Gaussian point cloud data, realize high-quality reconstruction and realistic visualization display of the volumetric cloud scene.

[0128] Embodiment two

[0129] The embodiment provides a volumetric cloud rendering system based on three-dimensional Gaussian splash, which comprises a preprocessing module, a Gaussian representation module, a lighting modeling module, a three-dimensional rendering module, an opacity clipping and density regulation module, a loss optimization module and an output module.

[0130] The preprocessing module is used for receiving multi-view images of a volumetric cloud scene, performing feature matching and structure reconstruction by using an SfM algorithm, and obtaining a scene sparse point cloud and corresponding camera poses;

[0131] The Gaussian representation module is used for converting the sparse point cloud into an anisotropic three-dimensional Gaussian ellipsoid representation, and each three-dimensional Gaussian ellipsoid comprises a position parameter, a scaling factor, a rotation factor, an opacity parameter and spherical harmonic function coefficients;

[0132] The lighting modeling module is used for modeling the light transmission process of the volumetric cloud, decomposing the light transmission process into a forward single scattering component and an internal multiple scattering component based on a radiative transfer equation, obtaining a lighting modeling formula which can be embedded in a rendering process, and accurately describing the lighting characteristics of the volumetric cloud;

[0133] The rendering module is used for projecting the three-dimensional Gaussian ellipsoid to a two-dimensional image plane, sorting and synthesizing the projection result based on depth information, calculating pixel color in combination with the lighting modeling formula, and obtaining the volumetric cloud rendering result under different views;

[0134] The opacity clipping and density regulation module is used for clipping and density regulation of low-contribution three-dimensional Gaussian ellipsoids, determining the contribution degree of the three-dimensional Gaussian ellipsoid to the rendering result according to the opacity index of the three-dimensional Gaussian ellipsoid, and performing delayed deletion and adaptive density adjustment on the low-contribution three-dimensional Gaussian ellipsoid, so as to reduce redundancy and improve rendering efficiency;

[0135] The loss optimization module is configured to construct a comprehensive loss function and optimize the three-dimensional Gaussian model, construct a comprehensive loss function composed of a basic reconstruction loss, a local perception loss based on a foreground mask, and an opacity sparsity regularization term, and perform iterative optimization based on the input image to enhance the structural details of the volume cloud region.

[0136] The output module is configured to output the optimized volume cloud rendering result and the three-dimensional Gaussian model, and realize high-quality rendering of the volume cloud.

[0137] In order to verify the improvement of the method proposed in the application on the performance of volume cloud rendering, the performance of the method proposed in the application and four comparative methods (3DGS, Mini-Splatting, MVGS and Scaffold-GS) on five volume cloud data sets (Cloud1-Cloud5) is compared comprehensively. The rendering results are shown in Figure 3 It can be seen that the rendering quality of the method proposed in the application is better than that of the other four comparative methods in terms of overall and detailed rendering quality, which verifies the superiority of the method of the application.

[0138] In order to evaluate the trade-off ability between model complexity and rendering quality of each method, the "number of three-dimensional Gaussian ellipsoids-image quality" relationship diagram on the five volume cloud data sets is further drawn, as shown in Figure 4 (a) in Figure 4 (b) is the relationship between the number of three-dimensional Gaussian ellipsoids and the LPIPS index. Figure 4

[0139] From Figure 4 It can be observed that the method proposed in the application uses the least number of three-dimensional Gaussian ellipsoids among all methods, but achieves the highest image quality.

[0140] Table 1 compares the model complexity of each method in the five volume cloud data sets, using the number of three-dimensional Gaussian ellipsoids as an evaluation index. The bold data is the optimal value, and the larger the data, the better, and the smaller the data, the better. It can be seen that the method of the application achieves better compression effect and greatly reduces the complexity of the model.

[0141] Table 1 Comparison of the number of three-dimensional Gaussian ellipsoids required by five algorithms

[0142]

[0143] Table 2 shows the quantitative evaluation results of the image quality of the five algorithms in the volume cloud rendering task. It can be seen that the method proposed in the application leads other methods in the three indicators of PSNR, SSIM and LPIPS, showing stronger structure restoration ability and perceptual quality performance.

[0144] ​Table 2 Quantitative comparison of five algorithms

[0145]

[0146] The present application performs an ablation experiment on the Cloud3 scene, and Table 3 shows the specific experimental settings and results. In Table 3, "P" represents adding the module, and "X" represents not adding the module.

[0147] The present application respectively constructs three comparative models:

[0148] M1: does not contain illumination modeling, foreground perception and opacity clipping; M2: contains only illumination modeling; M3: contains illumination modeling and foreground perception.

[0149] It can be seen that, compared with the above three configurations, the method of the present application contains all modules as a complete version.

[0150] Table 3 Ablation experiment results

[0151]

[0152] The experiment on the Cloud3 scene quantitatively shows the influence of each module on rendering accuracy and efficiency. Comparing M1 with M2 which only adds illumination modeling, it can be observed that PSNR and SSIM are improved, while LPIPS is reduced, indicating that illumination modeling effectively captures the single scattering and multiple scattering processes of volumetric clouds, significantly improving the structural fidelity and perceptual quality of the image. On this basis, further adding the local perception loss based on the foreground of the volumetric cloud, the PSNR is improved, and it can be seen that this loss term can guide the model to pay attention to the detail levels of the cloud foreground. Finally, the opacity clipping is introduced, the number of three-dimensional Gaussian ellipsoids is greatly reduced, the model size is compressed by nearly two-thirds, and at the same time, PSNR, SSIM continues to improve slightly, while LPIPS further decreases to 0.041, indicating that while significantly reducing the model size, the rendering quality not only does not decrease but also improves.

[0153] The above is only the preferred embodiment of the present application, it should be pointed out that: for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A volume cloud rendering method based on three-dimensional Gaussian splatting, characterized in that, The method comprises the following steps: S1, input a set of images of a volumetric cloud scene, perform feature matching and structure reconstruction by using a motion structure recovery algorithm, and obtain a sparse point cloud in the scene and a camera pose corresponding to the images; S2, represent the obtained sparse point cloud as an anisotropic three-dimensional Gaussian ellipsoid, and each three-dimensional Gaussian ellipsoid attribute comprises a position parameter, a scaling factor, a rotation factor, an opacity parameter and a spherical harmonic function coefficient; S3, model a light transmission process of the volumetric cloud based on a radiative transfer equation, decompose the light transmission process into a forward single scattering and an internal multiple scattering, and obtain a light modeling formula; For the forward single scattering part, only direct radiation from the sun direction is considered, and multiple propagation and re-scattering after scattering are ignored; The change of radiation intensity of a light ray when passing through the volumetric cloud medium is described by a differential form of the single scattering condition of the radiative transfer equation, and the differential form equation is as follows: ； wherein, is the radiation intensity attenuation term caused by absorption and scattering of light during propagation, reflecting the energy attenuation process; represents the newly added radiation intensity from the scattering of sunlight within the medium segment ; is the total extinction coefficient, including the absorption coefficient and the scattering coefficient; is the forward single scattering component; represents the forward single scattering coefficient; is the Henyey-Greenstein phase function, is the angle between the light and the viewing direction; is the radiation intensity of the incident sunlight; is the cumulative optical thickness of the light from the incident point to the position , used to calculate the energy attenuation ratio of the light in the medium propagation; When the exit point of the volumetric cloud medium is not illuminated by external incident light The boundary radiance When the exit point of the volumetric cloud medium is not illuminated by external incident light Integrating both sides of the equation gives the expression for the exit point of the volumetric cloud medium is not illuminated by external incident light ； wherein represents the path length of a light ray from to ; For the internal multiple scattering part, introduce a constant denotes the average radiation intensity formed by the internal and surrounding environment radiation field in the multiple scattering process, the internal multiple scattering component of the volume cloud The rate of change of the light along the light propagation path is: ​ ； wherein, denotes the internal multiple scattering coefficient, denotes the intensity attenuation of the light ray due to absorption and scattering during propagation, denotes the contribution of the radiation field inside the medium to the current directional radiation intensity after multiple scattering; When the exit point of the volumetric cloud medium is not illuminated by external incident light, the boundary radiance is given by the expression Integrating both sides of the equation gives the expression ； S4, embed the forward single scattering component and the internal multiple scattering component in the light modeling formula into a volumetric cloud rendering process, generate a rendering image of the volumetric cloud under different viewing angles, and obtain an initial three-dimensional Gaussian model; S5, determine the contribution of each three-dimensional Gaussian ellipsoid to the rendering result according to the opacity index of the three-dimensional Gaussian ellipsoid, and adopt a delayed deletion strategy to dynamically cut the three-dimensional Gaussian ellipsoid with low contribution, so as to reduce redundancy; S6, construct a comprehensive loss function comprising a basic reconstruction loss, a local perception loss based on a foreground mask and an opacity sparsity regularization term, and iteratively optimize the three-dimensional Gaussian model based on the input image as a supervision signal, so that the three-dimensional Gaussian model focuses on the volumetric cloud region and enhances the structure detail restoration capability, comprising: S61, calculate the basic reconstruction loss based on the pixel difference between the input image and the rendering result, guide the three-dimensional Gaussian model to learn finer features in the foreground region, and improve the overall rendering quality; S62, extract the foreground region of the volumetric cloud by using the alpha channel mask of the input image, and construct a local perception loss in the foreground region to enhance the learning of cloud edge and internal details; Foreground-aware loss The calculation formula is as follows: ； wherein, represents a value of the rendered image at pixel i, represents a value of the corresponding real image at pixel i, is a mask generated based on the alpha channel of the image, indicating whether pixel i is a cloud region or not. S63, combine the base reconstruction loss, the foreground mask based local perceptual loss and the opacity sparsity regularization with weighted combination to construct a comprehensive loss function The formula is as follows: ； wherein, , and denote the weight coefficients for the corresponding loss terms, denotes a pixel-wise LI loss between the reconstructed image and the ground truth image, denotes a structure similarity loss based on perceptual quality; S64, iteratively optimize the three-dimensional Gaussian model based on the comprehensive loss function, continuously adjust the attribute parameters of the three-dimensional Gaussian ellipsoid, including the position parameter, the scaling factor, the rotation factor, the opacity parameter and the spherical harmonic function coefficient, and gradually improve the authenticity and detail performance of the volumetric cloud rendering result; S7, output the optimized volumetric cloud rendering result and the three-dimensional Gaussian model, and realize high-quality rendering of the volumetric cloud.

2. The volume cloud rendering method based on three-dimensional Gaussian splatting according to claim 1, characterized in that, In step S2, the obtained sparse point cloud is represented as an anisotropic three-dimensional Gaussian ellipsoid, comprising the following substeps: S21. representing each three-dimensional point in the sparse point cloud as an anisotropic three-dimensional Gaussian ellipsoid , The formula is as follows: ； wherein denotes a position vector, denotes the center position of the Gaussian, denotes a three-dimensional covariance matrix, the superscript T denotes the transpose; S22, assigning an opacity property to each three-dimensional Gaussian ellipsoid ; S23, for the color attribute of the three-dimensional Gaussian ellipsoid, a spherical harmonic function with an order of 3 is used for modeling.

3. The three-dimensional Gaussian spill-based volume cloud rendering method of claim 1, wherein, In step S4, the forward single scattering component and the internal multiple scattering component in the light modeling formula are embedded into the volumetric cloud rendering process, comprising: S41, project the three-dimensional Gaussian ellipsoid to a two-dimensional image plane of a target viewing angle through a view transformation matrix to obtain a corresponding two-dimensional Gaussian ellipse; S42, globally sort the projected two-dimensional Gaussian ellipse according to the depth value from small to large; S43, for each pixel in the target view Perform color computation, compute forward single scattering term separately With internal multiple scattering term For the pixel Contribution to the final color value And ; S44, pixel The final color value is represented as: ； wherein is a pixel involving a subset of the three-dimensional Gaussian ellipsoid.

4. The volume cloud rendering method based on three-dimensional Gaussian splatting according to claim 3, characterized in that, In step S43, the forward single-scattering term To a pixel Contribution to final color value Is represented as: ； Internal multiple scattering term To a pixel Contribution to final color value Is represented as: ； wherein, is the color of the th three-dimensional Gaussian ellipsoid, is the product of the value of the th three-dimensional Gaussian ellipsoid and the learned opacity , denotes the accumulated transmittance at the th three-dimensional Gaussian ellipsoid, denotes the angle between the incident light direction and the view direction at the center of the th three-dimensional Gaussian ellipsoid.

5. The three-dimensional Gaussian spill-based volume cloud rendering method of claim 4, wherein, In step S5, the contribution of each three-dimensional Gaussian ellipsoid to the rendering result is determined according to the opacity index of the three-dimensional Gaussian ellipsoid, including: S51. For each three-dimensional Gaussian ellipsoid, calculate its opacity at the current view angle according to the total extinction coefficient and path length S52. For each three-dimensional Gaussian ellipsoid, calculate its opacity at the current view angle according to the total extinction coefficient S53. For each three-dimensional Gaussian ellipsoid, if its opacity at the current view angle is less than a threshold value for a certain number of consecutive frames, it is considered a redundant point ; S52, delete redundant points using soft constraint mechanism, add Loss as regularizer, introduce opacity-based loss term in training process Optimize three-dimensional Gaussian model, loss term The formula is as follows: ； wherein, is a regularization weight coefficient, used to adjust the influence degree of the opacity regularization term in the comprehensive loss function, and sparsely constrain the low opacity three-dimensional Gaussian ellipsoid.

6. The three-dimensional Gaussian spill-based volume cloud rendering method of claim 5, wherein, In step S5, a delayed deletion strategy is used to dynamically prune the low-contribution three-dimensional Gaussian ellipsoid, including: S53. During the model training process, the opacity of each three-dimensional Gaussian ellipsoid is recorded, and the average opacity in a plurality of consecutive training rounds is calculated; S54. When a certain three-dimensional Gaussian ellipsoid maintains extremely low opacity in a plurality of consecutive training rounds, it is indicated that the contribution of the three-dimensional Gaussian ellipsoid to the final pixel color is extremely small, and the three-dimensional Gaussian ellipsoid is marked as a pruning object; S55. A delayed deletion strategy is used for the pruning object, and batch pruning is performed every fixed number of steps. After a plurality of training rounds are iterated, the pruning is uniformly executed, and precise dynamic pruning of the three-dimensional Gaussian ellipsoid is realized.

7. A volume cloud rendering system based on three-dimensional Gaussian splatting for implementing the method of any one of claims 1 to 6, characterized in that Including: Preprocessing module, Gaussian representation module, light modeling module, three-dimensional rendering module, opacity pruning and density control module, loss optimization module and output module; The preprocessing module is configured to receive multi-view images of a volumetric cloud scene, perform feature matching and structure reconstruction using a motion structure recovery algorithm, and obtain a scene sparse point cloud and corresponding camera poses; The Gaussian representation module is configured to convert the sparse point cloud into an anisotropic three-dimensional Gaussian ellipsoid representation, each three-dimensional Gaussian ellipsoid including position parameters, scaling factors, rotation factors, opacity parameters, and spherical harmonic function coefficients; The light modeling module is configured to model the light transmission process of the volumetric cloud, decompose the light transmission process into a forward single scattering component and an internal multiple scattering component based on a radiative transfer equation, obtain a light modeling formula that can be embedded in a rendering process, and accurately describe the light characteristics of the volumetric cloud; The rendering module is configured to project the three-dimensional Gaussian ellipsoid onto a two-dimensional image plane, sort and synthesize the projection results based on depth information, calculate pixel colors in combination with the light modeling formula, and obtain volumetric cloud rendering results at different viewing angles; The opacity pruning and density control module is configured to prune and control the density of low-contribution three-dimensional Gaussian ellipsoids, determine the contribution of each three-dimensional Gaussian ellipsoid to the rendering result according to the opacity index of the three-dimensional Gaussian ellipsoid, and perform delayed deletion and adaptive density adjustment on low-contribution three-dimensional Gaussian ellipsoids; The loss optimization module is configured to construct a comprehensive loss function and optimize the three-dimensional Gaussian model, construct a comprehensive loss function including a basic reconstruction loss, a local perception loss based on a foreground mask, and an opacity sparsity regularization term, and perform iterative optimization based on input images to enhance the structural details of the volumetric cloud region; The output module is configured to output the optimized volumetric cloud rendering result and three-dimensional Gaussian model, and realize high-quality rendering of the volumetric cloud.

Images