Geometric enhancement method and 4D weather simulation and editing method based on environment reconstruction

Abstract

The invention provides a geometric enhancement method and a 4D weather simulation and editing method based on environment reconstruction, and the geometric enhancement method comprises the steps: carrying out the reasoning and optimization of an input sparse visual angle image through a DAM model, and obtaining an output pseudo-depth image; generating a pseudo normal graph under the corresponding view angle based on the pseudo depth map; performing Gaussian point cloud initialization and geometric feature optimization on the input sparse view angle image by using the SFM to obtain an output Gaussian point cloud; and combining the depth map and the normal map of the 3D GS rendering scene with the pseudo depth map and the pseudo normal map to calculate loss for supervised training, and optimizing the Gaussian point cloud. According to the method, DAM sparse visual angle depth reasoning, depth-normal graph joint optimization and SfM point cloud geometric preprocessing technical means are adopted, the technical effect of enhancing the authenticity of the geometric structure of the three-dimensional Gaussian scene is achieved, high-precision geometric support is provided for a subsequent weather editing module, and physical rationality rendering of the dynamic weather effect (such as rainfall / snowfall) is ensured.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a geometric enhancement method based on environment reconstruction, and a 4D weather simulation and editing method. Background Art

[0002] With the rapid development of digital twin technology, 3D scene modeling methods have undergone a significant transformation from manual modeling to automated reconstruction. Traditional industries rely on modeling software such as Blender and 3ds Max, as well as simulation platforms such as Carla and Airsim. They manually construct 3D models and then output them using rendering engines such as Eevee and Unity. However, these methods face inherent drawbacks such as high labor costs, long time consumption (single scene modeling can take weeks), and demanding hardware requirements. In recent years, automated 3D reconstruction technologies based on image and lidar data (such as SFM (Structure from Motion) and MVS (Multi-View Stereo)) have significantly reduced modeling costs. However, these technologies face two major bottlenecks: first, they cannot effectively restore the original weather effects at the time of filming (such as instantaneous meteorological conditions such as rain, snow, and fog); second, they lack the ability to dynamically adjust scene weather types, resulting in functional gaps in the construction of full-factor digital world models.

[0003] In the field of weather simulation technology, traditional methods mainly rely on image post-processing solutions: the early use of mask overlay method (such as Figure 1 and Figure 2 As shown in the figure (a), (b) shows the corresponding weather simulation effect achieved by superimposing masks of varying transparency). Transparency adjustment allows for quick editing of weather effects, but issues such as obvious artifacts and repeated elements exist. Advanced solutions (such as Digiweather) enhance realism through physical modeling (volumetric fog and droplet normals), but lack understanding of scene geometry and are still prone to spatial misalignment. The emergence of generative models (such as GANs and diffusion models) has driven technological innovation: WeatherDG combines language models to achieve text-driven weather conversion, patent CN202210989237.1 improves CycleGAN to generate multi-climate imagery, and Weather GAN suppresses content distortion through domain transfer. However, existing generative methods commonly suffer from scene structural distortion. Of course, there are also drawbacks such as spatial inconsistency in multi-perspective rendering (e.g., discontinuous rainfall trajectories across perspectives) and poor real-time performance (single-frame generation in the diffusion model takes >1 second).

[0004] Furthermore, within 3D reconstruction technology, implicit modeling (e.g., NeRF) and explicit modeling (e.g., 3D Gaussian splattering) face challenges with rendering efficiency and dynamic weather simulation, respectively: NeRF's fusion physics model can simulate dust and fog, but its rendering speed is extremely slow (>5 minutes per frame); while 3DGS supports real-time rendering, existing solutions (e.g., StyledStreets) can only simulate residual weather effects (wet roads, discolored walls) and are unable to present dynamic rainfall and snowfall. The core issues lie in three areas: the lack of efficient physical modeling for dynamic weather particle systems; insufficient mechanisms to ensure spatial consistency of weather effects from multiple perspectives; and the difficulty of achieving both high-precision physical simulation and real-time rendering performance (e.g., 4K@60fps). These technical bottlenecks severely restrict the application of digital twins in fields such as autonomous driving simulation and virtual reality, requiring high-fidelity reproduction of dynamic weather environments.

[0005] A search revealed a Chinese patent application numbered 202111626302.6, which discloses a weather rendering technology for use in games. This technology primarily explores the interaction between object models and snow. However, this technology has yet to address the challenge of ensuring the authenticity of geometric structures in rendering effects. Summary of the Invention

[0006] In response to one of the defects in the prior art, the purpose of this application is to provide a geometric enhancement method based on environment reconstruction, and a 4D weather simulation and editing method.

[0007] In a first aspect of the present application, a geometric enhancement method based on environment reconstruction is provided, comprising:

[0008] Use the DAM model to infer and optimize the input sparse view image to obtain the output pseudo depth map;

[0009] Based on the pseudo depth map, generating a pseudo normal map at a corresponding viewing angle;

[0010] Use SFM to initialize Gaussian point cloud and optimize geometric features of the input sparse view image to obtain the output Gaussian point cloud;

[0011] The depth map and normal map of the 3DGS rendered scene are combined with the pseudo depth map and the pseudo normal map to calculate the loss for supervised training and optimize the Gaussian point cloud.

[0012] Optionally, the use of the DAM model to infer and optimize the input sparse view image to obtain the output pseudo depth map includes:

[0013] Input a sparse view image to the DAM model and output a depth map of [0,1];

[0014] An offset is added to the portion of the depth map that exceeds a set threshold, and normalization is performed to obtain an output pseudo depth map.

[0015] Optionally, the step of using SFM to perform Gaussian point cloud initialization and geometric feature optimization on the input sparse view image to obtain an output Gaussian point cloud includes:

[0016] Using SFM to generate an initialized Gaussian point cloud for the pseudo depth map;

[0017] A spherical and evenly distributed point cloud mask is added outside the maximum radius of the initialized Gaussian point cloud to complete the geometric optimization of the entire SfM initialized Gaussian point cloud scene and obtain the output Gaussian point cloud.

[0018] A second aspect of the present application provides a 4D weather simulation and editing method based on environmental reconstruction, comprising:

[0019] The pseudo depth map, pseudo normal map, and Gaussian point cloud obtained by the geometric enhancement method of environment reconstruction are used for static and dynamic weather simulation rendering, as well as snow simulation rendering.

[0020] Optionally, performing static weather simulation rendering based on the pseudo depth map includes:

[0021] Based on the output pseudo depth map, pseudo normal map, Gaussian point cloud and original RGB image set, optimization training is performed to obtain the original Gaussian scene;

[0022] The original Gaussian scene is rasterized and rendered to obtain the depth D render ;

[0023] The depth D render Crop to [0, 1];

[0024] Based on the clipped depth D render , generate a blur mask;

[0025] Performing 3DGS framework rendering on the original Gaussian scene to obtain a rendered RGB image;

[0026] A static weather rendering image is generated based on the blur mask and the rendered RGB image.

[0027] Optionally, the depth D after clipping render , generate blur mask, including: α style =min(1,1-exp(-I style ×D render )), where I style is a pre-set hyperparameter that represents the overall blur intensity, i.e., the blur mask.

[0028] Optionally, generating a static weather rendering image based on the blurred mask and the rendered RGB image includes:

[0029] C render,blur =C blur ×α style +C render ×(1-α style )

[0030] C blur A customizable three-channel blur color ∈ [0, 255] 3 .

[0031] Optionally, performing static weather simulation rendering based on the pseudo depth map includes:

[0032] Based on the pseudo depth map, pseudo normal map, Gaussian point cloud and original RGB image set output by the DAM-based geometric enhancement method, optimization training is performed to obtain the original Gaussian scene; and a noisy Gaussian scene is simulated according to the physical laws of nature;

[0033] Perform rasterization rendering of the original Gaussian scene and the noisy Gaussian scene using a 3DGS framework to obtain corresponding rendered sub-noise images, rendered noise depth images, rendered depth images, and rendered RGB images;

[0034] Setting a noise color intensity threshold, performing color intensity filtering on the rendered sub-noise image, and obtaining a color intensity mask;

[0035] Comparing the scene depth between the rendered noise depth map and the rendered depth map to obtain a preliminary depth mask;

[0036] Setting a distance threshold, performing maximum depth filtering on the preliminary depth mask, and obtaining a depth mask;

[0037] Obtaining a stencil mask based on the color intensity mask and the depth mask;

[0038] Calculate the average brightness and brightness superposition coefficient of each pixel in the rendered RGB image;

[0039] Perform product and sum calculations on the brightness superposition coefficient, the mask, the rendered sub-noise image, and the rendered RGB image to obtain a single-frame weather rendering image;

[0040] A Gaussian mean offset is applied to the noise point position of the single-frame weather rendering image between connected frames to obtain a dynamic weather rendering image.

[0041] Optionally, perform snow simulation rendering based on the Gaussian point cloud, including:

[0042] Based on the pseudo depth map, pseudo normal map, Gaussian point cloud and original RGB image set output by the DAM-based geometric enhancement method, optimization training is performed to obtain the original Gaussian scene;

[0043] Define the unit vector in the shortest axis direction of each Gaussian point in the Gaussian point cloud as a normal vector, perform a dot product between the normal vector and the gravity vector estimated by the PCA component, and obtain a dot product result;

[0044] Setting a minimum initialization threshold, filtering the dot product result, and obtaining Gaussian points with upward normal vectors, i.e., initializing snow accumulation points;

[0045] Performing local fitting on the initialized snow accumulation points to obtain a number of local fitting planes of the initialized snow accumulation points;

[0046] Setting an angle threshold between the local fitting plane and the gravity vector, and filtering out local fitting planes exceeding the angle threshold;

[0047] Interpolate new Gaussian points on the filtered local fitting plane to densify the sparse initial snow points;

[0048] The opacity and the snow Gaussian color are set, and the original Gaussian scene and the densified Gaussian points are superimposed to obtain a snow scene rendering.

[0049] Optionally, performing local fitting on the initialization snow accumulation points to obtain a plurality of local fitting planes of the initialization snow accumulation points includes:

[0050] For the snow point p snow , the local plane is estimated using k nearest neighbor points, and the radius of the generated fitting plane is where R n is the k nearest neighbor point to p snow The set of distances, σ n R n The standard deviation of , median represents the median;

[0051] The interpolation of new Gaussian points on the filtered local fitting plane to densify the sparse initial snow points includes:

[0052] The local fitting plane is filled with new Gaussian point positions generated based on uniform random probability.

[0053] The geometric enhancement method based on environment reconstruction provided in this application adopts DepthAnything Model (DAM) sparse perspective depth reasoning, depth-normal map joint optimization and SfM point cloud geometry preprocessing technology to achieve the technical effect of enhancing the authenticity of the geometric structure of three-dimensional Gaussian scenes, and provides high-precision geometric support for subsequent weather editing modules, ensuring the physically reasonable rendering of dynamic weather effects (such as rain / snowfall).

[0054] Other technical effects brought about by the additional features will be further explained in the corresponding embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0056] Figure 1 The images of weather simulation for mountains and rivers in the prior art, where (a) is the original image and (b) is the effect image after simulation;

[0057] Figure 2 The images of the weather simulation of the sea in the prior art are shown, where (a) is the original image and (b) is the effect image after simulation.

[0058] Figure 3 is a flow chart of a DAM-based geometric enhancement method according to an exemplary embodiment;

[0059] Figure 4 is a flowchart of a static weather simulation rendering method according to an exemplary embodiment;

[0060] Figure 5 is a flow chart of a dynamic weather simulation rendering method according to an exemplary embodiment;

[0061] Figure 6 The figure is a flowchart of a snow simulation rendering method according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] The present application is described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present application, but are not intended to limit the present application in any form. It should be noted that, without departing from the concept of the present application, a number of variations and improvements may be made by those skilled in the art, and these all fall within the scope of protection of the present application. Parts not described in detail in the following examples may be implemented using existing technologies.

[0063] Glossary:

[0064] Deep Arbitrary Model (DAM): A model that takes an RGB image as input and outputs a depth map between 0 and 1.

[0065] Pseudo depth: DAM refers to the depth value inferred by the model, which may differ from the true depth of the actual scene; the true depth is the more accurate scene depth information obtained through sensors such as lidar.

[0066] Pseudo normal true value: It is a normal map obtained by gradient calculation based on pseudo depth. This normal map may deviate from the actual normal map.

[0067] In the prior art, the authenticity of the geometric structure of the reconstructed Gaussian scene needs to be improved. Based on the above problems, the embodiment of the present application provides a geometric enhancement method based on DAM to solve the above problems.

[0068] Reference Figure 3 As shown, in one embodiment of the present application, a DAM-based geometric enhancement method can adopt the following steps:

[0069] Step 1: Use the DAM model to infer and optimize the input sparse view image to obtain the output pseudo depth map;

[0070] Step 2: Generate a pseudo normal map at the corresponding viewing angle based on the pseudo depth map;

[0071] Step 3: Use SFM to initialize the Gaussian point cloud and optimize the geometric features of the input sparse view image to obtain the output Gaussian point cloud;

[0072] Step 4: Combine the depth map and normal map of the 3DGS rendered scene with the pseudo-depth map and pseudo-normal map to calculate the loss for supervised training and optimize the Gaussian point cloud.

[0073] Specifically, the DAM model refers to the Depth Anything Model, whose input is an RGB image and output is a depth map in the range of [0-1]. The above embodiment further optimizes the depth map output by the DAM, and the output pseudo-depth map is the optimized depth map. SfM refers to Structure from Motion, which is a traditional 3D reconstruction algorithm used to simply reconstruct a set of scene point clouds from a sparse image set. The loss in step 3 refers to the Hard-Soft depth regularization loss of the depth map and pseudo-depth map of the 3DGS rendered scene, and the normal map and pseudo-normal map of the 3DGS rendered scene combined with the L2 regularization loss.

[0074] The above-mentioned embodiments of the present application enhance the authenticity of the geometric structure of the reconstructed Gaussian scene by acquiring a pseudo-depth map and a Gaussian point cloud, and can render correct geometric information for use by the weather editing module, thereby solving the problem of scene structure distortion in the prior art.

[0075] In order to enable the pseudo depth map output by the above embodiment to more accurately simulate the geometric structure, in some specific implementations of the present application, in step 1 above, the DAM model is used to infer and optimize the input sparse view image to obtain the output pseudo depth map, and the following steps can be used:

[0076] S101, load the DAM model with pre-trained weights to infer the corresponding depth map for the input sparse perspective image.

[0077] Here, the depth map output by the DAM is a surjection of [0,1]. This depth map may contain sky information, specifically the sky pixels that appear in the image. Because their depth values ​​are theoretically set to positive infinity, the output depth map requires appropriate processing to further ensure geometric correctness. The portion exceeding a threshold of 0.95 can be defined as sky information. This can be determined manually or through semantic segmentation models to determine the presence of sky areas.

[0078] S102: Increase the depth interval between the sky area and the environment to ensure geometric correctness.

[0079] Specifically, the environment here refers to the pixels in the image other than the sky area, such as buildings, ground, etc. The increase in the interval is mainly achieved by adding a post-processing part to the algorithm: an additional offset is added to the parts of the output depth map with a depth greater than 0.95 (i.e., sky-related pixels), and then the final depth map is normalized to [0, 1].

[0080] In the above embodiment of the present application, by increasing the offset, there will be a difference between the final minimum depth value of the sky area and the maximum depth value of the environment area, so as to better simulate the geometric structure.

[0081] Based on the pseudo depth map obtained in the above embodiment, in some specific implementations of the present application, a corresponding pseudo normal map can be obtained by applying a Sobel operator to the pseudo depth map for smoothing derivation, which helps to reduce noise interference in the depth map and alleviate possible normal errors.

[0082] The Gaussian splattering framework generally requires an SfM-initialized point cloud to help accelerate the training convergence of the 3DGS model. However, for scenes with sky information, the SfM-initialized point cloud cannot effectively generate sky points. In order to optimize the geometric features of the sky part, in some specific embodiments of this application, step 3 uses SFM to initialize and optimize the Gaussian point cloud of the input sparse view image to obtain the output Gaussian point cloud. The following process can be used:

[0083] S301, using SfM to initialize the Gaussian point cloud of the input sparse view image;

[0084] S302: Add a spherical and evenly distributed point cloud mask outside the maximum radius of the entire initialized Gaussian point cloud.

[0085] The maximum radius here refers to the maximum radial distance of the 3D point cloud scene reconstructed using SfM technology relative to the coordinate origin. Typically, this origin is determined by the structure-from-motion algorithm optimization process.

[0086] The above embodiments of the present application can help initialize the sky point to optimize the depth of the sky as much as possible and reduce the possibility of floating points.

[0087] In order to enable the above-mentioned geometric enhancement method to effectively improve the authenticity of the geometric structure of the reconstructed Gaussian scene, in some specific implementations of the present application, step 4 may adopt the following process:

[0088] S401, combining the depth map of the 3DGS rendered scene and the pseudo depth map produced, calculates the Hard-Soft depth regularization loss to supervise the training process. Specifically,

[0089] The total depth loss is L depth =L hard +L soft

[0090] L hard With L soft It manifests as:

[0091]

[0092] Among them, GN means that the depth map is based on global regularization, and LN means that the depth map is based on local regularization. ref Depth map representing Gaussian splatter rendering. D DAM Represents the pseudo depth ground truth generated by DAM.

[0093] It should be noted that L hard With L soft Because the way of rendering depth is different, it is not the same. During the process of supervised training using depth loss, the structure of Gaussian point cloud is also optimized.

[0094] S402, when 3DGS renders the normal map, it will weightedly superimpose the unit vectors in the direction of the shortest axis of each Gaussian to output the normal vector N of each pixel, and calculate the L2 regularization loss L between the pseudo normal map normal .

[0095] L normal =||NN pseudo ||2,

[0096] Where N is the rendered normal map, N pseudo is the true value of the pseudo normal map.

[0097] In the above embodiment of the present application, the model corresponding to the geometric enhancement method is further supervised and trained by calculating the loss. By inputting a sparse perspective image, a geometrically enhanced pseudo depth map and Gaussian point cloud are obtained for subsequent weather simulation and editing. During training, the above embodiment supervises the depth information and normal information at different perspectives. Therefore, during the rendering process, the depth information obtained from different perspectives is based on the Gaussian point cloud and can remain consistent with each other.

[0098] Based on the same technical concept, in one embodiment of the present application, a 4D weather simulation and editing method based on environmental reconstruction includes:

[0099] Based on the pseudo depth map, pseudo normal map, and Gaussian point cloud obtained by the geometric enhancement method based on environment reconstruction provided by the above embodiment, static and dynamic weather simulation rendering is performed;

[0100] Snow simulation rendering is performed based on the Gaussian point cloud obtained by the geometric enhancement method based on environment reconstruction provided in the above embodiment.

[0101] In order to simulate static weather phenomena such as fog, haze, and dust, in one embodiment of the present application, Figure 4 As shown, a static weather simulation rendering method includes the following steps:

[0102] S2001, performing optimization training based on the pseudo depth map, pseudo normal map, SfM-initialized point cloud (Gaussian point cloud), and original RGB image set output by the DAM-based geometric enhancement method in the above embodiment to obtain an original Gaussian scene;

[0103] S2002, rasterize and render the original Gaussian scene to obtain the depth D render ;

[0104] S2003, the depth D render Crop to [0, 1];

[0105] S2004, based on the clipped depth D render , generate a blur mask;

[0106] Specifically, the calculation method of the blur mask is α style =min(1,1-exp(-I style ×D render )), where I style It is a pre-set hyperparameter, which indicates the overall haziness intensity. style is the depth haze coefficient, that is, the blur mask. style The natural exponential relationship with depth is the field of view attenuation relationship obtained by integration, assuming that the particles are uniformly distributed in the air.

[0107] S2005, performing 3DGS framework rendering on the original Gaussian scene to obtain a rendered RGB image;

[0108] S2006, generating a static weather rendering image based on the fuzzy mask and the rendered RGB image.

[0109] C render,blur =C blur ×α style +C render ×(1-α style )

[0110] Among them, C blur A customizable three-channel blur color ∈ [0, 255] 3 , for example, it can be set to gray in fog simulation. style Indicates the depth blur coefficient. This achieves high-quality rendering effects.

[0111] In the above embodiment of the present application, different depth blur coefficients are set, and C is customized. blur To change the hazy color, you can apply different hazy effects to the rendering in real time, and successfully simulate static weather phenomena such as fog, haze, and dust, thereby achieving high-quality rendering effects.

[0112] Although existing reconstruction algorithms can model dynamic objects of normal scale, they are difficult to process high-frequency noise such as raindrops. Therefore, one embodiment of the present application adds a new Gaussian to the original Gaussian scene to synthesize this type of weather effect. Gaussian modeling of rain and snow effects is achieved, and the corresponding Gaussian covariance is adjusted according to its actual shape distribution to accurately model the shape. Figure 5 As shown, a dynamic weather simulation rendering method includes:

[0113] S3001, based on the pseudo depth map, pseudo normal map, SfM-initialized point cloud (Gaussian point cloud) and original RGB image set output by the DAM-based geometric enhancement method of the above embodiment, optimization training is performed to obtain an original Gaussian scene; a noisy Gaussian scene is simulated according to the physical laws of nature;

[0114] Specifically, simulating a noisy Gaussian scene involves geometric modeling of the noisy Gaussian. For example, this can be achieved by adjusting the three-dimensional scale and rotation of the Gaussian. The scale and rotation matrices form the covariance matrix, so this can be considered as simulating a noisy Gaussian scene by adjusting the Gaussian's covariance.

[0115] S3002, performing rasterization rendering of the original Gaussian scene and the noisy Gaussian scene using a 3DGS framework, respectively, to obtain corresponding rendered sub-noise images, rendered noise depth images, rendered depth images, and rendered RGB images;

[0116] S3003, setting a noise color intensity threshold, performing color intensity filtering on the rendered sub-noise image, and obtaining a color intensity mask;

[0117] S3004, performing scene depth comparison on the rendered noise depth map and the rendered depth map to obtain a preliminary depth mask;

[0118] S3005, setting a distance threshold, performing maximum depth filtering on the preliminary depth mask, and obtaining a depth mask;

[0119] S3006, obtaining a matte mask based on the color intensity mask and the depth mask;

[0120] S3007, calculating the average brightness and brightness superposition coefficient of each pixel in the rendered RGB image;

[0121] S3008, performing product and sum calculations on the brightness overlay coefficient, the mask, the rendered sub-noise image, and the rendered RGB image to obtain a single-frame weather rendering image;

[0122] S3009, applying a Gaussian mean offset to the noise point positions of the single-frame weather rendering image between the connected frames to obtain a dynamic weather rendering image.

[0123] Specifically, the Gaussian mean offset is manually set. An appropriate offset is chosen based on the overall scale of the scene, and further fine-tuned by examining the final effect of the synthesized video.

[0124] The above-mentioned embodiments of the present application achieve real-time rendering efficiency while optimizing the simulation authenticity of falling rain and falling snow effects.

[0125] In order to make the noise Gaussian scene more consistent with the physical laws of nature, such as the hexagonal formation law of snowflakes and the stretching effect of raindrops in camera lens exposure, in some specific embodiments of the present application, in step S3001, the simulation process of the noise Gaussian scene simulates each raindrop and snowflake by a stretched Gaussian and three Gaussians superimposed at different angles. In order to prevent the noise Gaussian color from being incorrectly superimposed with the scene color during the rendering process, the noise can be rendered separately into a mask image for subsequent superposition.

[0126] In order to obtain a color intensity mask, in some specific embodiments of the present application, S3003 sets a noise color intensity threshold C thres , perform color intensity filtering on the rendered sub-noise map, retain the noise points that are greater than the noise color intensity threshold, and obtain the color intensity mask, specifically: C noise >C thres , C noise is the color intensity of the noise.

[0127] In order to generate an unobstructed noise mask, in some specific embodiments of the present application, S3004 can be used to generate an unobstructed noise mask by rendering the depth map D render and rendering noise depth map D noise By comparing and determining the occluded information of the noise point, a mask of the unoccluded noise point can be generated.

[0128] Specifically, the pixel area where the noise depth is smaller than the scene depth is set to 1, and the other areas are set to 0 to obtain a preliminary depth mask m occ :

[0129] m occ =(D noise <D render )

[0130] Of course, if there are several noise Gaussian points of different depths projected onto the same pixel, when the distant noise points are judged to be occluded, the mask will filter out the nearby Gaussian points together, forming a central hole phenomenon in the image. To avoid the central hole phenomenon, some specific embodiments of the present application can split the noise Gaussian scene into small scenes with a fixed number of noise points to avoid such problems.

[0131] On the basis of obtaining the preliminary depth mask, further maximum depth filtering is performed. In some specific implementations of the present application, in S3005, half of the maximum depth of the rendered depth map can be used as a distance threshold to perform maximum depth filtering on the preliminary depth mask to obtain a depth mask, specifically: (D noise <D render )∩(D noise <D thres )D render To render the depth map, D noise To render the noise depth map, D thres is the distance threshold.

[0132] The above-mentioned embodiments of the present application can effectively clarify noise rendering and enhance visual performance.

[0133] In order to obtain a more detailed mask, in some specific implementations of the present application, S3006, based on the color intensity mask and the depth mask, a mask mask is obtained, specifically: m=(D noise <D render )∩(D noise <D thres )∩(C noise >C thres ).

[0134] The mask m obtained in the embodiment of the present application can accurately combine the three-dimensional geometric characteristics of the scene to filter and render high-fidelity dynamic weather effects that conform to real-world logic.

[0135] In order to restore the brightness of the original Gaussian scene in the final rendering result, in some specific implementations of the present application, S3007 calculates the average brightness and brightness superposition coefficient of each pixel in the rendered RGB image.

[0136] Specifically, the calculation method is to compare the brightness L of each pixel p in the rendering image p and the average sky brightness L sky :

[0137]

[0138]

[0139] Among them, n p is the total number of sky pixels;

[0140] The brightness superposition coefficient f of each pixel p l for:

[0141] f ,p =exp(min(max((L sky -L p ),t max ),t min ))-1

[0142] t max With t min are manually set parameters, t min It is usually set to a negative value such as -0.1, which makes the brighter areas be assigned a negative f l,p , the noise will be the dimming effect of the backlight when finally superimposed. l,p Finally, the superposition operation of two layers (the layer rendered by the noise Gaussian scene and the layer rendered by the original Gaussian scene) can be achieved.

[0143] In order to simulate the effect of snowfall and rain, in some specific embodiments of the present application, S3008, the simulated rain and snow RGB image C fall It can be obtained by superimposing a total of k sub-noise layers C noise,i , i∈[0,k] rendering results are obtained, specifically:

[0144]

[0145] Among them, f l,i is the brightness superposition coefficient of the i-th noise layer, C render Indicates rendering of RGB images.

[0146] It should be noted that each sub-noise layer represents a rendered sub-noise map. The single-frame weather rendering simulation is based on the original scene RGB map. Figure 3 The process cumulatively superimposes k different noise layers.

[0147] In order to obtain dynamic effects, in a specific embodiment of the present application, S3009, by accessing and modifying the distribution mean property of the noise Gaussian at different timestamps (continuous frames of the picture), these dynamic weather elements (raindrops, snowflakes) can present consistent dynamic changes when rendering the original scene, simulating a real landing effect.

[0148] Based on the same technical concept, in one embodiment of the present application, a snow simulation rendering method, such as Figure 6 As shown, the following steps can be taken:

[0149] S4001, based on the pseudo depth map, pseudo normal map, Gaussian point cloud and original RGB image set output by the geometric enhancement method, optimization training is performed to obtain the original Gaussian scene;

[0150] S4002, defining the unit vector in the shortest axis direction of each Gaussian point in the Gaussian point cloud output by the DAM-based geometric enhancement method as a normal vector, performing a dot product between the normal vector and the negative value of the gravity vector estimated by the PCA component, and obtaining a dot product result;

[0151] S4003, setting a minimum initialization threshold, filtering the dot product result, and obtaining Gaussian points with upward normal vectors, i.e., initializing snow accumulation points;

[0152] S4004, performing local fitting on the initialized snow accumulation points to obtain local fitting planes of several initialized snow accumulation points;

[0153] S4005, setting an angle threshold between the local fitting plane and the gravity vector, and filtering out local fitting planes that exceed the angle threshold;

[0154] S4006, interpolating new Gaussian points on the filtered local fitting plane to densify the sparse initial snow points;

[0155] S4007, setting the opacity and snow Gaussian color, superimposing the original Gaussian scene and the densified Gaussian points to obtain a snow scene rendering.

[0156] The embodiments described in this application can ensure rendering speed while maintaining the authenticity of the snow rendering, a balance that other methods cannot achieve.

[0157] Since the initial snow accumulation is relatively sparse, the initial snow accumulation needs to be densified. In some specific implementations of the present application, first, for the initial snow accumulation point p snow , the local plane is estimated using k nearest neighbor points, and the radius of the generated local fitting plane is where Rn is the nearest neighbor point to p snow The set of distances, σ n R n Then, we set a threshold for the angle between the local fitting plane and the gravity vector and filter out local fitting planes with angles greater than the threshold. Finally, we use new Gaussian point positions generated based on uniform random probability to fill in the local fitting plane.

[0158] The fitting plane radius of the above embodiment of the present application can automatically adapt to the local point density. In addition, by setting the angle threshold between the local fitting plane and the gravity vector, the erroneous plane interpolation situation can be avoided.

[0159] It should be noted that the above static, dynamic, and snow simulation rendering methods can all control the weather effects by adjusting a series of parameters, including the degree of haze blur, the size, color, falling speed, and density of raindrops and snowflakes, etc. For example, adjusting the haze intensity I style , set t max With t min , the initialization threshold for comparing the normal vector and the gravity dot product, etc. These parameters can be flexibly adjusted externally in the framework.

[0160] The methods of the above embodiments of the present application are not only convenient for real-scene modeling and weather simulation editing, but also maintain real-time rendering performance and support intuitive rendering result output. These method frameworks are hardware-friendly and support execution on most consumer-grade GPUs. The different perspectives output by the generative model are often inconsistent in space, and the present application supervises the depth information and normal information at different perspectives during 3DGS training. Therefore, during the rendering process, the depth information obtained from different perspectives is based on the Gaussian point cloud and can remain consistent with each other. It can render real weather scene videos under continuous camera motion, and the weather effects also have temporal consistency.

[0161] The preferred features of the above embodiments can be used alone in any embodiment, or in any combination without conflict. In addition, parts not described in detail in the embodiments can be implemented using existing technologies.

[0162] The above describes some specific embodiments of the present application. It should be understood that the present application is not limited to the specific embodiments described above, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the substantive content of the present application. The above preferred features may be used in any combination as long as they do not conflict with each other.

Claims

1. A geometric enhancement method based on environment reconstruction, characterized in that: include: Use the DAM model to infer and optimize the input sparse view image to obtain the output pseudo depth map; Based on the pseudo depth map, generating a pseudo normal map at a corresponding viewing angle; Use SFM to initialize Gaussian point cloud and optimize geometric features of the input sparse view image to obtain the output Gaussian point cloud; The depth map and normal map of the 3DGS rendered scene are combined with the pseudo depth map and the pseudo normal map to calculate the loss for supervised training and optimize the Gaussian point cloud.

2. A geometric enhancement method based on environment reconstruction according to claim 1, characterized in that: The method of using the DAM model to infer and optimize the input sparse perspective image to obtain the output pseudo depth map includes: Input a sparse view image to the DAM model and output a depth map of [0,1]; An offset is added to the portion of the depth map that exceeds a set threshold, and normalization is performed to obtain an output pseudo depth map.

3. The geometric enhancement method based on environment reconstruction according to claim 1, characterized in that: The method of using SFM to perform Gaussian point cloud initialization and geometric feature optimization on the input sparse view image to obtain an output Gaussian point cloud includes: Using SFM to generate an initialized Gaussian point cloud for the pseudo depth map; A spherical and evenly distributed point cloud mask is added outside the maximum radius of the initialized Gaussian point cloud to complete the geometric optimization of the entire SfM initialized Gaussian point cloud scene and obtain the output Gaussian point cloud.

4. A 4D weather simulation and editing method based on environmental reconstruction, characterized in that: include: Based on the pseudo depth map, pseudo normal map and Gaussian point cloud obtained by the geometric enhancement method based on environment reconstruction described in claim 1, static and dynamic weather simulation rendering and snow simulation rendering are performed.

5. The 4D weather simulation and editing method based on environmental reconstruction according to claim 4, characterized in that: Based on the pseudo depth map, static weather simulation rendering is performed, including: Performing optimization training based on the pseudo depth map, pseudo normal map, Gaussian point cloud and original RGB image set to obtain an original Gaussian scene; The original Gaussian scene is rasterized and rendered to obtain the depth D render ; The depth D render Crop to [0, 1]; Based on the clipped depth D render , generate a blur mask; Performing 3DGS framework rendering on the original Gaussian scene to obtain a rendered RGB image; A static weather rendering image is generated based on the blur mask and the rendered RGB image.

6. The 4D weather simulation and editing method based on environmental reconstruction according to claim 5, characterized in that: The depth D after clipping render , generate blur mask, including: α style =min(1,1-exp(-I style ×D render )), where I style is a pre-set hyperparameter that represents the overall blur intensity, i.e., the blur mask.

7. The 4D weather simulation and editing method based on environmental reconstruction according to claim 6, characterized in that: The generating a static weather rendering image based on the blurred mask and the rendered RGB image includes: C render,blur =C blur ×α style +C render ×(1-α style ) C blur A customizable three-channel blur color ∈ [0, 255] 3 .

8. The 4D weather simulation and editing method based on environmental reconstruction according to claim 4, characterized in that: Based on the pseudo depth map, dynamic weather simulation rendering is performed, including: Based on the pseudo depth map, pseudo normal map, Gaussian point cloud and original RGB image set output by the DAM-based geometric enhancement method according to claim 1, optimization training is performed to obtain the original Gaussian scene; a noisy Gaussian scene is simulated according to the physical laws of nature; Perform rasterization rendering of the original Gaussian scene and the noisy Gaussian scene using a 3DGS framework to obtain corresponding rendered sub-noise images, rendered noise depth images, rendered depth images, and rendered RGB images; Setting a noise color intensity threshold, performing color intensity filtering on the rendered sub-noise image, and obtaining a color intensity mask; Comparing the scene depth between the rendered noise depth map and the rendered depth map to obtain a preliminary depth mask; Setting a distance threshold, performing maximum depth filtering on the preliminary depth mask, and obtaining a depth mask; Obtaining a stencil mask based on the color intensity mask and the depth mask; Calculate the average brightness and brightness superposition coefficient of each pixel in the rendered RGB image; Perform product and sum calculations on the brightness superposition coefficient, the mask, the rendered sub-noise image, and the rendered RGB image to obtain a single-frame weather rendering image; A Gaussian mean offset is applied to the noise point position of the single-frame weather rendering image between connected frames to obtain a dynamic weather rendering image.

9. The 4D weather simulation and editing method based on environmental reconstruction according to claim 4, characterized in that: Based on Gaussian point cloud, snow simulation rendering is performed, including: Based on the pseudo depth map, pseudo normal map, Gaussian point cloud and original RGB image set output by the DAM-based geometric enhancement method according to claim 1, optimization training is performed to obtain an original Gaussian scene; Define the unit vector in the shortest axis direction of each Gaussian point in the Gaussian point cloud as a normal vector, perform a dot product between the normal vector and the gravity vector estimated by the PCA component, and obtain a dot product result; Setting a minimum initialization threshold, filtering the dot product result, and obtaining Gaussian points with upward normal vectors, i.e., initializing snow accumulation points; Performing local fitting on the initialized snow accumulation points to obtain a number of local fitting planes of the initialized snow accumulation points; Setting an angle threshold between the local fitting plane and the gravity vector, and filtering out local fitting planes exceeding the angle threshold; Interpolate new Gaussian points on the filtered local fitting plane to densify the sparse initial snow points; The opacity and the snow Gaussian color are set, and the original Gaussian scene and the densified Gaussian points are superimposed to obtain a snow scene rendering.

10. The 4D weather simulation and editing method based on environmental reconstruction according to claim 9, characterized in that: The locally fitting the initialized snow accumulation points to obtain a plurality of local fitting planes of the initialized snow accumulation points includes: For the snow point p snow , the local plane is estimated using k nearest neighbor points, and the radius of the generated fitting plane is where R n is the k nearest neighbor point to p snow The set of distances, σ n R n The standard deviation of , median represents the median; The interpolation of new Gaussian points on the filtered local fitting plane to densify the sparse initial snow points includes: The local fitting plane is filled with new Gaussian point positions generated based on uniform random probability.

Images