Three-dimensional Gaussian splash modeling method and system based on scene prior in complex weather

Abstract

The invention discloses a three-dimensional Gaussian splash modeling method and system based on scene prior in complex weather. The method comprises the following steps of: 1, acquiring a group of low-illumination original image sequences, corresponding camera parameters and a pose relation between viewpoints; 2, removing rainfall particle shielding in the original image sequence to obtain a clear image sequence; obtaining a corresponding mask by adopting a segmentation large model; 3, extracting scene geometric priori information with illumination invariance, wherein the scene geometric priori information comprises a depth priori image and an edge gradient priori image; 4, constructing a three-dimensional Gaussian splashing model containing edge gradient, depth and illumination attributes, and rendering a prediction result by using a microrasterization method; and constructing a total loss function in combination with the mask, and optimizing the three-dimensional Gaussian splash model by optimizing the total loss function. According to the method, all-around elimination of rainfall particle and lens rainwater interference is realized, a robust scene geometry priori acquisition mechanism is innovatively constructed, and a multi-attribute constrained three-dimensional Gaussian splash model is established.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D reconstruction technology, specifically involving a 3D Gaussian splash modeling method and system based on scene geometric priors under complex weather conditions such as low light, rain, and snow, to achieve the synthesis of new views from any viewpoint with good lighting and no rain or snow interference. Background Technology

[0002] Three-dimensional scene reconstruction technology has important application value in fields such as autonomous driving, virtual reality and digital twins. In recent years, 3D Gaussian Splatting technology has become one of the mainstream technologies in this field due to its high-fidelity rendering effect and real-time performance. However, existing 3D Gaussian Splatting modeling methods usually assume that the input image sequence is clear and the lighting conditions are good. However, in practical applications, especially under complex weather conditions such as rain and snow, this technology often faces two key challenges: (1) Falling raindrops, suspended snow particles, and static occlusions such as water droplets attached to the lens surface are easily misjudged as inherent components of the scene, thus introducing obvious artifacts in the modeling process. (2) Complex weather is often accompanied by extreme lighting conditions such as low light or overexposure, which leads to sparse point clouds obtained by the Structure-from-Motion (SfM) method, and the images have serious noise and color distortion, which restricts the ability of 3D Gaussian Splatting technology to model accurate geometric structures.

[0003] Therefore, existing 3D scene reconstruction technologies often struggle to achieve high-quality reconstruction results under low light and severe weather conditions (such as rain and snow). On one hand, low light environments significantly enhance image noise, cause severe color distortion, and result in a large loss of texture information, leading to phenomena such as excessively smoothed scene geometry or structural holes in the modeling results. On the other hand, precipitation particles (such as raindrops and snowflakes) introduced by severe weather, as well as static occlusions (such as water droplets) attached to the lens surface, severely interfere with the extraction and matching of image features, resulting in numerous artifacts in the reconstructed scene. To address these issues, this invention provides a 3D Gaussian splash modeling method and system based on scene priors under complex weather conditions, enabling 3D Gaussian splash modeling under complex weather conditions such as rain and snow, and outputting a new viewpoint view with good lighting and free from interference from precipitation particles such as rain and snow. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A method and system for modeling 3D Gaussian splashes under complex weather conditions based on scene priors, including the following steps: Step 1: Acquire raw image sequence: Under complex weather conditions such as rain and snow, acquire a set of raw low-light images of the target scene. The motion recovery structure technique is used to obtain camera parameters and pose relationships between viewpoints.

[0005] Step 2: Preprocessing of the original image sequence: Removing the original image sequence using a fine-tuned diffusion model (Stable Diffusion). Raindrops, snowflakes, and other precipitation particles obscure the image, resulting in a clear image sequence. This provides high-quality input data for subsequent 3D Gaussian splash scene modeling; and employs a segmentation anything model to identify clear image sequences. Transparent or semi-transparent static obstructions such as water droplets attached to the surface of the lens generate a corresponding mask M.

[0006] Step 3: Extract scene geometric prior information with illumination invariance: Using a monocular depth estimation model and a pre-trained DexiNed edge detection operator, extract prior information from a clear image sequence. In this process, illumination-invariant depth information and scene geometric prior information of geometric structure are extracted to generate a scene geometric prior map unaffected by low illumination. The scene geometric prior map includes a depth prior map. and edge gradient prior map This provides stable geometric feature constraints for the subsequent three-dimensional Gaussian splash model G.

[0007] Step 4: Multi-attribute 3D Gaussian splash modeling based on scene geometric prior information: based on clear image sequences and its corresponding edge gradient prior map S and depth prior map A 3D Gaussian splash model G, incorporating edge gradients, depth, and lighting properties, is constructed. The predicted edge gradient map is then rendered using a differentiable rasterization method. Predicted depth map Predicting low-light maps And predicted residual illumination map Combine the mask M to construct the total loss function. Including reconstruction losses Marginal gradient consistency loss Deep consistency loss and exposure control loss By optimizing the total loss function Optimize the three-dimensional Gaussian splash model G.

[0008] Furthermore, the present invention also provides a three-dimensional Gaussian splash modeling system based on scene priors under complex weather conditions, including: a data acquisition module, a data preprocessing module, a scene geometric prior information extraction module, and a multi-attribute three-dimensional Gaussian splash modeling module.

[0009] The data acquisition module acquires raw image sequences, obtaining a set of low-light raw image sequences of the target scene under complex weather conditions such as rain and snow. .

[0010] The data preprocessing module preprocesses the original image sequence and removes impurities from the original image sequence using a fine-tuned diffusion model (StableDiffusion). Raindrops, snowflakes, and other precipitation particles obscure the image, resulting in a clear image sequence. This provides high-quality input data for subsequent 3D Gaussian splash scene modeling; and employs SegmentAnything to identify clear image sequences. Transparent or semi-transparent static obstructions such as water droplets attached to the surface of the lens generate a corresponding mask M.

[0011] The scene geometry prior information extraction module extracts scene geometry prior information with illumination invariance, and uses a monocular depth estimation model and a pre-trained DexiNed edge detection operator to extract prior information from a clear image sequence. In this process, illumination-invariant depth information and scene geometric prior information of geometric structure are extracted to generate a scene geometric prior map unaffected by low illumination. The scene geometric prior map includes a depth prior map. and edge gradient prior map This provides stable geometric feature constraints for the subsequent three-dimensional Gaussian splash model G.

[0012] The aforementioned multi-attribute 3D Gaussian splash modeling module, based on scene geometric prior information, performs multi-attribute 3D Gaussian splash modeling according to clear image sequences. and its corresponding edge gradient prior map S and depth prior map A 3D Gaussian splash model G, incorporating edge gradients, depth, and lighting properties, is constructed. The predicted edge gradient map is then rendered using a differentiable rasterization method. Predicted depth map Predicting low-light maps And predicted residual illumination map Combine the mask M to construct the total loss function. Including reconstruction losses Marginal gradient consistency loss Deep consistency loss and exposure control loss By optimizing the total loss function Optimize the three-dimensional Gaussian splash model G.

[0013] The beneficial effects of this invention are as follows: This invention overcomes the challenge of high-precision 3D reconstruction under low light and complex weather conditions: It is the first in China and abroad to propose and implement a high-quality 3D Gaussian splash modeling method and system for weather environments such as rain, snow and low light.

[0014] This invention achieves comprehensive elimination of precipitation particles and lens splatter interference: it combines a fine-tuned Stable Diffusion model with a large segmentation model (SAM). The former explicitly learns the physical distribution characteristics of precipitation particles in the image, accurately removing dynamic precipitation particles such as rain and snow, and restoring texture; the latter effectively identifies static obstructions such as water droplets adhering to the lens surface. This multi-layered preprocessing approach ensures that the data entering the 3D Gaussian splash optimization stage has an extremely high signal-to-noise ratio.

[0015] This invention innovatively constructs a robust scene geometry prior acquisition mechanism: Utilizing a monocular depth estimation model and a pre-trained DexiNed edge detection operator, depth and edge gradient information are extracted from a clear image sequence restored by a diffusion model. The scene geometry prior map generated in this way possesses illumination invariance, providing stable geometric constraints for the 3DGS model even in low-light environments.

[0016] A multi-attribute constrained 3D Gaussian splash model was established: In the optimization process of the 3D Gaussian splash model, this invention not only utilizes traditional color consistency loss, but also introduces edge gradient consistency loss, depth consistency loss, and exposure control loss. The predicted edge gradients and depth attributes are aligned with the extracted geometric priors and restored to normal illumination. This multi-attribute collaborative optimization mechanism ensures the accuracy of edge gradients and depth in the reconstructed scene, and can reconstruct scene images under normal illumination. Attached Figure Description

[0017] Figure 1 This is a flowchart of a three-dimensional Gaussian splash modeling method and system based on scene priors under complex weather conditions, as described in an embodiment of the present invention.

[0018] Figure 2 This is a framework diagram of a three-dimensional Gaussian splash modeling method and system based on scene priors under complex weather conditions, as described in an embodiment of the present invention.

[0019] Figure 3 These are the experimental results of the embodiments.

[0020] Figure 4 This is a graph showing the results of comparing the example with other experiments. Detailed Implementation

[0021] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and preferred embodiments.

[0022] like Figure 1 As shown, this embodiment provides a method and system for three-dimensional Gaussian splash modeling based on scene priors under complex weather conditions. Figure 2 The details of the method are shown, which includes the following steps: Step 1: This step aims to acquire a sequence of original images of the target scene from multiple perspectives under complex weather conditions such as low light, rain, and snow. .

[0023] Specifically, high-resolution images (e.g., 1200×800 pixels or higher) from multiple (e.g., 30 or more) different viewpoints are acquired. These images suffer from issues such as low light and complex weather conditions like rain and snow, including dimness, color distortion, interference from precipitation particles (rain or snow), and static occlusions on the lens surface such as water droplets. Then, the COLMAP algorithm is used to obtain camera parameters and pose relationships between viewpoints.

[0024] Step 2: This step aims to process the raw image sequences acquired under complex weather conditions such as low light, rain, and snow. Preprocessing is performed to obtain a clear image sequence. And the mask M of static obstructions on the lens surface.

[0025] 2.1 Design of precipitation particle removal module: In view of precipitation particles generated by rain or snow in the image, this invention designs a module for removing precipitation particle interference in the image based on a fine-tuned potential diffusion model (Stable Diffusion).

[0026] First, to address the problem of insufficient representation ability of existing models under low light conditions, this invention constructs a special training dataset for complex weather (including the MipNeRF360 dataset, a benchmark data version of rainy weather data in DeRainGS, and multi-view images of libraries, stadiums, and playgrounds under rainy and snowy weather that are collected and processed independently) and establishes ground truth pairs between degraded images and corresponding clear scenes to conduct supervised training of the model.

[0027] The precipitation particle removal module first prepares a pre-trained latent diffusion model. Then, it uses the aforementioned specialized dataset and the existing LoRA (Low-Rank Adaptive) fine-tuning method to retrain and supervise the pre-trained latent diffusion model. This allows the precipitation particle removal module to explicitly learn the physical distribution characteristics of precipitation particles in the image, thereby achieving accurate removal of precipitation particles while preserving the texture of the weak-light background.

[0028] During the inference phase, the original image sequence The input is a precipitation particle removal module, which can remove precipitation particles while preserving the spatial details of the scene, and output a clear image sequence with dense precipitation particles such as snow removed. .

[0029] 2.2 Obtaining the mask M based on the segmentation large model SAM; After removing precipitation particles from the image, this invention innovatively introduces the Segment Anything Model (SAM) for high-precision semantic segmentation and occlusion recognition, targeting static occlusions such as water droplets and oil stains adhering to the lens surface. SAM is a basic segmentation model based on the Transformer architecture, capable of high-quality segmentation of any object in an image based on input prompts. In this embodiment, the clear image sequence generated in step 2.1 is used... The input is fed into the SAM model, with the prompt "lens raindrops" serving as task guidance information. SAM identifies pixel regions belonging to "lens raindrops" and converts these regions into a mask M. According to the design of this invention, the pixel values ​​of the identified raindrop-occluded pixel regions are marked as 0, indicating that the pixel region is invalid; the pixel values ​​of the remaining valid scene regions are marked as 1. This mask M will be used in subsequent steps to shield the gradient backpropagation of occluded pixel regions when calculating the loss function, preventing the generation of erroneous geometric structures.

[0030] Step 3: Utilize the clear image sequence obtained in Step 2 Extract scene geometric priors such as depth and edge gradients that are illumination-invariant, and generate a depth prior map. and edge gradient prior map As a scene geometry prior, it provides stable geometric feature constraints for the subsequent 3D Gaussian splash model G.

[0031] 3.1 Extracting the depth prior map D based on the monocular depth estimation model Inference is performed on the input image using a monocular depth estimation base model, such as the Depth Anything model trained on large-scale data. The Depth Anything model, trained on a large-scale mixture of unlabeled and labeled data, directly extracts robust depth features through a powerful visual encoder. In this example, the input is the sharp image sequence generated in step 2.1. Generate clear image sequences The corresponding depth prior map D. This depth map provides accurate relative distance information of the scene, which is introduced into subsequent processes to constrain the geometry of the 3D Gaussian splash model G, effectively preventing geometric collapse and artifacts in low-texture areas or low-light scenes.

[0032] 3.2 Extracting the edge gradient prior map S based on the pre-trained DexiNed edge detection operator To accurately capture edge gradient information in images under low-light conditions, this invention uses a pre-trained DexiNed edge detection operator. The DexiNed edge detection operator can accurately capture edge gradient information from clear image sequences without requiring fine-tuning for specific scenes. The algorithm identifies and enhances edge features in image sequences, generating a highly continuous and refined edge gradient prior map S. Even in low-contrast regions caused by low light, the deep learning-based DexiNed edge detection operator can compensate for missing edge information through global semantic context, ensuring that the edge gradient prior map S has strong robustness.

[0033] The edge gradient prior map S is used as a geometric consistency constraint input into the subsequent 3D Gaussian splash model G to guide the distribution and shape optimization of the Gaussian sphere, effectively preventing artifacts in low-light areas.

[0034] Step 4: Construct a 3D Gaussian splash model G containing edge gradients, depth, and lighting properties and perform supervised training.

[0035] First, a three-dimensional Gaussian splash model G is constructed using a sparse point cloud (approximately 1000 three-dimensional points) generated by COLMAP. Then, the existing mature differentiable rasterization method is called (see formula (4)). Based on the camera parameters and the current camera pose, multiple attributes of the three-dimensional Gaussian splash model are projected onto the two-dimensional image space to obtain the predicted low-light map. Predicting edge gradient maps Predicted depth map And predicted residual illumination map Then predict the low-light map And predicted residual illumination map Perform linear fusion to obtain the predicted normal illumination map. .

[0036] Clear image sequence Edge gradient prior map S, depth prior map and modulated images (Formula (9)) serves as a constraint, combined with the mask M, to predict the result (for the predicted low-light image). Predicting edge gradient maps Predicted depth map and predicting normal illumination map We perform weighted calculations to construct the reconstruction loss separately. (Formula (6)), edge gradient consistency loss (Formula (7)), depth consistency loss (Equation (8)) and exposure control loss .

[0037] Finally, a total loss function integrating all the above components is constructed. (Formula (5)) During the 30,000 iterations, the number, position, shape, color and other attributes of the Gaussian spheres in the 3D Gaussian splash model G are optimized. To ensure training stability, the position learning rate is set to linearly decay from the initial value of 0.00025 to 0.0000025 (the main decay is completed within 20,000 rounds). Gaussian sphere gain and trimming operations are continuously performed before the 10,000th round to eliminate the interference of low light and static lens occlusion on the modeling of the 3D Gaussian splash model G.

[0038] It should be noted that, although the aforementioned clear image sequence While rain and snow artifacts have been effectively removed, low-light characteristics remain. To address the reconstruction challenges posed by static obstructions on the lens surface and low-light environments, this invention does not employ the traditional 3D Gaussian splash model that only includes position, shape, and color attributes. Instead, it constructs a multi-dimensional attribute 3D Gaussian splash model G. In the 3D Gaussian splash model G, each point in the sparse point cloud is modeled as a Gaussian sphere with basic attributes such as spatial position, covariance, color, and opacity. The Gaussian sphere is the basic unit in the 3D Gaussian splash model. Each Gaussian sphere is an ellipsoidal 3D Gaussian distribution, and its spatial influence range is defined by the covariance matrix, which determines its shape and orientation in 3D space (Equation 3). During rendering, the Gaussian sphere is projected onto a 2D Gaussian distribution on the image plane and participates in the cumulative calculation of pixel color (Equation 4). In this invention, by explicitly binding additional attribute coefficients, including edge gradient, depth, and illumination attributes, to each Gaussian sphere, the model can jointly update position, shape, color, and additional attribute coefficients during iterative optimization, thereby obtaining a three-dimensional Gaussian splash model G with good illumination and free from interference from precipitation particles such as rain and snow, as well as static obstructions attached to the lens surface, such as... Figure 3 As shown.

[0039] In the three-dimensional Gaussian splash model G, the influence of each Gaussian sphere in three-dimensional space is defined by the three-dimensional Gaussian probability density function, as follows: in, This represents the coordinates of the point where the influence (radiation field intensity) is to be calculated under the influence of the k-th Gaussian sphere. This represents the coordinates of the center point of the k-th Gaussian sphere. This represents the influence of the k-th Gaussian sphere at point x. Let represent the covariance matrix of the k-th Gaussian sphere, controlling the shape, size, and orientation of the Gaussian kernel in space. This represents the covariance matrix corresponding to the Gaussian sphere k. The inverse matrix.

[0040] Rasterization rendering: Utilizing existing, mature differentiable rasterizers, the 3D Gaussian splash model G is projected and rendered based on the current camera parameters and the current viewpoint pose, producing the following components: predicted edge gradient map. Predicted depth map Predicting low-light rendering results and predicting normal illumination map The rasterization rendering formula is as follows: in, Represents pixel coordinates, This represents the sorted set of Gaussian spheres covering the pixel. Let k be the opacity of the Gaussian sphere. For the opacity of the j-th Gaussian sphere, in the formula... and Depending on the rendering target, the corresponding physical quantities are as follows: When rendering a predicted low-light map hour, Refers to pixel color , The color attribute of the k-th Gaussian sphere ; When rendering the predicted edge gradient map hour, Refers to marginal gradient eigenvalues , Refers to the edge gradient property bound to the k-th Gaussian sphere ; When rendering the predicted depth map hour, Refers to depth value , Refers to the depth of the k-th Gaussian sphere in the camera coordinate system ; When rendering the predicted residual lighting map hour, Refers to lighting attributes , Refers to the lighting properties bound to the k-th Gaussian sphere .

[0041] Total loss function Losses from reconstruction Marginal gradient consistency loss Deep consistency loss and exposure control loss The weighted average of multiple sub-losses is as follows: in These are all hyperparameters, used to adjust the balance weight of each sub-loss term in the overall optimization objective.

[0042] Simultaneously, each loss term is weighted using a mask M, and the loss function is calculated only in the unoccluded regions to eliminate gradient contributions from occluded areas. The specific calculation method for each sub-loss term is as follows: Reconstruction losses : Used to constrain the synthetic image to approximate the true appearance, denoted as: in, Indicates pixel coordinates The occlusion mask value at the location, It belongs to hyperparameters. This represents structural dissimilarity loss, used to constrain the consistency between the predicted image and the ground truth in local texture and geometry, thus avoiding blurry reconstruction results. Represents norm operations.

[0043] Marginal gradient consistency loss : Used to constrain edge gradient information, this term ensures that even in low-light regions, the geometric distribution of the 3D Gaussian splash model G closely matches the physical edge gradient prior, and is expressed as: in, Represents pixel coordinates, Represents norm operations, This indicates dot product.

[0044] Deep consistency loss : Used to constrain geometric information, ensuring that the 3D Gaussian splash model G does not experience geometric collapse in low-texture or dark-light regions, and is not misled by incorrect depth of the water droplets in the lens, represented as: in, This represents the gradient operator, used to extract the spatial rate of change of the depth map in the horizontal and vertical directions.

[0045] Exposure control loss To achieve high-quality image synthesis, this invention employs exposure control loss optimized for rendered images across all viewpoints. , is represented as: Among them, modulated image , This is a hyperparameter, set to 0.4 in this method.

[0046] Table 1 presents the quantitative comparison results of the proposed method on the pool_rain dataset with the methods of "precipitation removal particle module + 3D Gaussian splashing", "Illumination-Adaptive-Transformer (IAT) + 3D Gaussian splashing", and "precipitation removal particle module + IAT + 3D Gaussian splashing" multi-stage methods. The evaluation metrics include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learning-Aware Patch Similarity (LPIPS). PSNR is mainly used to evaluate the pixel-level reconstruction accuracy and denoising level of the image. The higher the value of this metric, the higher the reconstruction accuracy and denoising level. SSIM focuses on evaluating the brightness, contrast, and geometric integrity of the image. The closer the value of this metric is to 1, the higher the structural fidelity. LPIPS is used to quantify the degree to which the image conforms to human visual perception. The lower the value of this metric, the clearer, more realistic, and artifact-free the image.

[0047] The method of this invention achieves optimal values ​​in both PSNR and SSIM metrics, indicating that it effectively removes precipitation particles such as rain and snow, as well as static obstructions attached to the lens surface, while preserving the geometric structure and edge details of the scene. Furthermore, the significantly lower LPIPS score further confirms that the normally lit images generated by the method of this invention have a sharpness and realism that more closely resembles human visual perception.

[0048] Table 1 Figure 4 This paper compares the rendering results of the method of this invention with those of multi-stage methods such as "de-precipitation particle module + 3D Gaussian splashing", "IAT + 3D Gaussian splashing", and "de-precipitation particle module + IAT + 3D Gaussian splashing". Specifically, while the "de-precipitation particle module + 3D Gaussian splashing" method can remove rain particles from the environment, it performs poorly under low light conditions and fails to handle static occlusions attached to the lens surface. The "IAT + 3D Gaussian splashing" method, while improving the scene brightness of the 3D Gaussian splashing model, often results in unnatural brightness and is still limited by rain and snow particles and static occlusions attached to the lens surface. Even the combined "de-precipitation particle module + IAT + 3D Gaussian splashing" method cannot effectively eliminate artifacts from static occlusions attached to the lens surface. In contrast, the method of this invention effectively overcomes the above limitations, achieving not only well-lit perspective rendering but also accurately removing rain particles and static occlusions attached to the lens surface, reconstructing a high-quality scene with clear textures and accurate structure.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A 3D Gaussian splash modeling method based on scene priors under complex weather conditions, characterized in that, Includes the following steps: Step 1: Acquire raw image sequence: Under complex weather conditions, acquire a set of raw low-light images of the target scene. The motion reconstruction structure technique is used to obtain camera parameters and pose relationships between viewpoints; Step 2: Preprocessing of the original image sequence: Removing the original image sequence using a fine-tuned diffusion model. Clear image sequences were obtained by obscuring the images from precipitation particles. ; Using a large segmentation model to identify clear image sequences A transparent or semi-transparent static obstruction attached to the surface of the lens generates a corresponding mask M; Step 3: Extract scene geometric prior information with illumination invariance: Using a monocular depth estimation model and a pre-trained DexiNed edge detection operator, extract prior information from a clear image sequence. The scene geometric prior information, which is invariant to illumination, is extracted from the image to generate a scene geometric prior map; the scene geometric prior map includes a depth prior map. and edge gradient prior map ; Step 4: Multi-attribute 3D Gaussian splash modeling based on scene geometric prior information: based on clear image sequences and its corresponding edge gradient prior map S and depth prior map A 3D Gaussian splash model G, incorporating edge gradients, depth, and lighting properties, is constructed. The predicted edge gradient map is then rendered using a differentiable rasterization method. Predicted depth map Predicting low-light maps And predicted residual illumination map Combine the mask M to construct the total loss function. Including reconstruction losses Marginal gradient consistency loss Deep consistency loss and exposure control loss By optimizing the total loss function Optimize the three-dimensional Gaussian splash model G.

2. The three-dimensional Gaussian splash modeling method based on scene priors under complex weather conditions as described in claim 1, characterized in that, Step 2 specifically includes the following steps: 2.1 Design of a precipitation particle removal module. This module is based on a fine-tuned latent diffusion model and is used to remove precipitation particle interference from images, converting the original image sequence... Input the precipitation removal particle module to generate a clear image sequence. ; 2.2 Obtaining the mask M based on the segmentation large model SAM; clear image sequence The input is fed into the segmentation model SAM. The input prompt "lens raindrop" serves as task guidance information. The segmentation model SAM identifies the pixel regions belonging to "lens raindrop" and converts the identified pixel regions into a mask M. The pixel values ​​of the identified occluded pixel regions are marked as 0, indicating that the pixel region is invalid. The pixel values ​​of the remaining valid scene regions are marked as 1.

3. The three-dimensional Gaussian splash modeling method based on scene priors under complex weather conditions as described in claim 2, characterized in that, For the precipitation removal particle module, a special training dataset for complex weather was constructed, and ground truth pairs between degraded images and corresponding clear scenes were established for supervised training of the model. The special training dataset includes the MipNeRF360 dataset, the rainy day benchmark data version of DeRainGS, as well as multi-view images of libraries, stadiums and playgrounds under rainy and snowy weather that were collected and processed independently.

4. The three-dimensional Gaussian splash modeling method based on scene priors under complex weather conditions as described in claim 3, characterized in that, For the precipitation removal particle module, firstly, a pre-trained potential diffusion model is prepared. Secondly, the special training dataset is combined with the existing LoRA fine-tuning method to retrain and supervise the fine-tuning of the pre-trained potential diffusion model. This enables the precipitation particle removal module to explicitly learn the physical distribution characteristics of precipitation particles in images.

5. The three-dimensional Gaussian splash modeling method based on scene priors under complex weather conditions as described in claim 4, characterized in that, Step 3 specifically includes 3.1 extracting the depth prior map D based on the monocular depth estimation model, and then processing the clear image sequence. Input a monocular depth estimation base model and generate a sequence of sharp images. The corresponding depth prior map D.

6. The three-dimensional Gaussian splash modeling method based on scene priors under complex weather conditions as described in claim 5, characterized in that, Step 3 specifically includes 3.2 extracting the edge gradient prior map S based on the pre-trained DexiNed edge detection operator, and using the pre-trained DexiNed edge detection operator to extract the edge gradient prior map S from the clear image sequence. The algorithm identifies and enhances edge features in image sequences, generating a highly continuous and refined edge gradient prior map S.

7. The three-dimensional Gaussian splash modeling method based on scene priors under complex weather conditions as described in claim 5, characterized in that, Step 4 specifically includes the following steps: First, a 3D Gaussian splash model G is constructed using sparse point clouds generated by COLMAP. Then, using the existing mature differentiable rasterization method, multiple attributes of the 3D Gaussian splash model are projected onto the 2D image space based on camera parameters and the current camera pose to obtain the predicted low-light map. Predicting edge gradient maps Predicted depth map And predicted residual illumination map Then predict the low-light map And predicted residual illumination map Perform linear fusion to obtain the predicted normal illumination map. ; Secondly, a clear image sequence Edge gradient prior map S, depth prior map and modulated images As a constraint, combined with the mask M, the predicted low-light map is analyzed separately. Predicting edge gradient maps Predicted depth map and predicting normal illumination map We perform weighted calculations and construct the reconstruction loss separately. Marginal gradient consistency loss Deep consistency loss and exposure control loss ; Finally, construct the total loss function. Optimize multiple properties in the 3D Gaussian splash model G.

8. The three-dimensional Gaussian splash modeling method based on scene priors under complex weather conditions as described in claim 7, characterized in that, Total loss function Losses from reconstruction Marginal gradient consistency loss Deep consistency loss and exposure control loss Weighted average of multiple sub-losses composition: in These are all hyperparameters, used to adjust the balance weight of each sub-loss term in the overall optimization objective.

9. The three-dimensional Gaussian splash modeling method based on scene priors under complex weather conditions as described in claim 8, characterized in that, Each loss is weighted by the mask M, and the loss function is calculated only in the unoccluded region. The specific calculation of each loss is as follows: Reconstruction losses Used to constrain the synthetic image to approximate the true appearance, denoted as: in, Indicates pixel coordinates The occlusion mask value at the location, It belongs to hyperparameters. This represents structural dissimilarity loss, used to constrain the consistency between the predicted image and the ground truth in local texture and geometry, thus avoiding blurry reconstruction results. Represents norm operations; Marginal gradient consistency loss Represented as: in, Represents pixel coordinates, Represents norm operations, Dot product; Deep consistency loss Represented as: in, This represents the gradient operator, used to extract the spatial rate of change of the depth map in the horizontal and vertical directions; Exposure control loss Represented as: Among them, modulated image , This is a hyperparameter.

10. A 3D Gaussian splash modeling system based on scene priors under complex weather conditions, characterized in that, It includes a data acquisition module, a data preprocessing module, a scene geometric prior information extraction module, and a multi-attribute 3D Gaussian splash modeling module; The data acquisition module acquires raw image sequences, obtaining a set of low-light raw image sequences of the target scene under complex weather conditions such as rain and snow. ; The data preprocessing module preprocesses the original image sequence and removes data from the original image sequence using a fine-tuned diffusion model. Medium precipitation particles obscure the image, resulting in a clear image sequence. And a large segmentation model is used to identify clear image sequences. A transparent or semi-transparent static obstruction attached to the surface of the lens generates a corresponding mask M; The scene geometry prior information extraction module extracts scene geometry prior information with illumination invariance, and uses a monocular depth estimation model and a pre-trained DexiNed edge detection operator to extract prior information from a clear image sequence. In the process, illumination-invariant depth information and scene geometric prior information of geometric structure are extracted to generate a depth prior map. and edge gradient prior map ; The aforementioned multi-attribute 3D Gaussian splash modeling module, based on a clear image sequence and its corresponding edge gradient prior map S and depth prior map A 3D Gaussian splash model G, incorporating edge gradients, depth, and lighting properties, is constructed. The predicted edge gradient map is then rendered using a differentiable rasterization method. Predicted depth map Predicting low-light maps And predicted residual illumination map Combine the mask M to construct the total loss function. Including reconstruction losses Marginal gradient consistency loss Deep consistency loss and exposure control loss By optimizing the total loss function Optimize the three-dimensional Gaussian splash model G.

Images