A method and system for three-dimensional gaussian splash optimization for degraded images

Abstract

The application belongs to the technical field of computer vision and three-dimensional reconstruction, and particularly relates to a three-dimensional Gaussian splash optimization method and system for degraded images. By introducing a perceptual consistency loss based on a pre-trained visual feature extraction network in the training process, the model is guided to focus on the overall structure and semantic information of the scene rather than the degraded pixel details, effectively suppressing the overfitting of the model to noise, blur and other degradation features, so that a stable gradient direction can be obtained in the early training process, significantly accelerating the convergence speed. At the same time, by performing scale-related low-pass filtering correction on the two-dimensional Gaussian projection in the screen space rendering stage, aliasing artifacts and edge jitter phenomena are effectively suppressed. The comprehensive use of the above-mentioned means enables the method to realize stable and efficient three-dimensional reconstruction and new view synthesis under various degradation scenes such as noise, blur, rain and fog, underwater scattering and the like, and the rendered image is clear, natural and free of structural artifacts.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D reconstruction technology, specifically relating to a 3D Gaussian sputtering optimization method and system for degraded images, which is used to achieve efficient and stable 3D reconstruction and new viewpoint synthesis under low-quality input image conditions. Background Technology

[0002] Image-based 3D reconstruction and novel viewpoint synthesis, aiming to recover the 3D structure of a scene from a limited number of 2D images and generate images from any new perspective, is a hot research topic in the field of computer vision. This technology has broad application prospects in areas such as autonomous driving, virtual reality, and robot navigation.

[0003] In recent years, implicit scene representation methods, represented by Neural Radiance Fields (NeRF), have made significant progress and can generate high-quality, realistic images. However, NeRF-based methods typically require intensive viewpoint sampling and a large amount of volumetric rendering computation, resulting in slow training and inference speeds, making it difficult to meet real-time requirements.

[0004] To address the aforementioned efficiency issues, the 3D Gaussian Splatting (3DGS) method was proposed and quickly became a research focus in this field. 3DGS employs an explicit 3D Gaussian ellipsoid as the basic representation unit of the scene. By projecting 3D Gaussian primitives onto a 2D image plane and using depth-based α-blending rendering, it can achieve high-quality synthesis of new viewpoints. Furthermore, its rasterization-based rendering pipeline offers a speed improvement of more than two orders of magnitude compared to NeRF's volumetric rendering, enabling real-time or near-real-time rendering capabilities.

[0005] However, the superior performance of existing 3DGS methods is highly dependent on the quality of the input image. Under ideal experimental conditions, 3DGS can indeed reconstruct detailed 3D models, but in real-world applications, due to limitations in the performance of the shooting equipment, complex and variable imaging environments, and acquisition conditions, the acquired multi-view images generally suffer from various degradation phenomena. These include rain, snow, and fog in autonomous driving scenarios, color attenuation and contrast reduction during underwater robot operations, and low-light noise and motion blur in handheld device shooting.

[0006] Existing 3DGS methods exhibit the following significant technical shortcomings when processing the aforementioned degraded images:

[0007] First, the loss function of the original 3DGS mainly relies on the pixel-level error between the rendered image and the input image. Under the condition of degraded image, noise, blur and other interferences will be mistaken as the real high-frequency details that need to be fitted, causing the model to fall into a local optimum of "learning degradation" in the early stage of optimization, with gradient direction oscillation, slow parameter update and extremely low training efficiency.

[0008] Second, the loss or distortion of high-frequency information in degraded images will seriously interfere with the adaptive density control mechanism of 3DGS, producing a large number of fragmented floating artifacts. Furthermore, the inconsistency in the degree of degradation under different viewpoints will destroy the multi-view geometric consistency constraint, resulting in visual defects such as local texture fragmentation and edge ghosting.

[0009] Third, the effective texture information of objects at different distances in degraded images is weakened unevenly, making the model extremely sensitive to changes in camera distance and viewpoint. When there is a large scale difference between the new viewpoint and the training viewpoint, the rendering results will exhibit severe jagged edges or excessive blurring. Summary of the Invention

[0010] In view of the above problems, this application provides a three-dimensional Gaussian spatter optimization method and system for degraded images, which is used to solve the technical problems involved in the background art.

[0011] To achieve the above objectives, in a first aspect, this application provides a three-dimensional Gaussian spatter optimization method for degraded images, comprising the following steps:

[0012] Acquire multi-view degraded images and corresponding camera parameters;

[0013] The three-dimensional Gaussian distribution parameters are initialized based on the multi-view degraded image. The three-dimensional Gaussian distribution parameters include position parameters, covariance matrix parameters, opacity parameters, and color parameters.

[0014] During the training iteration, the rendered image obtained by the current 3D Gaussian distribution rendering and the input image are respectively input into the pre-trained visual feature extraction network to extract their respective semantic feature maps, and a perceptual consistency loss is constructed based on the similarity between the semantic feature maps.

[0015] During the screen space rendering stage, the three-dimensional Gaussian distribution is projected into a two-dimensional Gaussian distribution, and the covariance matrix of the two-dimensional Gaussian distribution is low-pass filtered and corrected according to the current rendering scale.

[0016] The perceptual consistency loss and pixel-level reconstruction loss are weighted and fused to construct a total loss function, and the three-dimensional Gaussian distribution parameters are updated using the total loss function;

[0017] The above training steps are executed iteratively until the convergence condition is met. The camera parameters of the target viewpoint are input, and the optimized three-dimensional Gaussian distribution is used to render and generate a two-dimensional image of the target viewpoint.

[0018] Unlike existing technologies, the technical solution of this application introduces perceptual consistency loss based on a pre-trained visual feature extraction network during training. This guides the model to focus on the overall structure and semantic information of the scene rather than degraded pixel details, effectively suppressing overfitting of the model to degraded features such as noise and blur. This allows the optimization process to obtain a stable gradient direction early in training, significantly accelerating the convergence speed. Simultaneously, by applying scale-dependent low-pass filtering correction to the 2D Gaussian projection during the screen space rendering stage, aliasing artifacts and edge jitter are effectively suppressed. The combined application of these methods enables this approach to achieve stable and efficient 3D reconstruction and new viewpoint synthesis in various degraded scenarios such as noise, blur, rain, fog, and underwater scattering. The rendered images are clear, natural, and free of structural artifacts, with only a small increase in computational overhead, maintaining the original real-time rendering advantages of the 3D Gaussian splashing method.

[0019] In one embodiment of the present invention, the visual feature extraction network is a CLIP image encoder, and the perceptual consistency loss uses cosine similarity to measure the consistency between the rendered image and the input image in the CLIP feature space, specifically:

[0020] ;

[0021] in, For the rendered image, The input image, The feature vectors extracted and L2-normalized by the CLIP image encoder This is the cosine similarity function.

[0022] As one embodiment of the present invention, the projection of the three-dimensional Gaussian distribution into a two-dimensional Gaussian distribution specifically includes:

[0023] Projection transformation matrix based on the current viewpoint And Jacobi matrix Calculate the two-dimensional covariance matrix in screen space. ,in The three-dimensional covariance matrix is ​​a three-dimensional Gaussian distribution.

[0024] The low-pass filter correction is achieved in the following way:

[0025] ;

[0026] in, This is the corrected covariance matrix. The filtering parameters are related to the current rendering scale. It is an identity matrix.

[0027] As one embodiment of the present invention, the filtering parameters Determined based on the resolution of the currently rendered image:

[0028] ;

[0029] in, and These are the height and width of the currently rendered image, respectively. This is a preset constant.

[0030] As one embodiment of the present invention, the total loss function is:

[0031] ;

[0032] in, For the pixel-level reconstruction loss, This refers to the perceived consistency loss. The weighting coefficients for perceived consistency loss.

[0033] As one embodiment of the present invention, the pixel-level reconstruction loss Depend on The loss is weighted by the structural similarity loss:

[0034] ;

[0035] in, Pixel level loss, For structural similarity loss, represents the weighting coefficients for structural similarity loss.

[0036] As one embodiment of the present invention, the initialization of the three-dimensional Gaussian distribution parameters based on the multi-view degraded image specifically includes:

[0037] A sparse point cloud is generated from the multi-view degraded image using the structure-reconstruction-motion method;

[0038] Initialize a set of three-dimensional Gaussian distributions using each three-dimensional point in the sparse point cloud as the mean;

[0039] The position parameters of each 3D Gaussian distribution are initialized to the coordinates of the corresponding 3D point, the covariance matrix is ​​initialized to an isotropic matrix, the opacity is initialized to a preset value, and the color parameters are initialized to zero.

[0040] As one embodiment of the present invention, after projecting the three-dimensional Gaussian distribution into a two-dimensional Gaussian distribution during the screen space rendering stage, the method further includes:

[0041] The final color value of the target pixel is obtained by weighting and accumulating all two-dimensional Gaussian distributions that overlap with the target pixel in ascending order of depth.

[0042] To achieve the above objectives, in a second aspect, this application provides a three-dimensional Gaussian spatter optimization system for degraded images, comprising:

[0043] The data acquisition module is used to acquire multi-view degraded images and corresponding camera parameters;

[0044] An initialization module is used to initialize the three-dimensional Gaussian distribution parameters based on the multi-view degraded image;

[0045] The perceptual consistency constraint module includes a pre-trained visual feature extraction network, which is used to extract semantic features of the rendered image and the input image and calculate the perceptual consistency loss.

[0046] The filtering rendering module is used to project a three-dimensional Gaussian distribution into a two-dimensional Gaussian distribution in screen space, and to perform low-pass filtering correction on the covariance matrix of the two-dimensional Gaussian distribution according to the current rendering scale, and to perform color accumulation rendering.

[0047] An adaptive density control module is used to perform splitting, cloning, or culling operations on a three-dimensional Gaussian distribution based on the gradient magnitude and opacity value during the optimization process.

[0048] The joint optimization module is used to weightedly fuse the perceptual consistency loss and the pixel-level reconstruction loss to construct a total loss function, and to update the three-dimensional Gaussian distribution parameters through backpropagation.

[0049] The new perspective rendering module is used to generate a 2D image of the target perspective by rendering it using an optimized 3D Gaussian distribution based on the camera parameters of the target perspective.

[0050] Unlike existing technologies, the technical solution of this application achieves systematic support for each step of the above methods through the collaborative work of a data acquisition module, an initialization module, a perceptual consistency constraint module, a filtering and rendering module, an adaptive density control module, a joint optimization module, and a new perspective rendering module. Specifically, the perceptual consistency constraint module enables the system to resist interference from degraded images; the filtering and rendering module ensures the visual stability of the rendering results; the adaptive density control module allows the density of the 3D Gaussian distribution to be adaptively adjusted according to scene complexity during the optimization process; and the joint optimization module integrates multi-level supervision signals to achieve end-to-end optimization. These modules together constitute a complete 3DGS optimization system that can still operate stably and efficiently under degraded image conditions, exhibiting good engineering practicality and ease of deployment.

[0051] In one embodiment of the present invention, in the perceptual consistency constraint module, the visual feature extraction network is a CLIP image encoder, and the perceptual consistency loss uses cosine similarity to measure the consistency between the rendered image and the input image in the CLIP feature space; in the filtering rendering module, the low-pass filtering correction is implemented in the following way:

[0052] ;

[0053] in, Let be the original covariance matrix of the two-dimensional Gaussian distribution. This is the corrected covariance matrix. These are the filtering parameters related to the current rendering resolution. It is an identity matrix.

[0054] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0055] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of this application and other related content, and should not be considered as limitations on this application.

[0056] In the accompanying drawings of the instruction manual:

[0057] Figure 1 This is a flowchart illustrating the steps of a three-dimensional Gaussian spatter optimization method for degraded images according to this embodiment;

[0058] Figure 2This is a block diagram of a three-dimensional Gaussian spatter optimization system for degraded images according to this embodiment;

[0059] Figure 3 The first set of comparison images in this embodiment (from left to right: the original image, the image rendered using 3DGS, and the image rendered using this solution);

[0060] Figure 4 This is the second set of comparison images in this embodiment (from left to right: the original image, the image rendered using 3DGS, and the image rendered using this solution).

[0061] Figure 5 This is the third set of comparison images in this embodiment (from left to right: the original image, the image rendered using 3DGS, and the image rendered using this solution).

[0062] Figure 6 This is the fourth set of comparison images in this embodiment (from left to right: the original image, the image rendered using 3DGS, and the image rendered using this solution).

[0063] Figure 7 This is the fifth set of comparison images in this embodiment (from left to right: the original image, the image rendered using 3DGS, and the image rendered using this solution).

[0064] The reference numerals used in the above figures are explained as follows:

[0065] 1. Data acquisition module; 2. Initialization module; 3. Perceptual consistency constraint module; 4. Filtering and rendering module; 5. Adaptive density control module; 6. Joint optimization module; 7. New perspective rendering module. Detailed Implementation

[0066] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0067] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0068] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0069] Example 1

[0070] like Figure 1 As shown, this embodiment details the complete implementation process of the three-dimensional Gaussian spatter optimization method for degraded images proposed in this invention.

[0071] First, perform the data acquisition step: acquire 2D degraded images from multiple perspectives of the same scene, along with the camera parameters corresponding to each frame. A degraded image is defined as an image containing at least one of the following degradation features: noise, blur, rain / fog occlusion, underwater scattering, or color attenuation. Camera parameters include camera intrinsic parameters (such as focal length, principal point coordinates, and distortion coefficients) and camera pose parameters (such as rotation matrices and translation vectors in world coordinates). Camera parameters can be provided directly from the dataset or pre-calculated using motion reconstruction tools.

[0072] Next, the initialization step is performed: 3D Gaussian distribution parameters are initialized based on the acquired multi-view degraded images. The 3D Gaussian distribution is the basic unit of 3DGS scene representation. Each Gaussian distribution contains position parameters (3D spatial coordinates), covariance matrix parameters (describing the shape and orientation of the Gaussian ellipsoid), opacity parameters (controlling the contribution weight of the Gaussian element to the final color), and color parameters (usually using spherical harmonic function coefficients to represent view-related color information). During initialization, a sparse point cloud is generated from the multi-view degraded images using the structure-of-motion method, and a set of 3D Gaussian distributions is initialized with each 3D point in the sparse point cloud as the mean. This initialization method utilizes multi-view geometric constraints to ensure that the initial Gaussian positions are basically aligned with the real geometric structure of the scene.

[0073] The perceptual consistency constraint construction step is then performed: In each training iteration, differentiable rendering is performed using the current 3D Gaussian distribution parameters to obtain a rendered image from the target viewpoint; this rendered image and the real input image from the same viewpoint are respectively input into a pre-trained visual feature extraction network (such as CLIP image encoder) to extract their respective high-level semantic feature maps; a perceptual consistency loss function is constructed based on the similarity (such as cosine similarity) between the semantic feature maps of the two images. This loss function does not require the model to accurately reproduce noise, raindrops, or blurred details in the degraded image, but rather encourages the rendering result to maintain consistency with the input image in terms of object contours, scene layout, and overall appearance distribution, thereby effectively suppressing the model's overfitting to degraded information.

[0074] Then, a multi-scale filtering rendering step is performed: In the screen-space rendering stage, each 3D Gaussian distribution is projected into a 2D Gaussian distribution through a projection transformation, and the covariance matrix of the 2D Gaussian distribution is low-pass filtered and corrected according to the current rendering scale. This correction is achieved by superimposing an isotropic increment related to the rendering resolution on the covariance matrix, which is equivalent to performing low-pass filtering on the 2D Gaussian projection in screen space, simulating the pixel integration process, thereby suppressing aliasing artifacts and edge instability.

[0075] The next step is joint optimization training: the perceptual consistency loss is weighted and summed with the pixel-level reconstruction loss of the original 3DGS (a weighted sum of L1 loss and SSIM loss) to construct the total loss function; the total loss function is then used to update all 3D Gaussian distribution parameters (position, covariance, opacity, and color parameters) through backpropagation. Through this joint optimization of multi-level supervision signals, the model can maintain the accuracy of local color and texture using pixel-level loss, while maintaining the consistency of the overall semantic structure with the help of perceptual loss.

[0076] Finally, the new perspective image generation step is executed: the above training steps are iteratively executed until the model converges (meeting the preset number of iterations or loss threshold), the camera parameters of any new target perspective are input, and the same multi-scale filtering rendering process is executed using the optimized three-dimensional Gaussian distribution to generate a high-quality two-dimensional image under the target perspective.

[0077] Based on the above technical solutions, by introducing perceptual consistency loss based on a pre-trained visual feature extraction network during training, the model is guided to focus on the overall structure and semantic information of the scene rather than degraded pixel details. This effectively suppresses overfitting of the model to degraded features such as noise and blur, allowing the optimization process to obtain a stable gradient direction early in training and significantly accelerating the convergence speed. Simultaneously, by applying scale-dependent low-pass filtering correction to the 2D Gaussian projection during the screen-space rendering stage, aliasing artifacts and edge jitter are effectively suppressed. The combined application of these methods enables this approach to achieve stable and efficient 3D reconstruction and new viewpoint synthesis in various degraded scenarios such as noise, blur, rain, fog, and underwater scattering. The rendered images are clear, natural, and free of structural artifacts, with only a small increase in computational overhead, maintaining the original real-time rendering advantages of the 3D Gaussian splashing method.

[0078] In a preferred embodiment of this example, the visual feature extraction network employs the CLIP (Contrastive Language-Image Pre-Training) image encoder. CLIP is a visual-language model pre-trained on large-scale image-text pairing data. Its image encoder contains a multi-layer Transformer structure, capable of extracting image features with high-level semantic discriminative capabilities. Taking the CLIPViT-B / 32 architecture as an example, it contains 12 Transformer encoder layers, with an output feature dimension of 512.

[0079] In practical implementation, the rendered image will be and input image The inputs are fed into the CLIP image encoder to extract the feature vectors from the last layer, and then L2 normalization is performed to obtain the normalized feature vectors. and Subsequently, cosine similarity was used to measure the consistency between the two images in the CLIP feature space, and perceptual consistency loss was applied. Defined as:

[0080] ;

[0081] in, This represents the cosine similarity function. When two images are completely identical at the perceptual level... When the perceptual features of the two are orthogonal, The loss function has a well-defined boundary range (0 to 1), which facilitates weighted fusion with other loss terms.

[0082] It's worth noting that the CLIP image encoder doesn't require input resizing because it incorporates adaptive pooling layers, enabling it to handle input images of arbitrary resolution. This avoids damaging or altering original degradation features due to resizing. The knowledge gained from pre-training on large-scale image and text data gives it natural robustness to low-level degradation such as color shifts and local noise, allowing it to stably extract semantic features related to object contours, scene layout, and overall appearance distribution.

[0083] Based on the above technical solution, by employing the CLIP image encoder and utilizing cosine similarity to construct a perceptual loss, the powerful semantic representation capabilities obtained by the CLIP model through pre-training on large-scale image and text data are fully utilized. This feature extraction network has natural robustness to low-level degradation such as color shift and noise interference, and can stably extract high-level semantic features related to object contours, scene layout, and overall appearance distribution. This allows the perceptual consistency loss to effectively guide the model to focus on the core structure of the scene rather than degenerate details, providing reliable and stable supervision signals for the optimization process.

[0084] In the preferred embodiment of this example, the specific process of projecting the three-dimensional Gaussian distribution into a two-dimensional Gaussian distribution and the low-pass filtering correction is defined.

[0085] During the screen-space rendering phase, for each 3D Gaussian distribution, its 3D covariance matrix is ​​denoted as... Calculate the projection transformation matrix based on the camera parameters of the current viewpoint. (Used to transform points in the world coordinate system to the camera coordinate system) and the Jacobian matrix (Used to transform points in the camera coordinate system to the ray / screen coordinate system). A two-dimensional covariance matrix with a 3D Gaussian distribution in screen space. Calculated using the following formula:

[0086] ;

[0087] In actual rendering, only need to focus on The 2×2 top-left submatrix is ​​used to describe the elliptical shape and orientation of the Gaussian distribution in screen space.

[0088] Based on this, perform low-pass filter correction: An isotropic increment related to the current rendering scale is superimposed, as shown in the following formula:

[0089] ;

[0090] in, This is the corrected two-dimensional covariance matrix. The filtering parameters (scalars) are related to the current rendering scale. It is a 2×2 identity matrix.

[0091] The physical meaning of this correction lies in performing low-pass filtering on the two-dimensional Gaussian projection in screen space. When As the value increases, the corrected covariance matrix expands isotropically, which is equivalent to smoothing a Gaussian distribution; when When the value approaches zero, it degenerates into the original projection result. This mechanism effectively prevents jagged edges caused by insufficient sampling when the Gaussian projection is too small, and also suppresses the interference of high-frequency noise in the degraded image on the update of the three-dimensional Gaussian parameters through gradient backpropagation.

[0092] Based on the above technical solution, a specific calculation method for the 3D Gaussian distribution projection as a 2D Gaussian distribution in screen space is defined by a clear mathematical formula. Furthermore, scale-dependent low-pass filtering correction is implemented using covariance matrix addition. This filtering correction, by superimposing isotropic increments related to the rendering scale onto the 2D covariance matrix, is equivalent to performing low-pass filtering on the 2D Gaussian projection in screen space, simulating the pixel integration process. This mechanism effectively prevents aliasing caused by insufficient sampling when the Gaussian projection is too small, and also suppresses the interference of high-frequency noise in degraded images on the 3D Gaussian parameter update through gradient backpropagation, thereby significantly improving the geometric stability and visual coherence of the rendering results.

[0093] In a preferred embodiment of this example, the filtering parameters The resolution is determined based on the current rendered image resolution; the specific calculation formula is as follows:

[0094] ;

[0095] in, and These are the height and width (in pixels) of the currently rendered image, respectively. This is a preset constant (unit: pixels).

[0096] The core design idea of ​​this formula is that the filtering strength is inversely proportional to the rendering resolution. When the rendered image resolution is low (i.e., ...), the filtering strength is inversely proportional to the rendering resolution. (smaller) Larger resolution, enhanced filtering to suppress aliasing artifacts caused by undersampling at low resolutions; when rendering images at higher resolutions (i.e., (larger) The filter is smaller and weaker to retain more true high-frequency details.

[0097] In one specific embodiment, for a high-definition image with a resolution of 1920×1080, ,Pick ,but For lower resolutions (such as 720p), the same... The value will produce a larger This enhances the filtering strength. The value can be adjusted according to the specific scene and the degree of degradation: for scenes with rich textures and where details need to be preserved, a smaller value can be selected. Values ​​(e.g., 200); for scenes with sparse textures or severe degradation, larger values ​​can be selected. Value (e.g., 400).

[0098] Based on the above technical solution, the filtering parameters The filtering intensity is inversely proportional to the current rendered image resolution, allowing it to adaptively adjust based on the rendering resolution. At lower resolutions, a single pixel covers a larger spatial area, and stronger filtering effectively suppresses aliasing. At higher resolutions, the filtering intensity decreases to preserve more realistic details. This design ensures good consistency in rendering at different resolutions and view distances, significantly improving the model's generalization ability in scale-changing scenes and avoiding the severe aliasing or excessive blurring problems caused by changes in viewing distance in traditional methods.

[0099] In a preferred embodiment of this example, the total loss function is a weighted sum of the perceptual consistency loss and the pixel-level reconstruction loss, and the specific formula is as follows:

[0100] ;

[0101] in, For pixel-level reconstruction loss, To perceive consistency loss, The weighting coefficients for perceived consistency loss.

[0102] In practice, The value is set based on the degradation intensity and scene characteristics. In a preferred embodiment, Set the value to 0.1. When the degradation intensity is low, the value can be appropriately reduced. (For example, take 0.05) to avoid over-smoothing; when the degradation intensity is high (such as strong noise, large fuzzy kernel, or severe underwater scattering), it can be increased. To increase the value to 0.2–0.3, so as to enhance the dominant role of perceptual constraints.

[0103] Based on the above technical solution, a total loss function is constructed by weighted summing of perceptual consistency loss and pixel-level reconstruction loss, achieving complementarity and synergy between the two supervision signals. Pixel-level reconstruction loss constrains the model to maintain accurate color and local structure in pixel space, while perceptual consistency loss constrains the model to maintain consistency in overall layout and appearance distribution in semantic space. The weighted fusion of the two enables the model to fully utilize reliable local information in the input image during training while effectively resisting erroneous supervision in degraded regions, achieving a dynamic balance between preserving realistic details and suppressing degradation interference, thereby obtaining better rendering quality under degradation conditions.

[0104] In a preferred embodiment of this example, pixel-level reconstruction loss Depend on The loss is a weighted average of the structural similarity index Measure (SSIM), and the specific formula is as follows:

[0105] ;

[0106] in, Pixel level Loss (i.e., the mean of the absolute differences per pixel between the rendered image and the input image). Structural similarity loss (defined as 1 minus the difference between the two images) index), These are the weighting coefficients for the structural similarity loss. In a preferred embodiment, Take 0.2, that is Loss and The weights for the losses were 0.8 and 0.2, respectively.

[0107] The loss provides stable point-by-point supervision at the pixel level and has good robustness to outliers; The loss measure is structural similarity at the image patch level, which has better perceptual relevance in preserving local texture and edges. The weighted fusion of the two measures allows the pixel-level reconstruction loss to have both the stability of pixel-level supervision and the perceptual quality of structure preservation.

[0108] Based on the above technical solution, the pixel-level reconstruction loss is further limited by... The loss is weighted by the structural similarity loss. The loss provides stable point-by-point supervision at the pixel level, ensuring that the rendered image remains consistent with the input image in color and brightness; The loss measure is structural similarity at the image patch level, which has better perceptual relevance in preserving local texture and edges. The weighted fusion of the two results in pixel-level reconstruction loss having both the stability of pixel-level supervision and the perceptual quality of structural preservation, forming a multi-level supervision system from local to global and from pixel to semantic with perceptual consistency loss.

[0109] In a preferred embodiment of this example, the specific process of initializing the three-dimensional Gaussian distribution parameters based on multi-view degraded images includes the following sub-steps:

[0110] The first step involves generating a sparse point cloud from multi-view degraded images using the Structure from Motion (SfM) method. Specifically, open-source tools such as COLMAP can be used to sequentially perform feature extraction (e.g., SIFT features), feature matching (e.g., nearest neighbor matching), and incremental bundle adjustment on the training set images to recover the sparse 3D point cloud of the scene and the camera parameters for each frame. Under degraded image conditions, the robustness of feature matching may be somewhat affected, but COLMAP's multi-view geometric constraints can still recover the basic skeletal structure of the scene. In one specific embodiment, the sparse point cloud contains approximately 8,000 to 15,000 3D points.

[0111] The second step is to initialize a set of three-dimensional Gaussian distributions using each three-dimensional point in the sparse point cloud as the mean. Each three-dimensional Gaussian distribution consists of the following parameters: position parameters. Initialized to the world coordinates of the corresponding 3D points in the sparse point cloud; covariance matrix Initialized as an isotropic matrix ,in The initial scale is a preset value (such as the value taken in a preferred embodiment). Furthermore, for ease of optimization, the covariance matrix is ​​decomposed into a rotation matrix. With scaling matrix The product form The rotation matrix is ​​represented by quaternions, and the scaling matrix is ​​represented by a three-dimensional vector; opacity Initialize to a preset value (e.g., 0.1); color parameters use a third-order spherical harmonic function (16 coefficients in total) to represent view-related color information, with coefficients initialized to zero. After initialization, the total number of 3D Gaussian distributions is equal to the number of 3D points in the sparse point cloud.

[0112] Based on the above technical solution, a sparse point cloud is generated from multi-view degraded images using the structure-of-motion method, and a 3D Gaussian distribution is initialized with the mean of each 3D point in the sparse point cloud. This initialization method utilizes multi-view geometric constraints, ensuring that the position of the initialized 3D Gaussian distribution is basically aligned with the real geometric structure of the scene. This avoids the optimization instability problem caused by random initialization and provides a reasonable parameter starting point for subsequent optimization based on perceptual consistency loss, which helps to accelerate convergence and improve the final reconstruction accuracy.

[0113] In a preferred embodiment of this example, after the projection and filtering correction of the three-dimensional Gaussian distribution to the two-dimensional Gaussian distribution are completed in the screen space rendering stage, a color accumulation rendering step is also required.

[0114] Specifically, for each pixel in the target image First, determine all two-dimensional Gaussian distributions (i.e., pixels) that overlap with the given pixel. (Within the effective coverage area of ​​the two-dimensional Gaussian distribution), filter out all pixels. After overlapping Gaussian subsets, calculate the 2D Gaussian distribution of each pixel. The Gaussian weight value at each point is obtained by multiplying the probability density function of the two-dimensional Gaussian distribution by the opacity parameter of that Gaussian element. Next, all overlapping two-dimensional Gaussian distributions are sorted in ascending order of their three-dimensional center depth (i.e., from near to far), with the farthest depth Gaussian element being... Finally, α-blending accumulation is performed in this depth order:

[0115] ;

[0116] in, For pixels The final color value, For the first The color of each Gaussian element (calculated by the spherical harmonic function based on the viewing direction). For the first A Gaussian unit in a pixel The weighted value after opacity weighting. For the front The transmittance of Gaussian units is accumulated. This accumulation method ensures correct occlusion relationship modeling, that is, nearby Gaussian units can correctly occlude distant Gaussian units, so that the rendering result conforms to the foreground and background occlusion logic of the real scene.

[0117] Based on the above technical solution, during the screen space rendering stage, all two-dimensional Gaussian distributions overlapping with the target pixel are weighted and accumulated in ascending order of depth to obtain the final color value of the target pixel. This depth-ordered α-blending accumulation method ensures correct occlusion relationship modeling, that is, nearby Gaussian units can correctly occlude distant Gaussian units, making the rendering result conform to the front-to-back occlusion logic of the real scene, further improving the geometric realism and visual coherence of the new perspective image.

[0118] like Figure 2 As shown, this embodiment also provides a three-dimensional Gaussian spatter optimization system for degraded images, which includes seven functional modules.

[0119] Data acquisition module 1 is used to acquire multi-view degraded images and corresponding camera parameters. This module is responsible for reading image sequences and camera calibration data from external storage devices or real-time acquisition devices, and performing necessary preprocessing (such as distortion correction, image size unification, etc.).

[0120] Initialization module 2 is used to initialize the parameters of a three-dimensional Gaussian distribution based on the degraded image from multiple views. This module calls a motion reconstruction tool (such as COLMAP) to generate a sparse point cloud from the degraded image and initializes a set of three-dimensional Gaussian distributions with each point in the point cloud as the mean. Each Gaussian distribution contains position, covariance matrix, opacity, and color parameters.

[0121] The perceptual consistency constraint module 3 includes a pre-trained visual feature extraction network (such as the CLIP image encoder) used to extract semantic features between the rendered image and the input image and to calculate the perceptual consistency loss during training. This module is invoked during the forward propagation in the training phase, and the output perceptual loss value participates in gradient backpropagation as part of the total loss function.

[0122] The filtering and rendering module 4 projects a 3D Gaussian distribution into a 2D Gaussian distribution in screen space, performs low-pass filtering correction on the covariance matrix of the 2D Gaussian distribution according to the current rendering scale, and then performs color accumulation rendering. This module implements the rendering workflow and is the core execution unit that ensures the visual stability of the rendering.

[0123] The adaptive density control module 5 performs splitting, cloning, or culling operations on the 3D Gaussian distribution based on the gradient magnitude and opacity value during the optimization process. Specifically, for Gaussian cells with gradient magnitudes exceeding a preset threshold, a splitting operation is performed if the cell's scale is small (splitting one Gaussian cell into two smaller Gaussian cells), and a cloning operation is performed if the cell's scale is large (copying an identical Gaussian cell and slightly shifting it along the gradient direction). For Gaussian cells with opacity below a preset threshold, a culling operation is performed to remove redundant Gaussian cells. This module enables the density of the 3D Gaussian distribution to be adaptively adjusted according to the geometric complexity of the scene during the optimization process.

[0124] Joint optimization module 6 is used to construct a total loss function by weighted fusion of perceptual consistency loss and pixel-level reconstruction loss, and to update the 3D Gaussian distribution parameters through backpropagation. This module implements the defined total loss function and uses optimizers such as Adam to iteratively update all trainable parameters.

[0125] The new perspective rendering module 7 is used to generate a 2D image of the target perspective using an optimized 3D Gaussian distribution based on the camera parameters of the target perspective. This module is called during the inference phase after model training is complete, taking as input camera parameters of any new perspective and outputting a high-quality rendered image of that perspective.

[0126] The above modules can be implemented using software, hardware, or a combination of both, and deployed on general-purpose computing devices (such as servers, workstations, and embedded devices).

[0127] Based on the above technical solution, a systematic support for each step of the above method is achieved through the collaborative work of the data acquisition module, initialization module, perceptual consistency constraint module, filtering and rendering module, adaptive density control module, joint optimization module, and new perspective rendering module. Specifically, the perceptual consistency constraint module enables the system to resist interference from degraded images; the filtering and rendering module ensures the visual stability of the rendering results; the adaptive density control module allows the density of the 3D Gaussian distribution to be adaptively adjusted according to scene complexity during the optimization process; and the joint optimization module integrates multi-level supervision signals to achieve end-to-end optimization. These modules together constitute a complete 3DGS optimization system that can still operate stably and efficiently under degraded image conditions, exhibiting good engineering practicality and ease of deployment.

[0128] In a preferred embodiment of the system in this example, the perceptual consistency constraint module uses a CLIP image encoder as the visual feature extraction network, and the perceptual consistency loss uses cosine similarity to measure the consistency between the rendered image and the input image in the CLIP feature space; in the filtering rendering module, low-pass filtering correction is implemented in the following way:

[0129] ;

[0130] in, The original covariance matrix of a two-dimensional Gaussian distribution. This is the corrected covariance matrix. These are the filtering parameters related to the current rendering resolution. It is an identity matrix.

[0131] At the system level, the CLIP image encoder, as the core component of the perceptual consistency constraint module, extracts image features and calculates loss values ​​during the training phase. Its parameters remain frozen during training (i.e., they do not participate in gradient updates) and are used only as a feature extractor, thus not introducing additional trainable parameters. The covariance correction operation of the filtering rendering module is entirely based on mathematical calculations and does not involve network inference, resulting in extremely low computational overhead. The lightweight design of these two core modules enables the entire system to improve the quality of degraded image reconstruction while maintaining a rendering speed comparable to that of the original 3DGS.

[0132] Based on the above technical solution, at the system level, the perceptual consistency constraint module is further defined to use CLIP image encoder and cosine similarity to construct perceptual loss, and the filtering and rendering module uses... The system employs various methods to achieve low-pass filtering correction. These specific implementations enable the system to work synergistically across two core dimensions: semantic awareness and scale-adaptive filtering. The CLIP encoder provides semantic guidance to combat degradation interference, while multi-scale filtering offers rendering stability by suppressing artifacts. These two approaches complement each other, allowing the system to maintain excellent reconstruction performance under various degradation types and intensities. Furthermore, the CLIP encoder's use only during the training phase and the fact that filtering correction is based on pure mathematical calculations ensure that the system's rendering speed during the inference phase is essentially equivalent to that of the original 3DGS, demonstrating good real-time performance.

[0133] Example 2

[0134] This embodiment uses the "park" scene in the HydroViews real rainy day dataset as an example to explain in detail the specific implementation process of this method under degradation conditions such as raindrop occlusion and local blurring.

[0135] Step S1: Acquire multi-view degraded image data.

[0136] This embodiment acquires a multi-view image sequence of the "park" scene from the HydroViews dataset. The scene contains approximately 300 frames, each with a resolution of 1920×1080 pixels. The images exhibit obvious degradation features such as raindrop occlusion, localized motion blur, and decreased contrast, classifying it as a typical complex degradation scene.

[0137] Simultaneously, the camera intrinsic parameters and pose parameters corresponding to each frame are obtained. Camera intrinsic parameters include focal length (…). , ), principal point coordinates ( , The parameters include the rotation matrix R and translation vector T in the world coordinate system. These parameters are provided by the dataset or pre-calculated using motion structure recovery tools such as COLMAP.

[0138] To ensure the objectivity of the evaluation, one image out of every eight images is selected as the test set, and the rest are used as the training set. In this embodiment, the training set contains approximately 260 images, and the test set contains approximately 40 images.

[0139] Step S2: Initialize the 3D Gaussian scene representation.

[0140] The COLMAP tool is used to extract features, match features, and incrementally reconstruct images from the training set to obtain a sparse point cloud of the scene. In this embodiment, the sparse point cloud contains approximately 15,000 three-dimensional points.

[0141] Initialize a set of 3D Gaussian distributions using each 3D point in the sparse point cloud as the mean. Each Gaussian distribution consists of the following parameters:

[0142] Position parameters : Initialize to three-dimensional coordinates in a sparse point cloud.

[0143] covariance matrix Initialize as an isotropic Gaussian distribution, i.e. ,in As the initial scale, this embodiment takes... To facilitate optimization, the covariance matrix is ​​decomposed into a rotation matrix. With scaling matrix The product form: .

[0144] Opacity Initialize to 0.1.

[0145] Color parameters: The view-related color information is represented by a third-order spherical harmonic function (16 coefficients in total), and the coefficients are initialized to 0.

[0146] After initialization, the total number of three-dimensional Gaussian distributions is 15,000.

[0147] Step S3: Introduce perceptual consistency constraints for optimization.

[0148] This embodiment introduces a perceptual consistency constraint based on the CLIP model on the basis of the original 3DGS training framework.

[0149] (1) CLIP feature extraction

[0150] A pre-trained CLIP model (ViT-B / 32 architecture) is used as the visual feature extraction network. The model contains a 12-layer Transformer encoder, with an output feature dimension of 512.

[0151] In each training iteration, perform the following operations:

[0152] The target view image obtained by rendering the current three-dimensional Gaussian distribution. The input is fed into the CLIP image encoder to extract the feature vector from the last layer output, and then... Normalization yields This input image does not require resizing. The CLIP image encoder contains an adaptive pooling layer that can process input images of any resolution, so the original degradation features will not be destroyed or altered due to resizing.

[0153] Inputting the real degraded image I from the same viewpoint into the same encoder yields... .

[0154] (2) Calculation of perceived consistency loss

[0155] Cosine similarity is used to measure the consistency between two images in the CLIP feature space. Perceptual consistency loss is employed. Defined as:

[0156] ;

[0157] in This represents the cosine similarity function. When two images are completely identical at the perceptual level... When the perceptual features of the two are orthogonal, .

[0158] It should be noted that, It does not require the model to accurately reproduce raindrop noise or local blurred details in the degraded image, but encourages the rendering results to maintain consistency with the input image in terms of object outlines, scene layout, and overall appearance distribution. This feature effectively suppresses the model's overfitting to degradation information.

[0159] (3) Construction of total loss function

[0160] Original 3DGS loss function It is composed of a weighted average of pixel-level L1 loss and structural similarity loss:

[0161] ;

[0162] In this embodiment, Taking 0.2, L1 loss and The weights for the losses were 0.8 and 0.2, respectively.

[0163] Introduce the perceived consistency loss as an additional constraint term into the total loss function:

[0164] ;

[0165] In this embodiment, the perceived consistency loss weight The value is set to 0.1. This value was determined experimentally: when When the value is too small (e.g., less than 0.01), the perceptual constraint effect is not obvious, and the model still suffers from optimization instability under degraded images; when... When the value is too large (e.g., greater than 0.5), the perceptual loss dominates the training process, causing the model to overemphasize the overall structure and neglect local details, resulting in overly smooth rendering. Experiments show that... A good balance can be achieved within the range of 0.05 to 0.2.

[0166] Step S4: Perform multi-scale filtering rendering.

[0167] After projecting the three-dimensional Gaussian distribution onto the two-dimensional screen space, this embodiment introduces a scale-dependent low-pass filtering mechanism to suppress aliasing artifacts and edge instability.

[0168] (1) Two-dimensional Gaussian projection

[0169] Assume a three-dimensional Gaussian distribution The projection transformation matrix from the current perspective is: (From world coordinate system to camera coordinate system) and (The Jacobian matrix from camera coordinates to ray coordinates). Its two-dimensional covariance matrix in screen space. The calculation is as follows:

[0170] ;

[0171] In actual rendering, only need to focus on The 2×2 top-left submatrix is ​​used to describe the elliptical shape of the Gaussian distribution in screen space.

[0172] (2) Scale correlation filtering

[0173] To avoid insufficient sampling due to an excessively small Gaussian projection, and to suppress high-frequency noise in the degraded image, this embodiment... Make corrections:

[0174] ;

[0175] in It is a 2×2 identity matrix. These are the filtering parameters related to the current rendering scale.

[0176] The calculation method is as follows:

[0177] ;

[0178] In the formula and These are the height and width (in pixels) of the currently rendered image, respectively. This is an empirical constant (unit: pixels). In this embodiment, for a resolution of 1920×1080, =1080, take =300, then ≈0.278.

[0179] This design allows the filtering strength to increase as the rendering resolution decreases. When the rendered image resolution is low, a single pixel covers a larger spatial area, requiring stronger filtering to suppress aliasing; when the resolution is high, the filtering strength decreases to preserve more detail.

[0180] (3) Color accumulation

[0181] Corrected two-dimensional covariance matrix Used to calculate the position of each pixel Gaussian weights at each location:

[0182] ;

[0183] in Here is the opacity parameter for Gaussian distributions. Let be the projection coordinates of the Gaussian center in screen space.

[0184] Finally, the pixels are sorted in increasing depth order. The final color of a pixel is obtained by weighting and accumulating all overlapping Gaussian values.

[0185] ;

[0186] This accumulation process is consistent with the original 3DGS, ensuring correct occlusion relationship modeling.

[0187] Step S5: Joint training and parameter update.

[0188] This embodiment uses the Adam optimizer for parameter optimization, with the learning rate set as follows:

[0189] Gaussian position Initial learning rate: 1.6 × 10 -4 An exponential decay strategy is adopted, which decays to 0.99 times every 1000 iterations.

[0190] Covariance parameter (rotation quaternion) Scaling vector Initial learning rate 5×10 -3 .

[0191] Opacity Initial learning rate 5×10 -2 .

[0192] Spherical harmonic color coefficients: initial learning rate 2.5 × 10⁻⁶ -3 .

[0193] The total number of training iterations is 30,000. Adaptive density control is performed once every 100 iterations.

[0194] During training, the PSNR metric is calculated on the test set every 1000 iterations to monitor model convergence.

[0195] Step S6: Generate a new perspective image.

[0196] After training is complete, input the camera parameters of any unseen viewpoint in the test set, and execute the rendering process in step S4 using the optimized 3D Gaussian distribution to generate a high-quality 2D image from that viewpoint.

[0197] The experimental results of this embodiment on the HydroViews dataset "park" scenario are shown in Table 1 below:

[0198]

[0199] Table 1

[0200] Based on Table 1 above, it can be seen that the method of this invention achieves better rendering quality than the original 3DGS at 30,000 iterations in only 7,000 iterations. PSNR is improved by 2.68dB, SSIM by 0.07, and LPIPS by 0.02. Figures 3 to 5 In terms of visual effects, the original 3DGS rendering results have obvious raindrop artifacts and edge blurring, while the method of the present invention effectively suppresses the above problems, and the rendered image is clearer and more stable.

[0201] Example 3

[0202] This embodiment uses the "Curasao" scene in the SeaThru underwater dataset as an example to illustrate the implementation process of this method under degradation conditions such as color attenuation and contrast reduction caused by underwater light absorption and scattering.

[0203] This embodiment uses a multi-view image sequence of the "Curasao" scene from the SeaThru dataset. This scene contains approximately 200 frames, each with a resolution of 1920×1080 pixels. The images exhibit typical underwater degradation characteristics such as a noticeable blue-green tint, reduced contrast, and blurred details of distant objects.

[0204] The training / test split of the dataset follows the original protocol, with approximately 160 images used for training and 40 images used for testing. Camera parameters are provided by the dataset.

[0205] Using the same method as in Example 2, sparse point clouds were recovered from the degraded underwater image using COLMAP. In this example, the sparse point cloud contained approximately 8,000 three-dimensional points, which is less than the number in Example 2. This is because the weakening of texture information in the underwater image makes feature matching difficult.

[0206] A 3D Gaussian distribution was initialized with sparse point clouds, and the parameter settings were the same as in Example 2.

[0207] Compared to Example 2, the underwater degradation in this example is more severe, and the degradation types are mainly color shift and contrast reduction, rather than local noise. Therefore, adaptive adjustments are made to the perceptual consistency constraint:

[0208] (1) The feature extraction network remains unchanged and still uses CLIP ViT-B / 32. This network is pre-trained on large-scale image and text data, has a certain robustness to color shift, and can extract semantic features related to object shape and scene layout, rather than being limited by low-level color statistics.

[0209] (2) Adjustment of perceptual loss weights: In this embodiment, the perceptual loss weights are adjusted as follows: The efficiency was increased from 0.1 to 0.2. This is because reliable pixel-level supervision signals are scarcer in underwater images, requiring stronger perceptual constraints to guide the optimization direction. Experiments show that... When the coefficient is 0.2, the model converges more stably; if the coefficient is kept constant at 0.1, the model may still experience gradient oscillations in the early iterations.

[0210] (3) Optional color correction preprocessing: As a variant of this embodiment, a simple white balance preprocessing can be performed on the rendered image and the real image before inputting them into the CLIP network to reduce the interference of color shift on perceptual feature extraction. This preprocessing does not affect the gradient backpropagation of the three-dimensional Gaussian parameters.

[0211] The other loss function settings are the same as in Example 2.

[0212] The multi-scale filtering rendering process in this embodiment is basically the same as that in Embodiment 2. Considering that underwater images themselves have low contrast and few high-frequency details, the filtering parameters are adjusted accordingly. It can be reduced appropriately to retain more details.

[0213] Specifically, for a 1920×1080 resolution, take =200, then ≈0.185. This value is less than 0.278 in Example 2, so the filtering strength is weakened to avoid over-smoothing.

[0214] Other parameters are the same as in Example 2.

[0215] The training parameters were set basically the same as in Example 2, except that the total number of iterations was adjusted to 20,000. This is because the underwater scene is smaller and has fewer Gaussian units, resulting in faster model convergence.

[0216] The adaptive density control parameters are consistent with those in Example 2.

[0217] The new perspective image was generated using the same method as in Example 2.

[0218] The experimental results of this embodiment on the "Curasao" scenario of the SeaThru dataset are shown in Table 2 below:

[0219]

[0220] Table 2

[0221] Based on Table 2 above, the method of this invention achieves a PSNR of 29.60 dB, which is superior to the original 3DGS (28.31 dB), Instant-NGP (27.90 dB), and the SeaThru-NeRF method designed for underwater scenarios (29.30 dB). Figures 6 to 7 In terms of visual effects, the method of the present invention effectively restores the true color of underwater objects (such as the original hue of red coral) and significantly reduces blur artifacts in distant areas.

[0222] Example 4

[0223] This embodiment illustrates the key parameters in the method of this embodiment. , The fitting relationship between the degradation value and the degradation intensity provides guidance for parameter selection in practical applications.

[0224] In multiple scenarios of the HydroViews dataset, Gaussian noise of varying degrees was manually added. Test data with different degradation intensities were constructed using kernel sizes (0.05, 0.10, 0.20) and motion blur (kernel size = 3, 5, 7).

[0225] Test separately ={0.05,0.1,0.2,0.3} and Scaling factor The combination of {100, 200, 300, 400} is evaluated using PSNR and the number of convergence iterations as metrics.

[0226] (1) Adaptation rules

[0227] When the degradation intensity is low ( When ≤0.05, fuzzy kernel ≤3, A value of 0.05 to 0.1 is recommended. Too large a value... (≥0.2) will cause the rendering result to be overly smooth, resulting in a loss of detail.

[0228] When the degradation intensity is moderate (0.05 < When ≤0.15, 3<fuzzy kernel≤5, A value of 0.1 to 0.2 is preferable. This range strikes a balance between suppressing degradation interference and preserving true details.

[0229] When the degradation intensity is high ( When the value is >0.15 and the fuzzy kernel is >5, A value of 0.2 to 0.3 is recommended. At this value, the pixel-level monitoring signal is severely distorted, requiring stronger perceptual constraints to guide the optimization direction.

[0230] (2) Adaptation rules

[0231] For high-resolution rendering (≥1080p) or scenes with rich detail, take =200~300 ( (≈0.19~0.28), the filtering strength is moderate to preserve high-frequency details.

[0232] For low-resolution rendering (<720p) or scenes with sparse textures, take =300~400 ( (≈0.28~0.37), the filtering intensity is enhanced to suppress aliasing artifacts.

[0233] For severely degraded images (such as those with strong noise or large blur kernels), the size can be appropriately increased. Values ​​(≥400) are used to achieve a stronger smoothing effect.

[0234] (3) Engineering recommendations

[0235] In practical applications, an adaptive parameter configuration strategy is recommended:

[0236] ,in Normalized degradation intensity assessment value (0≤ ≤2).

[0237] That is, the filtering strength increases as degradation increases.

[0238] The above-described adaptive strategy can be used as a preferred embodiment of the present invention to improve the robustness of the method under different degradation conditions.

[0239] The lightweight 3DGS optimization method for degraded images proposed in this invention has the following differences and technical advantages compared with traditional 3DGS methods:

[0240]

[0241] In summary, the proposed 3D Gaussian spatter optimization method and system for degraded images, compared to traditional 3DGS methods, adds CLIP perceptual consistency loss to the supervision signal, enabling the model to focus on the overall scene structure rather than degraded details; in terms of rendering stability, multi-scale filtering effectively suppresses edge artifacts and floating objects; in terms of convergence speed, only 7000–20000 iterations are needed to achieve rendering quality superior to the traditional method's 30000 iterations; in terms of scale generalization capability, adaptive filtering achieves good adaptation to different resolutions; and in terms of computational overhead, it increases training time by less than 5% without affecting inference speed. This invention can be widely applied to practical scenarios requiring 3D reconstruction and viewpoint synthesis under degraded imaging conditions, such as intelligent driving, underwater exploration, security monitoring, and virtual reality.

[0242] Those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no structural conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A three-dimensional Gaussian spatter optimization method for degraded images, characterized in that, Includes the following steps: Acquire multi-view degraded images and corresponding camera parameters; The three-dimensional Gaussian distribution parameters are initialized based on the multi-view degraded image. The three-dimensional Gaussian distribution parameters include position parameters, covariance matrix parameters, opacity parameters, and color parameters. During the training iteration, the rendered image obtained by the current 3D Gaussian distribution rendering and the input image are respectively input into the pre-trained visual feature extraction network to extract their respective semantic feature maps, and a perceptual consistency loss is constructed based on the similarity between the semantic feature maps. During the screen space rendering stage, the three-dimensional Gaussian distribution is projected into a two-dimensional Gaussian distribution, and the covariance matrix of the two-dimensional Gaussian distribution is low-pass filtered and corrected according to the current rendering scale. The perceptual consistency loss and pixel-level reconstruction loss are weighted and fused to construct a total loss function, and the three-dimensional Gaussian distribution parameters are updated using the total loss function; The above training steps are executed iteratively until the convergence condition is met. The camera parameters of the target viewpoint are input, and the optimized three-dimensional Gaussian distribution is used to render and generate a two-dimensional image of the target viewpoint.

2. The three-dimensional Gaussian spatter optimization method for degraded images according to claim 1, characterized in that, The visual feature extraction network is a CLIP image encoder, and the perceptual consistency loss uses cosine similarity to measure the consistency between the rendered image and the input image in the CLIP feature space, specifically: ； in, For the rendered image, The input image, The feature vectors extracted and L2-normalized by the CLIP image encoder This is the cosine similarity function.

3. The three-dimensional Gaussian spatter optimization method for degraded images according to claim 1, characterized in that, The process of projecting a three-dimensional Gaussian distribution into a two-dimensional Gaussian distribution specifically involves: Projection transformation matrix based on the current viewpoint And Jacobi matrix Calculate the two-dimensional covariance matrix in screen space. ,in The three-dimensional covariance matrix is ​​a three-dimensional Gaussian distribution. The low-pass filter correction is achieved in the following way: ； in, This is the corrected covariance matrix. The filtering parameters are related to the current rendering scale. It is an identity matrix.

4. The three-dimensional Gaussian spatter optimization method for degraded images according to claim 3, characterized in that, The filter parameters Determined based on the resolution of the currently rendered image: ； in, and These are the height and width of the currently rendered image, respectively. This is a preset constant.

5. The three-dimensional Gaussian spatter optimization method for degraded images according to claim 1, characterized in that, The total loss function is: ； in, For the pixel-level reconstruction loss, This refers to the perceived consistency loss. The weighting coefficients for perceived consistency loss.

6. The three-dimensional Gaussian spatter optimization method for degraded images according to claim 5, characterized in that, The pixel-level reconstruction loss Depend on The loss is weighted by the structural similarity loss: ； in, Pixel level loss, For structural similarity loss, represents the weighting coefficients for structural similarity loss.

7. The three-dimensional Gaussian spatter optimization method for degraded images according to claim 1, characterized in that, The initialization of the three-dimensional Gaussian distribution parameters based on the multi-view degraded image specifically includes: A sparse point cloud is generated from the multi-view degraded image using the structure-reconstruction-motion method; Initialize a set of three-dimensional Gaussian distributions using each three-dimensional point in the sparse point cloud as the mean; The position parameters of each 3D Gaussian distribution are initialized to the coordinates of the corresponding 3D point, the covariance matrix is ​​initialized to an isotropic matrix, the opacity is initialized to a preset value, and the color parameters are initialized to zero.

8. The three-dimensional Gaussian spatter optimization method for degraded images according to claim 1, characterized in that, The process of projecting the three-dimensional Gaussian distribution into a two-dimensional Gaussian distribution during the screen space rendering stage also includes: The final color value of the target pixel is obtained by weighting and accumulating all two-dimensional Gaussian distributions that overlap with the target pixel in ascending order of depth.

9. A three-dimensional Gaussian spatter optimization system for degraded images, characterized in that, include: The data acquisition module is used to acquire multi-view degraded images and corresponding camera parameters; An initialization module is used to initialize the three-dimensional Gaussian distribution parameters based on the multi-view degraded image; The perceptual consistency constraint module includes a pre-trained visual feature extraction network, which is used to extract semantic features of the rendered image and the input image and calculate the perceptual consistency loss. The filtering rendering module is used to project a three-dimensional Gaussian distribution into a two-dimensional Gaussian distribution in screen space, and to perform low-pass filtering correction on the covariance matrix of the two-dimensional Gaussian distribution according to the current rendering scale, and to perform color accumulation rendering. An adaptive density control module is used to perform splitting, cloning, or culling operations on a three-dimensional Gaussian distribution based on the gradient magnitude and opacity value during the optimization process. The joint optimization module is used to weightedly fuse the perceptual consistency loss and the pixel-level reconstruction loss to construct a total loss function, and to update the three-dimensional Gaussian distribution parameters through backpropagation. The new perspective rendering module is used to generate a 2D image of the target perspective by rendering it using an optimized 3D Gaussian distribution based on the camera parameters of the target perspective.

10. The three-dimensional Gaussian spatter optimization system for degraded images according to claim 9, characterized in that, In the perceptual consistency constraint module, the visual feature extraction network is a CLIP image encoder, and the perceptual consistency loss uses cosine similarity to measure the consistency between the rendered image and the input image in the CLIP feature space; in the filtering rendering module, the low-pass filtering correction is implemented in the following way: ； in, Let be the original covariance matrix of the two-dimensional Gaussian distribution. This is the corrected covariance matrix. These are the filtering parameters related to the current rendering resolution. It is an identity matrix.

Images