Gaussian sputtering general rendering enhancement method based on space fusion

Abstract

The invention discloses a Gaussian sputtering general rendering enhancement method based on space fusion. The method comprises the following steps: obtaining a real image to be processed; constructing a degradation simulator of a Gaussian sputtering style, adding a gating mechanism into the degradation simulator, selecting a degradation type in an image degradation process, and generating a training data set; inputting a real image to be processed into the degradation simulator to generate a training image; and a Gaussian intensifier is constructed, the Gaussian intensifier utilizes a spatial information fusion network to carry out feature fusion and enhancement rendering processing on the training image through a view fusion module and a depth modulation module, and a final enhancement result is generated. According to the method, highly-correlated and high-quality training images can be mixed, and details are rendered by using the depth information of the target image, so that the output rendering quality of the image is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image rendering, and in particular relates to a Gaussian sputtering universal rendering enhancement method based on space fusion. Background Art

[0002] Novel View Synthesis refers to the task of rendering a target image given a source image, source pose, and target pose. Novel view synthesis has a wide range of applications in 3D reconstruction, AR / VR, and other fields. As users increasingly seek realistic and immersive experiences, end users strongly demand efficient and high-quality methods to synthesize novel views. The emergence of Neural Radiance Fields (NeRF) has revolutionized the way novel view synthesis is handled, leading to significant progress in this field. However, there are still some challenges in terms of slow training and rendering speed. To achieve real-time novel view synthesis, 3D Gaussian Sputtering (3DGS) uses a more efficient rasterization rendering technique, deviating from the ray tracing method of NeRF, and has become a transformative technology in the field of explicit rendering and computer graphics. However, GS-based methods still face challenges in rendering high-quality image details. Even when using high-quality training frameworks, their outputs often exhibit severe rendering artifacts such as noise and blurriness.

[0003] To address the above problems, it is urgent to propose a universal rendering enhancement method based on spatial fusion Gaussian sputtering. Summary of the invention

[0004] To solve the above technical problems, the present invention proposes a Gaussian sputtering universal rendering enhancement method based on spatial fusion, adopts a universal Gaussian enhancer, that is, a network-independent post-processor, and adopts a degradation-driven view mixing method to improve the rendering quality of the GS model, so as to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above object, the present invention provides a Gaussian sputtering universal rendering enhancement method based on spatial fusion, comprising the following steps:

[0006] Obtain the real image to be processed;

[0007] Constructing a Gaussian sputtering style degradation simulator, adding a gating mechanism to the degradation simulator, selecting the degradation type in the image degradation process, and generating a training data set;

[0008] Inputting the real image to be processed into the degradation simulator to generate a training image;

[0009] A Gaussian enhancer is constructed. The Gaussian enhancer uses a spatial information fusion network to perform feature fusion and enhanced rendering processing on the training image through a view fusion module and a depth modulation module to generate a final enhanced result.

[0010] Optionally, the degradation types in the training dataset include: blur, noise, downsampling and artifacts.

[0011] Optionally, before the view fusion module and the depth modulation module respectively perform feature fusion and enhanced rendering on the training image, the process further includes:

[0012] According to the different viewpoint positions of the training images, the degraded target frame and two adjacent supporting frames are filtered out, and then the degraded target frame and the two adjacent supporting frames are input into the view fusion module.

[0013] Optionally, the process of performing feature fusion on the training image based on the view fusion module includes: extracting features from the degraded target frame and two adjacent supporting frames, and then using deformable convolution to fuse cross-view features, and reconstructing to generate an enhanced view of the target frame.

[0014] Optionally, the enhanced view of the target frame is shown in the following formula:

[0015] H t =V(G t-1 ,D(G t ),G t+1 )

[0016] =V(G t-1 ,L t ,G t+1 )

[0017] Among them, L t represents the degraded target frame image, corresponding to the ground truth value G t , G t-1 and G t+1 represents the ground truth of two adjacent views relative to the target frame, H t is the enhanced view of the target frame, function D is used to simulate the degradation of the ground truth, and function V is used to fuse multiple frames.

[0018] Optionally, the process of performing enhanced rendering on the training image based on the depth modulation module includes: acquiring depth information corresponding to the degraded target frame, and under the guidance of the depth information, performing enhanced rendering on the enhanced view of the target frame to obtain a final enhanced result.

[0019] Optionally, the final enhancement result is shown in the following formula:

[0020] E t =S(Ht ,D t )

[0021] Among them, E t For the final enhancement result, H t is the enhanced view of the target frame, D t It represents the depth information corresponding to the target frame, and the function S performs depth modulation under the guidance of the depth information.

[0022] Optionally, the loss function corresponding to the view fusion module and the depth modulation module is as follows:

[0023] L c =H t -I gt

[0024] L d =E t -I gt

[0025] The total loss function of the spatial information fusion network is:

[0026] L total =L c +λ d L d

[0027] Among them, λ d is the trade-off parameter of the total loss, I gt is the true value of the image, H t is the enhanced view of the target frame, E t is the final enhanced result.

[0028] Compared with the prior art, the present invention has the following advantages and technical effects:

[0029] We also propose a GS-style degradation simulator and construct a large-scale training dataset to effectively simulate the native rendering artifacts of GS, thereby achieving efficient training.

[0030] The present invention constructs a Gaussian enhancer, utilizes a spatial information fusion network, and performs feature fusion and enhanced rendering processing on training images through a view fusion module and a depth modulation module, respectively, which can mix highly relevant and high-quality training images, and use the depth information of the target image to complete the rendering details, thereby improving the output rendering quality of the image. Through the Gaussian enhancer of the present invention, the rendering effect of the GS model can be effectively eliminated, and a highly realistic synthetic view can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1 Schematic diagram of the process of the Gaussian sputtering universal rendering enhancement method based on spatial fusion according to an embodiment of the present invention, (a) is a complete process diagram, (b) is a Gaussian enhancer processing process diagram, and (c) is a GS style degradation simulator processing process diagram. DETAILED DESCRIPTION

[0033] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0034] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] In related technologies, in recent years, the emergence of Neural Radiance Fields (NeRF) has pushed the task of novel view synthesis to a new height, completely changing the way of representing and synthesizing complex 3D scenes. However, there are still some challenges in terms of slow training and rendering speed. In order to achieve real-time novel view synthesis, Gaussian Sputtering (GS) methods, including 3DGS and 2DGS, adopt a more efficient rasterization rendering technology, which shows excellent performance in real-time novel view synthesis (NVS) and becomes a transformative technology in the field of explicit rendering and computer graphics.

[0036] Subsequent work has further studied anti-aliasing, reducing memory usage, replacing spherical harmonics, enhancing the ability to model high-frequency signals based on reflective surfaces, and dynamic scene modeling, and has been widely used in advanced tasks such as human / avatar modeling, surface reconstruction, inverse rendering, and physical simulation. However, although rasterization allows Gaussian sputtering to avoid difficult sampling problems along rays and achieves good results, it is still prone to problems such as aliasing, blurring, and needle artifacts, and still faces challenges in rendering high-quality image details.

[0037] In view of the above problems, this embodiment proposes a Gaussian sputtering universal rendering enhancement method based on spatial fusion, and designs a Gaussian enhancer, a universal network-independent post-processor that can directly enhance the synthetic views generated by various GS models. Based on the Gaussian enhancer, spatial information fusion technology is used to mix highly relevant and high-quality training images, and the depth information of the target image is used to complete the rendering details, thereby improving the output rendering quality based on the GS model.

[0038] To achieve this, this embodiment also proposes a typical GS-style degradation simulator, which is crucial to simulate its degradation mechanism because previous studies have not used GS-style degradation for image and video restoration.

[0039] Degradation models and image super-resolution (SR) have been a long-standing research topic in the field of image processing. The most common and simplest degradation model is bicubic downsampling, which is widely used in many non-blind SR methods, including simple downsampling and Gaussian blur-based downsampling. Since Dong et al. introduced convolutional neural networks (CNNs) to the SR field, many subsequent works have achieved impressive reconstruction performance on LR images created by bicubic downsampling, including residual networks, dense connections, channel attention, residual in residual dense blocks, and transformer structures. Although these methods have made significant progress, the models are too simple and the performance of non-blind methods may drop significantly when applied to images with different degradation levels. Many existing SR methods based on deep neural networks rely on bicubic downsampling and traditional degradation models or their simple variants. Some blind SR methods have also been proposed for unknown degradation situations. Ji et al. modeled the degradation process as a blur kernel and noise pool learned from test images, which were then used to synthesize training samples for the SR model. To simulate the random factors in the degradation process, they concatenated random vectors to the high-resolution image before degradation and input them to the neural network. In KMSR, a pool of kernels is constructed from real-world photos using a generative adversarial network, which is then used to synthesize more realistic training data. Although blind SR methods have made progress, most of them are trained using a pre-collected pool of kernels, which makes them not truly blind and difficult to generalize to real-world images. Several works have been proposed to improve the generalization ability of SR networks in real-world scenarios, including building blind degradation models based on multiple degradation factors such as Gaussian blur, noise, and JPEG compression. Significant progress has been made through various methods, including using a single SR network with multiple degradation levels, kernel estimation, and representation learning. BSRGAN and Real ESRGAN expand the degradation modeling space and enhance variability by combining comprehensive degradation types and randomly sampled degradation parameters. Gu et al. proposed a kernel estimation method based on iterative correction and introduced DAN and DASR to further improve blind SR results. Other classic blind SR works include zero-shot learning, meta-learning, optimization methods, real datasets, and unsupervised methods. Zhang et al. introduced a more practical degradation model that includes multiple blur types such as occlusion blur, downsampling, camera sensor noise, and JPEG compression, and combines a random shuffling strategy. However, there is no work specifically targeting GS degradation.

[0040] To address the above issues, this application proposes a GS-style degradation simulator and constructs a large-scale training dataset for modeling GS rendering degradation, such as needle artifacts. A degradation-driven spatial information fusion framework is also proposed, consisting of a view fusion module and a depth modulation module. In order to make full use of the effective information from adjacent frames, view selection is first performed to filter out high-quality adjacent images, and then these images are input into the view fusion module. At the same time, the depth information of the target image is also used to complete the rendering details.

[0041] like Figure 1 As shown in (c), the GS-style degradation simulator in this embodiment adds a gating mechanism to randomly select basic degradation types during the degradation process, generating a recognizable training dataset similar to the degradation style of GS rendered images. According to traditional degradation models, the three main factors affecting real image degradation are blur, noise, and downsampling. This embodiment designs various degradation types for the target view, taking into account 2DGS, 3DGS and their derivative methods, and incorporating the degradation style unique to the GS model, such as needle artifacts, into the simulation process.

[0042] Blur: Blur is a common image degradation, which also appears in GS rendering results. In order to simulate the blur of the GS model, this embodiment applies isotropic and anisotropic Gaussian kernels to simulate the blur of the target frame.

[0043] Noise: Noise is ubiquitous in real images as it can be caused by various sources. Due to the high storage cost of 3DGS, many methods have adopted compression techniques, which inevitably lead to compression noise in JPEG images. In addition, this embodiment also simulates sputtered Gaussian noise, which is another important factor in GS degradation, and camera sensor noise obtained by processing raw sensor data through the image signal processing (ISP) pipeline.

[0044] Downsampling: Different downsampling methods can significantly affect the performance of 3DGS. Therefore, this embodiment uses downsampling as a degradation factor in GS style degradation. In order to maintain the same resolution, this embodiment tries upsampling, downsampling, downsampling, upsampling, and uses bilateral, bicubic and other methods respectively.

[0045] Artifacts: Compared with NeRF and other popular super-resolution models, 2DGS and 3DGS exhibit obvious needle-like artifacts in some datasets, such as in train scenes. In addition, the rendered images of 2DGS, especially in unbounded scenes, show visible banding artifacts. In order to solve the artifacts unique to the GS model, this embodiment implements the relevant degradation design. In addition to these artifacts, other typical artifacts such as cloud-like artifacts and floating artifacts also exist.

[0046] Gating mechanism: The degradation level in the GS model varies in different datasets. When designing a degradation model, if we only consider the complex degradation situation in the degradation process where all or most of the basic degradation types are used in the degradation process, it is often not possible to accurately simulate the degradation situation of the real GS model. Therefore, the GS style degradation simulator proposed in this embodiment has a gating mechanism that randomly selects the basic degradation types included in the degradation process.

[0047] like Figure 1 As shown in (b), the spatial information fusion network in this embodiment is composed of two main modules: a view fusion module and a depth modulation module. By integrating these modules, this embodiment can complete the details of the target frame, thereby improving the overall visual quality.

[0048] Specifically, in order to maximize the use of effective information from adjacent frames, this embodiment first filters out the two high-quality support frames closest to the target frame according to the different viewpoint positions of the training images, and then inputs them into the view fusion module. Then, the view fusion component extracts the degraded target frame (L t ) and two adjacent support frames (G t-1 , G t+1 ), then deformable convolution is used to fuse cross-view features, and finally a reconstruction module is used to generate an enhanced view of the target frame (H t ):

[0049] H t =V(G t-1 ,D(G t ),G t+1 )

[0050] =V(G t-1 ,L t ,G t+1 )

[0051] Among them, L t represents the degraded target frame image obtained by the GS style degradation simulator, corresponding to the ground truth value G t . G t-1 and G t+1 represents the ground truth of two adjacent views relative to the target frame, and H t is the enhanced view of the target frame produced by the view fusion process. Function D is used to model the degradation of the ground truth, while function V is responsible for fusing multiple frames.

[0052] By integrating deformable convolutions into the sampling function, the present embodiment is able to effectively delve into complex local temporal contexts. This feature enables the view fusion module of the present embodiment to elegantly adapt to substantial movement between multiple frames. In addition, in combination with deformable convolutions, the present embodiment learns offsets as sampling parameters to implicitly capture forward and backward motion information. These offsets facilitate precise alignment, ensuring temporal consistency and smooth transitions. Through these techniques, the present embodiment is able to generate high-quality enhanced views of the target frame.

[0053] In addition to view fusion, this embodiment also uses a depth modulation module to further enhance the visual effect of the target frame. After view fusion, an enhanced view of the target frame is obtained. Inspired by the excellent geometric structure of 2DGS, this embodiment believes that the depth information of the GS model can be fully utilized to improve the rendering results. By utilizing the depth information obtained during the rendering process, this embodiment can provide strong guidance for the three-dimensional structure of the target frame, thereby further improving the details of objects close to the view plane:

[0054] E t =S(H t ,D t )

[0055] Among them, E t is the final enhanced result after SIF processing, D t Represents the depth information corresponding to the target frame. Under the guidance of the depth information, function S performs depth modulation on the fused image to complete the image details.

[0056] This embodiment uses two loss functions, one is the fusion loss in the view fusion stage, and the other is the final loss in the depth modulation stage. The calculation method of the fusion loss and the final loss is as follows:

[0057] L c =H t -I gt

[0058] L d =E t -I gt

[0059] The total loss function of the entire network is:

[0060] L total =L c +λ d L d

[0061] Among them, λ d is the trade-off parameter of the total loss, I gt is the true value of the image.

[0062] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A Gaussian sputtering general rendering enhancement method based on spatial fusion, characterized in that: The following steps are involved: Obtain the real image to be processed; Constructing a Gaussian sputtering style degradation simulator, adding a gating mechanism to the degradation simulator, selecting the degradation type in the image degradation process, and generating a training data set; Inputting the real image to be processed into the degradation simulator to generate a training image; A Gaussian enhancer is constructed. The Gaussian enhancer uses a spatial information fusion network to perform feature fusion and enhanced rendering processing on the training image through a view fusion module and a depth modulation module to generate a final enhanced result.

2. The Gaussian sputtering universal rendering enhancement method based on spatial fusion according to claim 1 is characterized in that: The types of degradation in the training dataset include: blur, noise, downsampling, and artifacts.

3. The Gaussian sputtering universal rendering enhancement method based on spatial fusion according to claim 1 is characterized in that: Before performing feature fusion and enhanced rendering on the training image by the view fusion module and the depth modulation module respectively, the method further includes: According to the different viewpoint positions of the training images, the degraded target frame and two adjacent supporting frames are filtered out, and then the degraded target frame and the two adjacent supporting frames are input into the view fusion module.

4. The Gaussian sputtering universal rendering enhancement method based on spatial fusion according to claim 3 is characterized in that: The process of performing feature fusion on the training image based on the view fusion module includes: extracting features from the degraded target frame and two adjacent support frames, then fusing cross-view features using deformable convolution, and reconstructing to generate an enhanced view of the target frame.

5. The Gaussian sputtering universal rendering enhancement method based on spatial fusion according to claim 4 is characterized in that: The enhanced view of the target frame is shown as follows: H t =V(G t-1 ,D(G t ),G t+1 )=V(G t-1 ,L t ,G t+1 ) Among them, L t represents the degraded target frame image, corresponding to the ground truth value G t , G t-1 and G t+1 represents the ground truth of two adjacent views relative to the target frame, H t is the enhanced view of the target frame, function D is used to simulate the degradation of the ground truth, and function V is used to fuse multiple frames.

6. The Gaussian sputtering universal rendering enhancement method based on spatial fusion according to claim 4 is characterized in that: The process of performing enhanced rendering on the training image based on the depth modulation module includes: acquiring depth information corresponding to the degraded target frame, and under the guidance of the depth information, performing enhanced rendering on the enhanced view of the target frame to obtain a final enhanced result.

7. The Gaussian sputtering universal rendering enhancement method based on spatial fusion according to claim 6 is characterized in that: The final enhancement result is shown in the following formula: E t =S(H t ,D t ) Among them, E t For the final enhancement result, H t is the enhanced view of the target frame, D t It represents the depth information corresponding to the target frame, and the function S performs depth modulation under the guidance of the depth information.

8. The Gaussian sputtering universal rendering enhancement method based on spatial fusion according to claim 1, characterized in that: The loss function corresponding to the view fusion module and the depth modulation module is shown in the following formula: L c =||H t -I gt || THE d =||And t -THE gt || The total loss function of the spatial information fusion network is: L total =L c +λ d L d Among them, λ d is the trade-off parameter of the total loss, I gt is the true value of the image, H t is the enhanced view of the target frame, E t is the final enhanced result.

Images