3D gaussian blur rendering acceleration method based on gradient-aware supersampling and frame interpolation

Abstract

The application relates to the technical field of computer graphics, and discloses a 3D Gaussian blur rendering acceleration method based on gradient-aware supersampling and frame interpolation. The method comprises the following steps: acquiring a low-resolution frame sequence output by a 3D Gaussian blur rendering engine; processing a current low-resolution frame by using a gradient-aware supersampling module to generate a high-resolution reconstruction frame; processing continuous high-resolution reconstruction frames and low-resolution frames by using a time sequence frame interpolation module to synthesize a high-resolution intermediate frame; and outputting a rendering image sequence composed of the high-resolution reconstruction frame and the high-resolution intermediate frame. The application realizes real-time high-resolution rendering of a large-scale scene by combining supersampling and frame interpolation.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, specifically to a method for accelerating 3D Gaussian splash rendering based on gradient-aware supersampling and frame interpolation. Background Technology

[0002] 3D Gaussian Splash (3DGS), as a powerful 3D scene representation method, can achieve high-quality real-time rendering by modeling the scene as a set of Gaussian primitives. Although 3DGS has achieved real-time rendering in various fields, its performance is limited by the massive number of parameters and algorithm efficiency, especially in latency-sensitive applications such as virtual reality (VR), augmented reality (AR), and mobile devices, making it difficult to scale to ultra-large scenes or ultra-high resolution output.

[0003] To improve the rendering efficiency of 3DGS, existing research mainly focuses on the following three directions: (1) Compact representation: methods such as Generalized Exponential Splatting (GES) and Discontinuity-aware Gaussian Splatting (DisC-GS) achieve more compact representation by modifying Gaussian primitives or using boundary-aware formulas. (2) Gaussian pruning: methods such as Light Gaussian and Mini-Splatting prune redundant Gaussian primitives based on spatial distribution to reduce the number of primitives. (3) Rendering acceleration: methods such as AdR-Gaussian (Accelerating Gaussian Splatting with Adaptive Radius) and GSCore (Efficient Radiance Field Rendering via Architectural Support for 3DGaussian Splatting) accelerate rasterization through adaptive radius filtering or hardware-level optimization.

[0004] However, existing technologies face the following major obstacles: First, despite the aforementioned improvements, achieving real-time rendering remains challenging and computationally expensive when scaling to ultra-dense scenes containing millions of splashes or performing 4K ultra-high resolution rendering. Second, existing supersampling or frame generation techniques (such as DLSS, XeSS, or FSR) often suffer from quality degradation or artifacts when directly applied to 3DGS due to their inability to adapt to the inherent continuity of Gaussian representation.

[0005] In the traditional rendering field, commercial systems such as DLSS (Deep Learning Super Sampling), XeSS (Xe Super Sampling), and FSR (FidelityFX Super Resolution) can reconstruct 4K resolution or high frame rate images from low-resolution input by employing supersampling and frame interpolation techniques, thereby achieving high-quality real-time performance. This paradigm shift prompted this invention to explore whether similar methods could be effectively applied to 3DGS. However, directly integrating existing technologies into 3DGS faces significant challenges because simple integration often leads to image quality degradation and cannot adapt to the inherent continuity characteristics of Gaussian representation. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a method for accelerating 3D Gaussian splash rendering based on gradient-aware supersampling and frame interpolation. Instead of further optimizing the 3DGS rendering pipeline itself, this invention focuses on post-rendering processing and draws inspiration from modern real-time graphics pipelines to explore methods for accelerating 3DGS.

[0007] To address these challenges, this invention proposes a 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation. This is a unified framework for real-time, large-scale 3DGS rendering that jointly performs supersampling and frame interpolation. Inspired by supersampling and frame interpolation techniques in traditional rendering pipelines, this method performs differentiable post-rendering processing on low-resolution 3DGS output, thereby achieving both high resolution and high frame rate rendering without modifying the underlying rendering pipeline.

[0008] First, this invention designs a gradient-aware supersampling method that utilizes the continuous differentiability of Gaussian units to extract image gradients as auxiliary cues. These gradient features guide image interpolation and are combined with a GRU-based thinning network to efficiently reconstruct high-resolution rendering results while maintaining perceptual fidelity.

[0009] Then, this invention proposes a lightweight temporal frame interpolation method that integrates temporal information and image gradient information from consecutive frames to synthesize intermediate images, thereby significantly improving the rendering frame rate.

[0010] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A method for accelerating 3D Gaussian splash rendering based on gradient-aware supersampling and frame interpolation includes: Obtain the low-resolution frame sequence output by the 3D Gaussian Splash rendering engine; A gradient-aware supersampling module is used to process the current low-resolution frame to generate a high-resolution reconstructed frame. The gradient-aware supersampling module extracts the image gradient using the continuous differentiability of Gaussian units and guides a gradient-aware interpolation network and a geometric-temporal cyclic thinning network to perform super-resolution reconstruction based on the image gradient. A temporal frame interpolation module is used to process consecutive high-resolution reconstructed frames and low-resolution frames to synthesize high-resolution intermediate frames; wherein, the temporal frame interpolation module fuses temporal information and image gradient information to generate high-resolution frames at intermediate moments, which serve as high-resolution intermediate frames. The output consists of a sequence of rendered images composed of the high-resolution reconstructed frames and the high-resolution intermediate frames.

[0011] In one embodiment, the gradient-aware supersampling module includes: The gradient-aware interpolation network is configured to generate an initial high-resolution image based on the Hermitian interpolation principle by using the pixel intensity and corresponding spatial gradient of the low-resolution frame through convolution and transpose convolution operations. The geometric-temporal recurrent refinement network is configured to employ an improved gated recurrent unit architecture, which integrates the gradient, depth map, normal-view inner product map of the current low-resolution frame and historical features of the previous frame to generate residual features to optimize the initial high-resolution image and obtain a high-resolution reconstructed frame.

[0012] In one embodiment, the gradient-aware interpolation network is configured to generate an initial high-resolution image based on the Hermitian interpolation principle, utilizing the pixel intensity and corresponding spatial gradient of the low-resolution frame through convolution and transpose convolution operations, specifically including: Given a low-resolution frame pixels And the 8 adjacent pixels form a group of size. Consider interpolation node blocks that satisfy the following conditions for bivariate polynomials. : ; ; ; in, ; For a given oversampling factor S, let the low-resolution frame contain elements that are... Centered Interpolation node blocks correspond to high-resolution images China and Israel Centered on, size is The image patch, where, , , These are the corresponding center coordinates in high-resolution space; High-resolution image obtained by interpolation In this context, the value of each pixel is calculated using a local polynomial at its corresponding spatial location, i.e. And polynomial In Centered The interpolation node is determined independently within the block; therefore... It is considered as a sampling of a continuous intensity surface on a high-resolution mesh, which is reconstructed block by block by a series of local polynomials; Each pixel within an image block is represented as a linear combination of the pixel intensity and spatial gradient of that block in the low-resolution frame, i.e.: ; in, This indicates the column vectorization operator. For Centered Interpolation node blocks; For high-resolution images and The corresponding size is The image patch; M is a fixed coefficient matrix determined by the interpolation conditions; The process of generating high-resolution images using local polynomials described above can be achieved through cascaded convolutional layers and transposed convolutional layers: First, through convolutional layers From the input low-resolution frame Extract the combined features of corresponding pixel intensity and spatial gradient. : ; Through transposed convolutional layers By performing upsampling and local weighted synthesis on a high-resolution grid, a high-resolution image is obtained. ; .

[0013] In one embodiment, the geometry-temporal recurrent thinning network is configured to employ an improved gated recurrent unit architecture, fusing the gradient, depth map, normal-view inner product map of the current low-resolution frame and historical features of the previous frame to generate residual features to optimize the initial high-resolution image, thereby obtaining a high-resolution reconstructed frame, specifically including: The geometric-temporal recurrent refinement network employs an improved gated recurrent unit architecture; the input to the geometric-temporal recurrent refinement network includes: the current low-resolution frame. The gradient map G, depth map D, and normal-view inner product map generated by transparent blending in the 3D Gaussian splash rendering process. Distorted historical features after perspective alignment ; Historical features output from preceding time gait It is obtained by back-projecting the depth map and camera pose of the previous viewpoint into three-dimensional space, and then re-projecting it onto the current pixel plane based on the camera pose of the current viewpoint and twisting it. Geometric-temporal cyclic refinement network Based on these inputs, an updated historical feature representation is generated for the current low-resolution frame. : ; The coefficient vector in the polynomial space is calculated using the updated historical feature representation. residual characteristics : ; The final high-resolution reconstructed frame Represented as: .

[0014] In one embodiment, the overall loss function of the gradient-aware supersampling module for: ; Karl Bonnell loss is used to constrain pixel-level photometric consistency in high-resolution reconstructed frames. ; for The corresponding real image, Represented as a constant; Laplacian loss is used to enhance the fine-grained detail consistency between the generated high-resolution reconstructed frames and the real images; ; Represents the L1 norm; Weighting coefficient Used to balance the constraints of the two losses during the optimization process.

[0015] In one embodiment, the timing frame interpolation module includes: The forward-backward warping unit is configured to warp the high-resolution features of the previous frame and the low-resolution features of the next frame to the intermediate time step, respectively, based on the camera motion vector. The feature fusion network is configured to fuse bidirectional twisted features using a lightweight U-shaped network structure and output a mixed weight map and color prediction results. Combined with the forward twisting results, a high-resolution intermediate frame is obtained. The temporal loop unit is configured to update historical features based on the gated loop unit to model inter-frame temporal dependencies.

[0016] In one embodiment, the forward-backward warping unit is configured to warp the high-resolution features of the previous frame and the low-resolution features of the next frame to an intermediate time step, respectively, based on the camera motion vector, specifically including: First, the high-resolution reconstructed frame generated by gradient-aware supersampling from the previous frame... and the corresponding depth map and normal diagram The resulting high-resolution reconstruction features are based on the first motion vector from time 0 of the previous frame to time 0.5 in the middle. Perform the first forward twist operation to obtain the first forward twist feature. Secondly, the next frame is a low-resolution rendered frame directly generated by 3D Gaussian splash rendering. and the corresponding depth map Normal diagram and gradient plot The low-resolution rendering features are composed of the second motion vector from time 1 in the next frame to time 0.5 in the middle. Perform a second forward twist operation to obtain the second forward twist feature. ; The first and second motion vectors are collectively denoted as motion vectors. , For the source frame time, For the target time, , ;Calculated based on the following mapping relationship For any source frame time Pixel coordinates in a two-dimensional image or feature Its origin time Twisting to the target moment The motion vector is derived from the following formula: ; In the formula, and Representing the source frame time respectively and target time Corresponding camera parameters; This represents a function that back-projects two-dimensional pixels to three-dimensional points based on pixel coordinates and camera parameters. This represents a function that projects points in three-dimensional space back onto a two-dimensional image plane. When performing the second forward warp operation, the gradient map is considered in light of the scaling sensitivity of gradient information. The distortion needs to be calculated separately according to the following formula: ; in, This represents a forward warp operator that spatially resamples image data based on motion vectors. For spatial gradient operators, This is the distorted gradient map.

[0017] In one embodiment, the feature fusion network is configured to fuse bidirectionally warped features using a lightweight U-shaped network structure and output a mixed weight map and color prediction results. Combined with the forward warping results, a high-resolution intermediate frame is obtained, specifically including: The first forward distortion feature is applied through a convolutional layer. Second forward twisting feature After initial encoding, the data is fed into a lightweight U-shaped network for fusion processing and outputs the fused features. The lightweight U-shaped network is composed of multiple stacked basic blocks, each of which integrates max pooling and upsampling operations. A spatial attention module is introduced into the skip connection; In the upsampling stage, pixel rearrangement is used instead of traditional transposed convolution; Fusion features Input a decoding branch consisting of convolution and deconvolution operations, and obtain the fused color prediction result. The fusion features Another decoding branch consisting of convolution and upsampling operations is input to obtain the hybrid weight map. Simultaneously, by analyzing the high-resolution reconstructed frames... The first forward warp operation is applied to obtain the warp result of the previous frame. ,Right now Finally, combining the forward warping results, the high-resolution intermediate frame image is calculated using the following formula. : .

[0018] In one embodiment, the timing loop unit is configured to update historical features based on the gated loop unit to model inter-frame temporal dependencies, specifically including: The timing cyclic unit uses the modified historical features With current fusion features As input, output the updated historical features. ; wherein, the historical features after deformation It is the historical feature output by the timing loop unit at the previous time step. The result is obtained by applying the same spatiotemporal alignment process as the first forward warp operation, i.e.: ; The update operation is performed by a loop update function based on a gated loop unit. The execution and update process are represented as follows: .

[0019] In one embodiment, the overall loss function of the time-series frame interpolation module for: ; Losses for Karl Bonnell: ; To perceive loss: To compensate for sensory loss due to occlusion: ; Indicates the mean absolute error; For regularization terms: ; , , These are the weighting coefficients.

[0020] Compared with the prior art, the beneficial technical effects of the present invention are: 1. This invention proposes a novel post-rendering acceleration framework that jointly optimizes supersampling and frame interpolation, achieving for the first time efficient reconstruction of high-resolution, high-frame-rate rendering results from low-resolution 3DGS output. This method requires no modification to the underlying 3DGS rendering pipeline, is modular and plug-and-play, and can be widely applied to existing 3DGS systems, significantly improving their practicality and deployment capabilities in latency-sensitive scenarios such as virtual reality and augmented reality.

[0021] 2. This invention designs a gradient-aware supersampling module that fully utilizes the continuous differentiability of 3DGS to extract the image spatial gradient as a guiding signal, and combines it with a GRU refined network to achieve efficient and high-fidelity super-resolution reconstruction. Compared with traditional interpolation methods, the gradient-aware supersampling module can effectively restore image details while maintaining lightweight computation, avoiding jagged edges and blurring, and improving visual quality.

[0022] 3. This invention proposes a lightweight temporal frame interpolation module, based on a lightweight U-shaped network, which fuses temporal and differentiable spatial information to achieve high-quality intermediate frame synthesis. This module is highly efficient, supports parallel processing, and can significantly improve the rendering frame rate without introducing noticeable latency, making it suitable for real-time interactive scenarios.

[0023] 4. This invention has been systematically evaluated on multiple public datasets. Experimental results show that the method of this invention outperforms existing mainstream supersampling and frame interpolation methods in both rendering quality and efficiency. Furthermore, the framework is compatible with various 3DGS acceleration technologies, further enhancing its real-time rendering capabilities in 4K resolution and high-density scenes, demonstrating good scalability and practical value. Attached Figure Description

[0024] Figure 1 This is a framework diagram of the 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation of the present invention.

[0025] Figure 2 This is a schematic diagram of the gradient-aware oversampling module of the present invention.

[0026] Figure 3 This is a schematic diagram of the lightweight timing frame interpolation module of the present invention.

[0027] Figure 4 This is a qualitative comparison diagram of the oversampling and frame interpolation tasks in the embodiments of the present invention. Detailed Implementation

[0028] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] The proposed 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation is a unified framework that combines supersampling and frame interpolation to accelerate 3DGS rendering. The overall framework of this invention is as follows: Figure 1 As shown, this invention first proposes a gradient-aware supersampling module that utilizes the differentiability of 3DGS rendered images to achieve high-quality supersampling. Furthermore, this invention also proposes a lightweight temporal frame interpolation module based on a compact UNet-like backbone network, aiming to reduce computational costs by effectively utilizing temporal and image gradient information. These two modules work together to reduce the average rendering time per frame, thereby improving overall rendering efficiency.

[0030] The following is a description of the modules in this application: 1. Gradient-aware supersampling module.

[0031] like Figure 2As shown, this invention proposes a gradient-aware supersampling network module. This network can utilize geometric and historical prior knowledge more effectively and robustly, thereby achieving superior supersampling performance. It comprises two main components: (1) gradient-aware interpolation and (2) geometry-temporal cyclic refinement.

[0032] (1) Gradient-aware interpolation: Inspired by Hermitian interpolation, this invention proposes a novel gradient-aware interpolation method. Hermitian interpolation combines the function value and its corresponding derivative at the interpolation nodes. For each interpolation node, the polynomial is constrained to match the value of the original function at that point and its derivative reaching a specified order.

[0033] For a given pixel The present invention considers it together with its 8 adjacent pixels to form a The interpolation node blocks. Although two-dimensional interpolation schemes may vary depending on node distribution and derivative availability, this invention considers bivariate polynomials satisfying the following conditions. : ; ; ; in .

[0034] For a given oversampling factor S (S is a positive integer representing the magnification factor in the height and width directions), let the low-resolution frame contain elements that are magnified in the height and width directions. Centered Interpolation node blocks correspond to high-resolution images China and Israel Centered on, size is Image patches. Among them, , These are the corresponding center coordinates in high-resolution space.

[0035] High-resolution image obtained by interpolation In this context, the value of each pixel is calculated using a local polynomial at its corresponding spatial location, i.e. And polynomial In Centered The interpolation node is determined independently within the block. Therefore, It can be viewed as a sampling of a continuous intensity surface on a high-resolution mesh, which is reconstructed block by block by a series of local polynomials.

[0036] Furthermore, each pixel within an image block can be represented as a linear combination of the pixel intensity and spatial gradient of that block in the low-resolution frame, i.e.: ; in, This indicates the column vectorization operator. For Centered Interpolation node blocks; For high-resolution images and The corresponding size is The image patch; M is a fixed coefficient matrix determined by the interpolation conditions.

[0037] The process described above, which uses linear weighting of local polynomials to generate high-resolution image patches, can be efficiently implemented through cascaded convolutional layers and transposed convolutional layers. Specifically, firstly, through convolutional layers... Extract combined features of pixel intensity and spatial gradient from the input low-resolution frame; then, pass through a transposed convolutional layer. This involves upsampling and locally weighted synthesis of the high-resolution grid. This structure is equivalent to performing the aforementioned Hermitian interpolation operation in parallel across the entire image.

[0038] (2) Geometric-temporal loop refinement: While the introduction of gradients enhances the recovery of image details, these gradients themselves originate from the partial derivatives of the original continuous image function. Therefore, they are still limited by the sampling frequency, making it difficult to avoid aliasing artifacts. Recent research has shown that combining a geometric buffer (G-buffer) and temporal cues can significantly improve reconstruction quality. Inspired by this, this invention proposes a geometric-temporal recurrent thinning network to further enhance the interpolation output.

[0039] The network employs a gated recurrent unit (GRU) architecture. The network inputs include: the current low-resolution frame I, and the corresponding gradient map G, depth map D, and normal-view inner product map generated through transparent blending during the 3DGS rendering process. In addition, distorted historical features aligned to the viewpoint are also input. . Historical features output from preceding time gait Based on the depth map and camera pose of the previous viewpoint, it is back-projected into 3D space, and then re-projected onto the current pixel plane based on the current viewpoint's camera pose, resulting in a warped representation. The Geometric-Temporal Recurrent Network (RefineNet) uses this to generate updated historical feature representations for the current low-resolution frame. : .

[0040] The coefficient vector in the polynomial space is calculated using the updated historical feature representation. residual characteristics : ; The final high-resolution reconstructed frame Represented as: .

[0041] (3) Oversampling module loss design: To ensure accurate upsampling and restoration of true detail during super-resolution reconstruction, this invention designs a loss function consisting of two constraints for the gradient-aware supersampling module. Specifically, given a supersampling image... With the truth image This invention employs Karl Bonnell loss. Basic pixel-level photometric consistency constraints are applied to the oversampled image, where For a very small positive number (e.g.) This is used to ensure the stability of numerical calculations and avoid gradient vanishing or division by zero errors; Laplace loss is employed. This is used to enhance the fine-grained detail consistency between the generated image and the real image. Therefore, the overall loss function of the gradient-aware supersampling module can be expressed as: ; Among them, the weighting coefficient Used to balance the constraint strength of the two loss functions in the optimization process.

[0042] 2. Lightweight time frame interpolation module.

[0043] like Figure 3 As shown, to further improve rendering efficiency, this invention proposes a lightweight spatiotemporal fusion interpolation module. This module fully utilizes the inherent spatiotemporal coherence of 3D Gaussian Splash (3DGS), with lightweight and high efficiency as its core design objectives. By effectively mining gradient information, geometric priors, and historical temporal features, the spatiotemporal fusion interpolation module can achieve real-time frame interpolation while ensuring rendering quality. This module consists of three core sub-modules: (1) a forward-backward optical flow deformation module; (2) a feature fusion network; and (3) a temporal loop unit.

[0044] (1) Forward-backward twist: This invention first warps the high-resolution image of the previous frame and the low-resolution image of the next frame to the midpoint of the time interval. Specifically, first, the high-resolution reconstructed frame generated by gradient-aware supersampling from the previous frame is processed. , and their corresponding depth maps and normal diagram The resulting high-resolution reconstruction features are based on the first motion vector from time 0 of the previous frame to time 0.5 in the middle. Perform the first forward twist operation to obtain the first forward twist feature. Then, the next frame is a low-resolution rendered frame directly generated by 3D Gaussian splash rendering. , and their corresponding depth maps Normal diagram and gradient plot The resulting low-resolution rendering features are based on the second motion vector from time 1 in the next frame to time 0.5 in the middle. Perform a second forward twist operation to obtain the second forward twist feature. .

[0045] The above two motion vectors The calculation is based on the following mapping relationship: for any source frame time Pixel coordinates in a two-dimensional image or feature Its origin time Twisting to the target moment The motion vector is derived from the following formula: ; In the formula, and Representing the source frame time respectively and target time The corresponding camera parameters (equivalent to the concept of camera pose, used to define the spatial position and orientation of the camera); the UnProj(·) function represents back-projecting two-dimensional pixels to three-dimensional spatial points based on pixel coordinates and camera parameters; the Proj(·) function represents projecting three-dimensional spatial points back to the two-dimensional image plane; It should be noted that, when performing the second forward warp operation, the gradient map is considered to be scaled sensitively. The distortion needs to be calculated separately according to the following formula: ; Where w(·) represents the forward warp operator that performs spatial resampling of image data based on motion vectors. This is a spatial gradient operator. The present invention employs high-resolution features from the previous frame and low-resolution features from the next frame to reduce latency and support parallel processing, thereby improving computational efficiency while maintaining temporal consistency.

[0046] (2) Feature fusion network: To achieve high-precision and high-efficiency video frame interpolation, this invention designs a feature fusion network. This network aims to effectively fuse features obtained from bidirectional distortion, thereby enhancing detail representation while maintaining structural consistency.

[0047] Specifically, the first forward twist feature is processed by a convolutional layer. and the second forward twisting feature Initial feature encoding is performed, followed by input into a lightweight U-shaped network for fusion processing and output of fused features. The lightweight U-net described is an improvement upon the U-Net network architecture described in the paper "Ronneberger O, Fischer P, Brox T. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015." The improvements include: The convolutional blocks consisting of two layers in the original U-Net are replaced with basic blocks consisting of stacked convolution, regularization, max pooling, and upsampling operations, in order to increase the receptive field and reduce the number of parameters and computation while maintaining the network's expressive power. In the skip connection path of the original network, the spatial attention module proposed in the paper "Wu R, Liu Y, Ning G, et al.'Ultralight vm-unet: Parallel vision mamba significantly reduces parameters for skin lesion segmentation.' Patterns, 2025, 6(11)." is integrated to enhance the ability to express key spatial features; In the upsampling stage, pixel rearrangement is used instead of the traditional transpose convolution operation to improve upsampling efficiency.

[0048] Fusion features Input a decoding branch consisting of convolution and deconvolution operations, and obtain the fused color prediction result. The fusion features Another decoding branch consisting of convolution and upsampling operations is input to obtain the hybrid weight map. Simultaneously, obtain the distortion result of the previous frame. It is through the high-resolution reconstructed frame The result obtained by applying the first forward twist operation is, i.e. ,in The motion vector is defined from the previous frame to the intermediate time step. Finally, combining the forward warp result, the high-resolution intermediate frame image is calculated using the following formula. : .

[0049] This adaptive fusion strategy effectively balances the reliability of distorted content with the accuracy of fusion prediction. It not only preserves the fine structure of the image but also alleviates flickering in time, thereby significantly improving the visual consistency and realism of the interpolation results.

[0050] (3) Timing Cyclic Unit: To enhance temporal consistency, this invention proposes a temporal cyclic unit for explicitly modeling inter-frame dependencies. The temporal cyclic unit utilizes deformed historical features. With current fusion features As input, output the updated historical features. ; wherein, the historical features after deformation It is the historical feature output by the timing loop unit at the previous time step. The result is obtained by applying the same spatiotemporal alignment process as the first forward warp operation described above, namely: ; The update operation is performed by the loop update function U(·) based on the gated loop unit, and its update process is expressed as follows: ; in This is a loop update function. Hidden state. By progressively accumulating image features and geometric priors from previous frames, a compact spatiotemporal representation of the scene is constructed. This design helps achieve consistency between inter-frame motion estimation and appearance interpolation, ensuring smooth and stable 3DGS scene rendering.

[0051] (4) Loss design of the temporal frame interpolation module: To improve image detail recovery quality and enhance the temporal consistency of image sequences, this invention designs the following loss function. First, Karl Bonnell loss is employed. To improve the robustness of reconstructed images to outliers, a perceptual loss based on the VGG-19 network is introduced from the paper "Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014." This approach enhances the perceptual quality of reconstructed images through high-level feature constraints. It constrains the adaptive fusion process to ensure that it only affects... The invention restores unreliable regions in the data. Introduction The regularization term is defined as follows: Furthermore, to further suppress inter-frame flickering, this invention introduces an occlusion perception loss, which takes the following form: ; In the formula Mean absolute error Finally, the overall loss function is expressed as: ; in, , These are the weighting coefficients for each loss term.

[0052] In summary, the contributions of this invention are as follows: 1. This invention proposes a 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation, which jointly performs supersampling and frame interpolation for real-time large-scale 3DGS rendering.

[0053] 2. This invention proposes a gradient-aware supersampling method, which utilizes the continuous differentiability of Gaussian units to extract image gradients and achieve gradient feature-guided image interpolation.

[0054] 3. This invention proposes a lightweight temporal frame interpolation method that integrates temporal information of consecutive frames with image gradient information to achieve efficient inter-frame interpolation.

[0055] 4. Extensive qualitative and quantitative experiments were conducted to verify the effectiveness of the proposed method. Experimental results show that the method achieves higher rendering efficiency and visual fidelity.

[0056] This invention evaluates its rendering acceleration and quality preservation capabilities in real-world large-scale scenes on several challenging public datasets. Experiments used three widely used benchmark datasets: Mip-NeRF 360, Tanks and Temples, and DeepBlending, covering diverse scenes from unbounded outdoor scenes to complex indoor environments, demonstrating strong representativeness and generalization. The model was trained on the Garden scene of Mip-NeRF 360 and evaluated on all scenes. The final results are reported based on the average performance of 300 frame sequences.

[0057] In quantitative evaluation, this invention uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) as reference metrics for image quality, and simultaneously reports the average frame rate (FPS) to measure rendering efficiency. Since this invention is a post-processing enhancement framework whose input is the low-resolution output of 3DGS rendering, it uses standard 3DGS rendering results as ground truth to ensure that differences in all comparison methods originate only from oversampling or frame interpolation modules, rather than from the underlying 3D reconstruction quality.

[0058] On the Mip-NeRF 360, Tanks and Temples, and Deep Blending datasets, this invention uses high-resolution images rendered by standard 3DGS as ground truth and generates low-resolution inputs for post-processing through 4x downsampling. This invention compares the proposed method with various approaches, including traditional interpolation methods (Bicubic), learned super-resolution methods (NinaSR-B1, RT4KSR, ECBSR), video frame interpolation methods (FLAVR, InterpAny, GIMM-VFI), and 3DGS-specific acceleration methods (LightGaussian, MiniSplatting, Speedy-Splat, AdR-Gaussian).

[0059] Table 1. Quantitative evaluation results of the oversampling and frame interpolation tasks

[0060] The quantitative results of this invention are shown in Table 1. Experimental results show that this invention achieves the highest rendering frame rate among all compared methods, while exhibiting competitive visual quality in both PSNR and SSIM metrics. Notably, although video frame interpolation methods typically rely on high-resolution ground truth for supervision, the LTFI module of this invention, which interpolates only the super-resolution preceding frame with the low-resolution following frame, still achieves visual quality close to or even surpasses these methods. Figure 4 The qualitative comparison further verifies the advantages of the present invention in terms of detail restoration and temporal consistency.

[0061] Table 2. Quantitative evaluation results of the oversampling and frame interpolation tasks

[0062] To further verify the versatility of this invention, it was integrated with several existing 3DGS acceleration methods, and the rendering frame rate was evaluated at 1440p and 4K resolutions. Table 2 shows that this invention significantly improved rendering efficiency in all integrated schemes, indicating that it has good complementarity and compatibility with existing acceleration methods.

[0063] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0064] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0065] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0066] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0067] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for accelerating 3D Gaussian splash rendering based on gradient-aware supersampling and frame interpolation, characterized in that, include: Obtain the low-resolution frame sequence output by the 3D Gaussian Splash rendering engine; A gradient-aware supersampling module is used to process the current low-resolution frame to generate a high-resolution reconstructed frame. The gradient-aware supersampling module extracts the image gradient using the continuous differentiability of Gaussian units and guides a gradient-aware interpolation network and a geometric-temporal cyclic thinning network to perform super-resolution reconstruction based on the image gradient. A temporal frame interpolation module is used to process consecutive high-resolution reconstructed frames and low-resolution frames to synthesize high-resolution intermediate frames; wherein, the temporal frame interpolation module fuses temporal information and image gradient information to generate high-resolution frames at intermediate moments, which serve as high-resolution intermediate frames. The output consists of a sequence of rendered images composed of the high-resolution reconstructed frames and the high-resolution intermediate frames.

2. The 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation according to claim 1, characterized in that, The gradient-aware supersampling module includes: The gradient-aware interpolation network is configured to generate an initial high-resolution image based on the Hermitian interpolation principle by using the pixel intensity and corresponding spatial gradient of the low-resolution frame through convolution and transpose convolution operations. The geometric-temporal recurrent refinement network is configured to employ an improved gated recurrent unit architecture, which integrates the gradient, depth map, normal-view inner product map of the current low-resolution frame and historical features of the previous frame to generate residual features to optimize the initial high-resolution image and obtain a high-resolution reconstructed frame.

3. The 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation according to claim 2, characterized in that, The gradient-aware interpolation network is configured to generate an initial high-resolution image based on the Hermitian interpolation principle, utilizing the pixel intensity and corresponding spatial gradient of low-resolution frames through convolution and transpose convolution operations. Specifically, it includes: Given a low-resolution frame pixels And the 8 adjacent pixels form a group of size. Consider interpolation node blocks that satisfy the following conditions for bivariate polynomials. : ； ； ； in, ; For a given oversampling factor S, let the low-resolution frame contain elements that are... Centered Interpolation node blocks correspond to high-resolution images China and Israel Centered on, size is The image patch, in which, , , These are the corresponding center coordinates in high-resolution space; High-resolution image obtained by interpolation In this context, the value of each pixel is calculated using a local polynomial at its corresponding spatial location, i.e. And polynomial In Centered The interpolation node is determined independently within the block; therefore... It is considered as a sampling of a continuous intensity surface on a high-resolution mesh, which is reconstructed block by block by a series of local polynomials; Each pixel within an image block is represented as a linear combination of the pixel intensity and spatial gradient of that block in the low-resolution frame, i.e.: ； in, This indicates the column vectorization operator. For Centered Interpolation node blocks; For high-resolution images and The corresponding size is The image patch; M is a fixed coefficient matrix determined by the interpolation conditions; The process of generating high-resolution images using local polynomials described above can be achieved through cascaded convolutional layers and transposed convolutional layers: First, through convolutional layers From the input low-resolution frame Extract the combined features of corresponding pixel intensity and spatial gradient. : ； Through transposed convolutional layers By performing upsampling and local weighted synthesis on a high-resolution grid, a high-resolution image is obtained. ; 。 4. The 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation according to claim 3, characterized in that, The geometric-temporal recurrent thinning network is configured to employ an improved gated recurrent unit architecture, fusing the gradient, depth map, normal-view inner product map of the current low-resolution frame and historical features from the previous frame to generate residual features to optimize the initial high-resolution image, resulting in a high-resolution reconstructed frame. Specifically, this includes: The geometric-temporal recurrent refinement network employs an improved gated recurrent unit architecture; the input to the geometric-temporal recurrent refinement network includes: the current low-resolution frame. The gradient map G, depth map D, and normal-view inner product map generated by transparent blending in the 3D Gaussian splash rendering process. Distorted historical features after perspective alignment ; Historical features output from preceding time gait It is obtained by back-projecting the depth map and camera pose of the previous viewpoint into three-dimensional space, and then re-projecting it onto the current pixel plane based on the camera pose of the current viewpoint and twisting it. Geometric-temporal cyclic refinement network Based on these inputs, an updated historical feature representation is generated for the current low-resolution frame. : ； The coefficient vector in the polynomial space is calculated using the updated historical feature representation. residual characteristics : ； The final high-resolution reconstructed frame Represented as: 。 5. The 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation according to claim 1, characterized in that, The overall loss function of the gradient-aware supersampling module for: ; Karl Bonnell loss is used to constrain pixel-level photometric consistency in high-resolution reconstructed frames. ； for The corresponding real image, Represented as a constant; Laplacian loss is used to enhance the fine-grained detail consistency between the generated high-resolution reconstructed frames and the real images; ； Represents the L1 norm; Weighting coefficient Used to balance the constraints of the two losses during the optimization process.

6. The 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation according to claim 1, characterized in that, The timing frame interpolation module includes: The forward-backward warping unit is configured to warp the high-resolution features of the previous frame and the low-resolution features of the next frame to the intermediate time step, respectively, based on the camera motion vector. The feature fusion network is configured to fuse bidirectional twisted features using a lightweight U-shaped network structure and output a mixed weight map and color prediction results. Combined with the forward twisting results, a high-resolution intermediate frame is obtained. The temporal loop unit is configured to update historical features based on the gated loop unit to model inter-frame temporal dependencies.

7. The 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation according to claim 6, characterized in that, The forward-backward warping unit is configured to warp the high-resolution features of the previous frame and the low-resolution features of the next frame to the intermediate time step according to the camera motion vector, specifically including: First, the high-resolution reconstructed frame generated by gradient-aware supersampling from the previous frame... and the corresponding depth map and normal diagram The resulting high-resolution reconstruction features are based on the first motion vector from time 0 of the previous frame to time 0.5 in the middle. Perform the first forward twist operation to obtain the first forward twist feature. Secondly, the next frame is a low-resolution rendered frame directly generated by 3D Gaussian splash rendering. and the corresponding depth map Normal diagram and gradient plot The low-resolution rendering features are composed of the second motion vector from time 1 in the next frame to time 0.5 in the middle. Perform a second forward twist operation to obtain the second forward twist feature. ; The first and second motion vectors are collectively denoted as motion vectors. , For the source frame time, For the target time, , ;Calculated based on the following mapping relationship For any source frame time Pixel coordinates in a two-dimensional image or feature Its origin time Twisting to the target moment The motion vector is derived from the following formula: ； In the formula, and Representing the source frame time respectively and target time Corresponding camera parameters; This represents a function that back-projects two-dimensional pixels to three-dimensional points based on pixel coordinates and camera parameters. This represents a function that projects points in three-dimensional space back onto a two-dimensional image plane. When performing the second forward warp operation, the gradient map is considered in light of the scaling sensitivity of gradient information. The distortion needs to be calculated separately according to the following formula: ； in, This represents a forward warp operator that spatially resamples image data based on motion vectors. For spatial gradient operators, This is the distorted gradient map.

8. The 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation according to claim 7, characterized in that, The feature fusion network is configured to fuse bidirectionally warped features using a lightweight U-shaped network structure and output a mixed weight map and color prediction results. Combined with the forward warping results, a high-resolution intermediate frame is obtained, specifically including: The first forward distortion feature is applied through a convolutional layer. Second forward twisting feature After initial encoding, the data is fed into a lightweight U-shaped network for fusion processing and outputs the fused features. The lightweight U-shaped network is composed of multiple stacked basic blocks, each of which integrates max pooling and upsampling operations. A spatial attention module is introduced into the skip connection; In the upsampling stage, pixel rearrangement is used instead of traditional transposed convolution; Fusion features Input a decoding branch consisting of convolution and deconvolution operations, and obtain the fused color prediction result. The fusion features Another decoding branch consisting of convolution and upsampling operations is input to obtain the hybrid weight map. Simultaneously, by analyzing the high-resolution reconstructed frames... The first forward warp operation is applied to obtain the warp result of the previous frame. ,Right now Finally, combining the forward warping results, the high-resolution intermediate frame image is calculated using the following formula. : 。 9. The 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation according to claim 8, characterized in that, The temporal loop unit is configured to update historical features based on the gated loop unit to model inter-frame temporal dependencies, specifically including: The timing cyclic unit uses the modified historical features With current fusion features As input, output the updated historical features. ; wherein, the historical features after deformation It is the historical feature output by the timing loop unit at the previous time step. The result is obtained by applying the same spatiotemporal alignment process as the first forward warp operation, i.e.: ； The update operation is performed by a loop update function based on a gated loop unit. The execution and update process are represented as follows: 。 10. A 3D Gaussian splash rendering acceleration method based on gradient-aware supersampling and frame interpolation according to claim 8, characterized in that, The overall loss function of the temporal frame interpolation module for: ； Losses to Karlbonier: ； To perceive loss: To compensate for sensory loss due to occlusion: ； Indicates the mean absolute error; For regularization terms: ； , , These are the weighting coefficients.

Images