Three-dimensional Gaussian splashing and rasterization pipeline fused view synthesis system and method

Abstract

The invention discloses a three-dimensional Gaussian splashing and rasterization pipeline fused view synthesis system and method. The method comprises the following steps of: 1, acquiring two-dimensional Gaussian primitives; 2, converting the two-dimensional Gaussian primitive into two triangular grids, and converting the two triangular grids into a world coordinate system; 3, baking the spherical harmonic coefficient of the color and the opacity data in a texture form; 4, enabling each Gaussian primitive to correspond to a texture block on the texture image set, enabling a triangle to cover the texture block in a UV space, and obtaining a Gaussian weight; 5, based on the obtained Gaussian weight, simulating an alpha mixing behavior in the three-dimensional Gaussian splashing pipeline by using a time-based jitter anti-aliasing method; and 6, training a symbol vector field SDF branch while training, and mutually supervising and constraining the output of the symbol vector field and the output of the two-dimensional Gaussian. The three-dimensional Gaussian splash can be rendered in a traditional rasterization pipeline of the unreal engine 4, and the fidelity equivalent to that of a special training pipeline is kept.

Description

Technical Field

[0001] This invention belongs to the field of 3D scene representation and new view synthesis technology, specifically involving a view synthesis system and method that integrates 3D Gaussian Splatting (3DGS) with rasterization pipeline. Background Technology

[0002] New view synthesis (NVS), a crucial long-term goal in computer vision and graphics, has been the reconstruction of 3D scene representations and photorealistic representations. The objective of NVS is to reconstruct a 3D scene from a set of images taken from different angles, allowing for the generation of new views from any angle. The advent of Neural Radiation Fields (NeRF) in 2020 significantly impacted this field. NeRF is an implicit volume rendering model whose core idea is to learn the color and density of points using a neural network and calculate the pixel color seen by the camera using volume rendering integrals. However, the biggest bottleneck in NeRF-related technologies lies in their slow rendering speed, lack of real-time interactivity, and the complexity of neural network computation, making direct integration into traditional GPU rendering pipelines difficult. To address this issue, a series of optimization methods have been proposed. Among them, Mobile-NeRF achieved a breakthrough by fusing NeRF with the traditional rasterization pipeline, optimizing the mesh structure during NeRF training, and employing a micro-MLP for shading.

[0003] 3D Gaussian Splashing (3DGS) technology has broken NeRF's dominance in the field of new perspective compositing. The core idea of ​​3DGS is that the scene is composed of a large number of 3D Gaussian distributions (each point contains position, color, size, and orientation); each Gaussian distribution forms an ellipse when projected onto a 2D image plane; and rendering uses a GPU-based, parallel computing-based tiled rasterization pipeline, enabling real-time rendering. Compared to NeRF, 3DGS not only has a shorter training time but also achieves superior rendering quality. Its specially designed tile-based rasterization pipeline ensures extremely fast rendering speeds. With the rapid development of the technology, 3DGS has quickly expanded into multiple fields, including anti-aliased rendering, scene editing, large-scale scene reconstruction, and geometric surface reconstruction. However, 3DGS's dedicated pipeline is strongly coupled to a dedicated rendering pipeline, which hinders the import of training results into traditional graphics engines (such as Unreal Engine) for subsequent editing, physical simulation, and integration into existing traditional rendering pipelines, making it inconvenient for use in more fields.

[0004] The traditional rendering pipeline (Rasterization Pipeline) is the foundational structure of all current real-time graphics engines. For example, UE, Unity, OpenGL, and DirectX all rely on triangle meshes to represent geometry; vertex shaders handle geometric transformations; fragment shaders handle color and lighting; and Z-buffers control occlusion. This type of rendering pipeline is extremely mature, supporting real-time interaction, lighting models, shadows, physical simulation, dynamic textures and material systems, and post-processing effects (anti-aliasing, blurring, HDR, etc.). However, the traditional rendering pipeline also has the following drawbacks: First, it has a strong geometric dependency. The core input of traditional rendering is a triangular mesh. Real-world scenes are captured by cameras or reconstructed from multiple views, resulting in sparse point clouds, volumetric representations, implicit fields, etc., which require additional mesh reconstruction operations.

[0005] Secondly, it is difficult to represent volume, translucency and scattering effects. Traditional rasterization is based on "surface" rendering rather than "volume". It can only render the surface color of an object and is powerless to deal with physical phenomena such as light absorption, scattering and transparency mixing inside the volume. In order to achieve special effects (smoke, fog, glass), approximate methods (such as transparency sorting, alphablending, screen space effects) are usually used.

[0006] Third, memory utilization is low, and the reconstruction and rendering efficiency is very poor in dense scenes. For dense scenes (such as millions of voxels or point clouds), traditional pipelines still need to convert them into meshes through "meshing". Each point or face occupies a vertex buffer, and the computational burden of operations such as Z-buffer increases sharply.

[0007] Fourth, there is a mismatch between lighting and geometry. In neural rendering, color and geometry are jointly determined through optimization. Traditional rendering, on the other hand, separates materials and lighting, which leads to problems in reconstruction tasks such as: lighting miscalculation; misaligned geometric boundaries; and shadows that are inconsistent with the real scene. Summary of the Invention

[0008] To overcome the shortcomings of the existing technology, the present invention aims to provide a view compositing system and method that integrates 3D Gaussian Splatting (3DGS) with the rasterization pipeline, enabling 3D Gaussian Splatting to be rendered in the traditional rasterization pipeline of Unreal Engine 4 (UE4) while maintaining fidelity comparable to that of a dedicated training pipeline. It also provides convenience for real-time graphics engines using traditional rendering pipelines to integrate with the 3D Gaussian Splatting method.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A view synthesis system that integrates 3D Gaussian splashing and rasterization pipeline includes a data input and preprocessing module, a 2D Gaussian splashing module, a Gaussian element to mesh conversion module, a parameter baking module, a UV mapping and normalization module, a symbolic vector field joint training and constraint module, and a rasterization rendering and editing module. The data input and preprocessing module preprocesses the input data to obtain a dataset, which serves as the initial state for training two-dimensional Gaussian units. The two-dimensional Gaussian splashing module optimizes the preprocessed dataset to obtain two-dimensional Gaussian elements; The Gaussian primitive to mesh conversion module, parameter baking module, and UV mapping and normalization module work together to convert the 2D Gaussian primitives trained by the 2D Gaussian splashing module into polygonal meshes that can be rendered in the GPU rendering pipeline. The spherical harmonic coefficients and opacity parameters of the 2D Gaussian primitives are baked and mapped to the texture atlas. By establishing the correspondence between the polygonal mesh and the texture atlas, the consistency of the 2D standard Gaussian function UV space is achieved, which facilitates the rendering of 2D images and the editing of 3D scenes in the rasterization rendering and editing modules. The symbolic vector field joint training and constraint module constrains the 2D Gaussian splashing module, making the Gaussian point cloud closer to the real surface and improving geometric accuracy.

[0010] In the data input and preprocessing module, the input dataset consists of multi-view images of the same scene. The multi-view image sequence is analyzed using a structure from motion (SfM) method to recover camera parameters and sparse point clouds of a simple 3D scene structure as the initial state for training two-dimensional Gaussian units. The two-dimensional Gaussian splash training module uses the two-dimensional Gaussian splash algorithm to optimize the point cloud and obtain a set of two-dimensional Gaussian primitives with color, normal and scale. The two-dimensional Gaussian splash uses a flattened Gaussian ellipsoid (elliptical disk) to represent objects in the scene. The Gaussian primitive to mesh module converts the two-dimensional Gaussian primitives trained by the two-dimensional Gaussian splash training module into polygonal meshes that can be rendered in the GPU rendering pipeline. The parameter baking module maps the spherical harmonic coefficients (used for color calculation) and opacity parameters of Gaussian elements to texture maps. The UV mapping and normalization module establishes the correspondence between polygonal meshes and texture maps, and achieves numerical consistency of the two-dimensional standard Gaussian function in UV space. The symbolic vector field joint training and constraint module, through joint optimization of the symbolic vector field and two-dimensional Gaussian splash training, constrains the Gaussian point cloud to be closer to the real surface, improves geometric accuracy and generates a collision mesh.

[0011] The rendering process of the rasterization rendering and editing module is executed in Unreal Engine 4 through HLSL shaders. Combined with the powerful development capabilities of UE4, the system supports in-depth post-processing of the final rendered scene. Users can freely edit the scene according to their needs, which greatly expands the application boundaries of Gaussian sputtering technology.

[0012] A view synthesis method for fusing 3D Gaussian splashing and rasterization pipelines includes the following steps; Step 1: Train the scene using the original Gaussian splashing technique and flatten the 3D Gaussian primitives (which exist in the form of ellipsoids in spatial geometry) used to represent the scene to obtain 2D Gaussian primitives, each of which contains normal information; Step 2: Each trained 2D Gaussian primitive will be converted into two triangular meshes, and these two triangles will be transformed into the world coordinate system as geometry that can be rendered by a traditional GPU. Step 3: After each 2D Gaussian element is converted into two triangular meshes, its spherical harmonic coefficients and opacity data used to represent color are baked in the form of a texture. Step 4: Sample both baked triangular meshes to the same texture block. In texture space (UV space), each Gaussian primitive corresponds to a texture block in the texture atlas. Align the center of the texture block with the Gaussian center. The triangle covers this texture block in UV space (use nearest neighbor sampling to ensure that the two triangles read the same block) and obtain the Gaussian weights. Step 5: Based on the acquired Gaussian weights, use the time-based dithering anti-aliasing method (Dither TemporalAA) to simulate the alpha mixing behavior in the 3D Gaussian splash pipeline, thereby visually approximating the synthesis effect of the 3D Gaussian splashing technology. Step 6: While training, train a signed distance field (SDF) branch and use the output of the two-dimensional Gaussian to supervise and constrain each other, so as to reduce the phenomenon of some Gaussian elements drifting away from the object surface and improve the reconstruction rendering quality.

[0013] In step 1, the operation of "flattening" the three-dimensional Gaussian primitives is used, transforming them into a set of two-dimensional Gaussian planes embedded in the three-dimensional space (geometrically visible as two-dimensional elliptical disks). Each two-dimensional Gaussian primitive has multiple attributes: center point... Two tangential vectors normal line The two corresponding scaling vectors Opacity Color information represented using spherical harmonic functions and rotation matrix ; By constructing a homogeneous transformation matrix: in: , Represents a diagonal matrix; Tangent plane coordinates Mapped to world coordinates: in: Represents the center point of a two-dimensional Gaussian element; It is a scaling vector of a two-dimensional Gaussian element in two directions in UV space; These are two tangential vectors of a two-dimensional Gaussian element in the UV space; That is, the homogeneous transformation matrix mentioned earlier; Flattened standard two-dimensional Gaussian: .

[0014] in: That is, the coordinates of the tangent plane of the two-dimensional Gaussian element. .

[0015] In step 2, each two-dimensional Gaussian primitive is converted into two triangular meshes and transformed to world coordinates to generate GPU-renderable geometry (two triangular meshes) for each two-dimensional Gaussian primitive, making it a standard Mesh in the traditional rendering pipeline; In the standard normal distribution: σ is the mean, and σ is the standard deviation in this distribution: In the Gaussian tangent plane ( Within the space, taking the range of ±3σ, construct a square and divide it into two triangles using a standardized two-dimensional Gaussian. Using the homogeneous transformation matrix mentioned in step 1 Transform the vertices of a triangle from the tangent plane to world space: for points on the tangent plane World coordinates are .

[0016] In step 3 The baking process writes the spherical harmonic coefficients and opacity of each Gaussian primitive representing its color into a small texel in the texture atlas. The color of each Gaussian primitive is represented by a 48-dimensional spherical harmonic function vector. The 48-dimensional information of the spherical harmonic coefficients is stored in 12 four-channel RGBA textures, and the opacity is stored in a single-channel texture. In the UE4 engine, the texture atlas supports a maximum of 8192×8192, which can hold information of approximately 67M Gaussian primitives. During the rendering stage, the GPU shader obtains the lighting, color, and opacity information corresponding to the Gaussian primitives by sampling the texture atlas.

[0017] In step 4, it is ensured that two triangles corresponding to the same Gaussian element sample identical texture units (texels) in the texture space (UV space), and the Gaussian weight is calculated within the same texture unit using UV distance. In the texture dataset, each Gaussian element is assigned to a texture block, with the center of the texture block used as the Gaussian center. The UV vertices of the two triangles are mapped to cover this texture block (i.e., the two triangles together cover the entire pixel range of the square in UV space). Nearest neighbor sampling is used to ensure that the shader's texture reads for this block all point to the same texture block. For images in the dataset, if each Gaussian element corresponds to one texture block, then the small texture grid corresponding to a certain Gaussian element is located in... Then the UV vertex coordinates of the two corresponding triangles are: Triangular Grid A: Triangular grid B: Calculate the local coordinates from UVs to Gaussian coordinates in the GPU shader. The mapping, and the difference from the center of the texture block. Finally, the Gaussian weights are calculated as shown in the following formula: in It is a conversion from UV coordinates to Gaussian local coordinates. The difference between the mapping and the center of the texture block, Let be the standard deviation of the two-dimensional Gaussian distribution.

[0018] In step 5, the time-based dithering anti-aliasing method discretizes the opacity into a 0 / 1 Boolean mask for each frame or pixel using a time-based dithering mask, and averages it over consecutive frames in the time dimension to approximate continuous... mix; For a Gaussian meta-element to be rendered, the opacity is adjusted on the fragment shader by baking. with Gaussian weights Multiply to calculate the final opacity of the primitive: Will Compare the binary mask with a random threshold per pixel per frame (the threshold changes periodically over time or a Halton Boolean sequence is used) to determine whether the binary mask is written to the color buffer; In Dither Temporal AA and temporal accumulation, binary writing of consecutive frames is visually restored to a continuous alpha blending effect.

[0019] In step 6, the Signed Vector Field (SDF) is used to describe the signed distance from any point in space to the surface of an object; the SDF is a continuous scalar field that represents the signed distance from any point in space to the surface of an object. Returns a real number If the point is located outside the surface of the object, then It is a positive number; if the point is located inside the surface of the object, then It is a negative number; if the point lies on the surface of the object, then It is zero.

[0020] The beneficial effects of this invention are: In the existing fields of 3D scene representation and new view compositing, 3D Gaussian Splashing (3DGS) technology, through its unique differentiable rasterization pipeline, significantly shortens training time while achieving higher-precision real-time rendering. Although this dedicated pipeline provides remarkable real-time rendering performance, the heavy reliance of 3DGS-based methods on it limits the further development and wider application of this technology. This invention proposes a system and method that integrates 3D Gaussian Splashing with a traditional rasterization pipeline to enhance interactive rendering, enabling Gaussian elements to complete rendering within the traditional rasterization pipeline of Unreal Engine 4 (UE4) while maintaining fidelity comparable to dedicated training pipelines.

[0021] Traditional 3D Gaussian splatter rendering requires pixel-by-pixel calculation of Gaussian weights and transparency mixing during image generation, resulting in high computational complexity and difficulty in fully utilizing the parallel rendering hardware of GPUs. This invention represents Gaussian primitives as standard triangular meshes, transforming the rendering process entirely into hardware-parallelizable fragment rendering operations. Simultaneously, a unified texture atlas is used to store Gaussian color spherical harmonic coefficients and opacity information, avoiding frequent texture switching operations. This structural design significantly improves rendering efficiency, enabling real-time interactive rendering frame rates while maintaining high visual fidelity. Furthermore, storing millions of Gaussian primitives requires storing complete spherical harmonic coefficients and covariance information, resulting in extremely high memory consumption. This invention employs texture atlas technology to uniformly encode the spherical harmonic coefficient data of multiple Gaussians in a compact form within a fixed-size floating-point texture. For example, 48-dimensional spherical harmonic coefficients can be split and stored across 12 RGBA floating-point textures, with single-channel textures recording opacity information. This design not only reduces the redundancy of Gaussian data access, but also reduces the GPU's texture binding frequency and bandwidth pressure, enabling the entire rendering system to maintain a stable high frame rate output under limited video memory conditions.

[0022] To avoid visual errors caused by discretized triangle representation, this invention assigns an independent texture block to each Gaussian in texture space and ensures that the two triangles corresponding to a Gaussian primitive are sampled to the same texture region through "center alignment" and "nearest neighbor sampling". During the rendering stage, the fragment shader calculates the color using baked spherical harmonics and dynamically adjusts the brightness using a Gaussian weighting function based on local distance.

[0023] Furthermore, this invention introduces a time-based dithering anti-aliasing (Dither TAA) method, which randomizes the opacity threshold of fragments and achieves visually continuous transparent blending through multi-frame temporal averaging, thereby approximating the effect of traditional 3D Gaussian splashing. Blending effects. Through this mechanism, the present invention can maintain a smooth lighting transition and transparent overlay visual effect that is almost identical to the original Gaussian rendering even in real-time rendering.

[0024] This invention introduces a Signed Vector Field (SDF) branch during training, which is mutually supervised with the 2D Gaussian branch. The SDF branch learns the implicit geometry of the scene, and its output value represents the signed distance from a spatial point to the object surface. Through multiple constraint mechanisms such as depth consistency loss, normal consistency loss, and point cloud constraint loss, the Gaussian center point is made close to the real surface of the object, thereby effectively suppressing Gaussian drift and surface discontinuities. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the structure of the present invention.

[0026] Figure 2 This is a schematic diagram summarizing the method of the present invention.

[0027] Figure 3 This is a schematic diagram for rendering quality testing.

[0028] Figure 4 This is a schematic diagram of the grid extraction test. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the accompanying drawings.

[0030] This invention discloses a view synthesis system and method for fusing three-dimensional Gaussian splashing and rasterization pipelines. To facilitate a more intuitive understanding of the structure, working principle, and interaction relationships between the various modules of the invention, the technical solution is described in detail below with reference to the accompanying drawings.

[0031] from Figure 1 As can be seen, the view synthesis system that fuses 3D Gaussian splashing with the rasterization pipeline consists of seven modules: a data input and preprocessing module, a 2D Gaussian splashing module, a Gaussian element to mesh conversion module, a parameter baking module, a UV mapping and normalization module, a symbolic vector field joint training and constraint module, and a rasterization rendering and editing module. These modules complement each other, forming the overall framework of the proposed method for enhancing interactive rendering by fusing 3D Gaussian splashing with the traditional rasterization pipeline.

[0032] Data Input and Preprocessing Module: This module takes a series of images representing the scene as a dataset, processes them using the Structure from Motion (COLMAP) method, and obtains a sparse 3D point cloud of the scene, as well as the shooting positions and poses of all photos. The sparse point cloud representing the scene serves as the initial state for the subsequent 2D Gaussian splashing module. COLMAP, involved in this invention, is a Structure from Motion algorithm. Its workflow is as follows: Feature point detection and matching are performed on the input multi-view 2D image sequence to establish visual associations across images; then, incremental or global COLMAP algorithms are used to simultaneously estimate the camera pose parameters of each image; based on this, the 3D spatial coordinates corresponding to the matched feature points are calculated using triangulation principles to generate a sparse point cloud model; finally, through a bundled adjustment and optimization step, joint nonlinear optimization is performed on all 3D point coordinates and camera parameters, thereby achieving high-precision reconstruction of the scene's 3D structure and camera motion trajectory.

[0033] 2D Gaussian Splatting Module: 2D Gaussian Splatting (2DGS) is a method based on 3D Gaussian splatting for constructing geometrically accurate and real-time renderable 3D scenes from multi-view images. This method compresses the volumetric representation of traditional 3D Gaussian primitives into 2D Gaussian plane primitives (elliptical disks) embedded in 3D space. Each 2D Gaussian primitive, with position, tangent vector, scale, and opacity as parameters, naturally corresponds to a local tangent plane of the scene surface. By introducing precise intersection point calculation based on rays and Gaussian planes, perspective-corrected differentiable sputtering rendering is achieved, thus avoiding the geometric errors caused by multi-view inconsistencies in traditional 3D Gaussian splatting. Simultaneously, 2D Gaussian splatting introduces depth distortion constraints and normal consistency constraints during optimization, causing the 2D Gaussians to converge in the depth direction and maintain consistency with the real surface normal, resulting in surface reconstruction results with low noise, clear boundaries, and accurate structure.

[0034] Gaussian Element to Mesh Module: This module converts Gaussian elements in 3D space into a standard triangular mesh representation. Based on the spatial position, rotation direction, scale information, and normal direction of each Gaussian element, it generates a corresponding 2D flattened geometric patch and outputs vertex data and texture coordinates conforming to GPU rasterization rendering standards, achieving structural compatibility between Gaussian data and traditional rendering pipelines.

[0035] The parameter baking module converts the trained Gaussian meta-parameters (including color spherical harmonics, transparency, opacity, and other radiative properties) into a GPU-readable texture data format, and stores and indexes them uniformly as a texture atlas. Through this module's preprocessing, complex high-dimensional Gaussian parameters can be efficiently accessed in a structured texture format, significantly reducing data transfer and storage overhead during rendering.

[0036] The UV mapping and normalization module generates normalized texture coordinates (UV coordinates) for each Gaussian triangle mesh after Gaussian geometry is generated, and binds them to the texture atlas generated by the parameter baking module. Through this module, each Gaussian primitive can accurately correspond to an independent texture block in texture space, achieving a one-to-one mapping from Gaussian to texture, and ensuring that all meshes use the same parameter area during sampling, guaranteeing consistent color sampling and seamless edge connections.

[0037] The Symbolic Vector Field Joint Training and Constraint Module is used to introduce a symbolic vector field and its gradient vector field as geometric constraints during Gaussian splash training. Through joint optimization of multiple loss functions, it achieves geometric supervision and surface constraints on the 2D Gaussian branch. This module can simultaneously learn the explicit rendering features and implicit geometry of the scene during the training phase, making the center point of the Gaussian units close to the real surface, preventing floating, overlapping, and rendering artifacts, and significantly improving reconstruction accuracy and visual consistency.

[0038] Rasterization Rendering and Editing Module: This module takes data that has undergone Gaussian pixel-to-mesh conversion, parametric baking, and UV mapping normalization and inputs it into the traditional GPU rasterization pipeline to achieve real-time 3D scene rendering and interactive editing. By simulating Gaussian mixing for transparency and color blending at the pixel level, combined with Dither Temporal AA and depth testing mechanisms, this module achieves visual effects similar to neural rendering while maintaining the high frame rate and editability of traditional rendering systems.

[0039] A view synthesis method for fusing 3D Gaussian splashing and rasterization pipelines includes the following steps; Step 1: Train the scene using the original Gaussian splashing technique and flatten the 3D Gaussian primitives (which exist in the form of ellipsoids in spatial geometry) used to represent the scene to obtain 2D Gaussian primitives, each of which contains normal information; Step 2: Each trained 2D Gaussian primitive will be converted into two triangular meshes, and these two triangles will be transformed into the world coordinate system as geometry that can be rendered by a traditional GPU. Step 3: After each 2D Gaussian element is converted into two triangular meshes, its spherical harmonic coefficients and opacity data used to represent color are baked in the form of a texture. Step 4: To ensure that both baked triangular meshes sample the same texture block, each Gaussian primitive in the texture space (UV space) corresponds to a texture block in the texture atlas. The center of the texture block is aligned with the Gaussian center, and the triangle covers this texture block in the UV space (using nearest neighbor sampling to ensure that the two triangles read the same block). Gaussian weights are obtained. Step 5: Based on the acquired Gaussian weights, use the time-based dithering anti-aliasing method (Dither TemporalAA) to simulate the alpha mixing behavior in the 3D Gaussian splash pipeline, thereby visually approximating the synthesis effect of the 3D Gaussian splashing technology. Step 6: While training, train a signed distance field (SDF) branch and use the output of the two-dimensional Gaussian to supervise and constrain each other, so as to reduce the phenomenon of some Gaussian elements drifting away from the object surface and improve the reconstruction rendering quality.

[0040] In step 1, the 2D Gaussian splashing technique (2DGS) is used because when representing a 3D scene, the 2D Gaussian splashing technique uses 3D Gaussian primitives (geometrically visible as Gaussian ellipsoids in space) to represent the scene. It is essentially a volume representation, which is different from the mesh used as a surface representation when representing a 3D scene in a real-time graphics engine. Furthermore, it does not have an explicit definition of normal vectors, making it difficult to accurately reconstruct the surface.

[0041] By "flattening" the three-dimensional Gaussian primitives, we transform them into a set of two-dimensional Gaussian planes embedded in three-dimensional space (geometrically visible as two-dimensional elliptical disks). Each two-dimensional Gaussian primitive has multiple attributes: center point... Two tangential vectors normal line The two corresponding scaling vectors Opacity Color information represented using spherical harmonic functions and rotation matrix ; By constructing a homogeneous transformation matrix: in: , Represents a diagonal matrix; Tangent plane coordinates Mapped to world coordinates: in: Represents the center point of a two-dimensional Gaussian element; It is a scaling vector of a two-dimensional Gaussian element in two directions in UV space; These are two tangential vectors of a two-dimensional Gaussian element in the UV space; That is, the homogeneous transformation matrix mentioned earlier; Flattened standard two-dimensional Gaussian: .

[0042] in: That is, the coordinates of the tangent plane of the two-dimensional Gaussian element. .

[0043] In step 2, each two-dimensional Gaussian primitive is converted into two triangular meshes and transformed to world coordinates to generate GPU-renderable geometry (two triangular meshes) for each two-dimensional Gaussian primitive, making it a standard Mesh in the traditional rendering pipeline; In the standard normal distribution: σ is the mean, and σ is the standard deviation. In this distribution: the probability of falling within ±1σ is approximately 68.27%, the probability of falling within ±2σ is approximately 95.45%, and the probability of falling within ±3σ is approximately 99.73%. In the Gaussian tangent plane ( Within the space, taking a range of ±3σ, for a standardized two-dimensional Gaussian, take... Construct a square and divide it into two triangles, with the vertices of the two triangles being: and ; Using the homogeneous transformation matrix mentioned in step 1 Transform the vertices of a triangle from the tangent plane to world space: for points on the tangent plane World coordinates are .

[0044] In step 3, "baking," in the field of computer graphics, refers to the process of pre-calculating and storing dynamically calculated, real-time generated, or complex lighting, material, and geometric information in a static texture map or data file. The aim is to improve the efficiency and speed of the final rendering. During real-time rendering, complex shadows, light bounces, or detailed materials are recalculated for each frame, requiring significant computational resources. Baking, on the other hand, pre-calculates the results that require substantial computational resources and "solidifies" them onto the texture. During real-time runtime, the engine only needs to directly call the existing texture, eliminating the need for complex calculations and improving the frame rate.

[0045] The baking process writes the spherical harmonic coefficients and opacity of each Gaussian primitive representing its color into a small texel in the texture atlas. The color of each Gaussian primitive is represented by a 48-dimensional spherical harmonic function vector. The 48-dimensional information of the spherical harmonic coefficients is stored in 12 four-channel RGBA textures, and the opacity is stored in a single-channel texture. In the UE4 engine, the texture atlas supports a maximum of 8192×8192, which can hold information of approximately 67M Gaussian primitives. During the rendering stage, the GPU shader obtains the lighting, color, and opacity information corresponding to the Gaussian primitives by sampling the texture atlas.

[0046] In step 4, it is ensured that two triangles corresponding to the same Gaussian element sample identical texture units (texels) in the texture space (UV space), and the Gaussian weight is calculated within the same texture unit using UV distance. Each Gaussian element is assigned to a texture block in the texture dataset, with the center of the texture block used as the Gaussian center. The UV vertices of the two triangles are mapped to cover this texture block (i.e., the two triangles together cover the entire pixel range of the square in UV space). Nearest neighbor sampling is used to ensure that the shader's texture reads for this block all point to the same texture block. For a 1k image in the dataset with a texture resolution of 1024×1024, if each Gaussian element corresponds to one small texture grid (1x1), then the small texture grid corresponding to a certain Gaussian element is located at... Then the UV vertex coordinates of the two corresponding triangles are: Triangular Grid A: Triangular grid B: Calculate the local coordinates from UVs to Gaussian coordinates in the GPU shader. The mapping, Range scaling And calculate the difference from the center of the texture block. Finally, the Gaussian weights are calculated as shown in the following formula: in It is a conversion from UV coordinates to Gaussian local coordinates. The difference between the mapping and the center of the texture block, Let be the standard deviation of a two-dimensional Gaussian distribution, and then take . .

[0047] In step 5, a time-based dithering anti-aliasing method (Dither Temporal AA) is used to simulate alpha mixing in a three-dimensional Gaussian splash pipeline. (Hybrid) behavior, because in traditional rasterization pipelines, directly using alpha to blend multiple semi-transparent 2D Gaussian primitives encounters sorting problems. The time-based dithering anti-aliasing method discretizes the opacity into a 0 / 1 Boolean mask for each frame or pixel using a time-based dithering mask, and averages it over consecutive frames in the time dimension, approximating continuous... mix; For a Gaussian meta-element to be rendered, the opacity is adjusted on the fragment shader by baking. with Gaussian weights Multiply to calculate the final opacity of the primitive: Will Compare the binary mask with a random threshold per pixel per frame (the threshold changes periodically over time or a Halton Boolean sequence is used) to determine whether the binary mask is written to the color buffer; In Dither Temporal AA and temporal accumulation, binary writing of consecutive frames is visually restored to a continuous alpha blending effect.

[0048] In step 6, the Signed Distance Field (SDF) is a function used to describe the signed distance from any point in space to the surface of an object; the SDF is a continuous scalar field that represents the distance from any point in space to the surface of an object. Returns a real number If the point is located outside the surface of the object, then It is a positive number; if the point is located inside the surface of the object, then It is a negative number; if the point lies on the surface of the object, then The value is zero. The SDF branch is introduced for three main purposes: first, to constrain Gaussian elements from drifting off the surface; second, to provide geometric consistency supervision; and third, to generate explicit surface meshes.

[0049] Principle Explanation The principle of this invention is summarized as follows: Figure 2 As shown: For each optimized point cloud, it is converted into two polygonal meshes with specially designed UV mappings, corresponding to textures generated by spherical harmonics and opacity. The Gaussian function distribution is simulated by calculating the distance from the UV coordinates of the sampling point to the center of the texture unit. This method enables Gaussian primitives to achieve rendering quality comparable to a dedicated rasterization pipeline designed specifically for 3D Gaussian sputtering. Furthermore, joint training with the symbolic vector field and Gaussian sputtering not only improves rendering quality but also constrains the point cloud to the object surface. The final extracted mesh can be integrated into the scene to support physical simulations. Hierarchical modeling principle: By connecting the various components in the image through a tree-like hierarchical structure, the model can capture the hierarchical relationships of components in the image during the architecture search and retraining phases, thereby improving the model's understanding of complex images.

[0050] This invention combines two-dimensional Gaussian sputtering with a symbolic vector field to align geometric structure with appearance representation, ensuring that the point cloud is constrained by the object surface and providing an accurate object contour framework for physical collision simulation. The symbolic vector field branch employs the Eikonal loss function to ensure that the gradient of the predicted symbolic vector field remains normalized. The fundamental purpose of the Eikonal loss function is to force a scalar field (e.g., a symbolic vector field) to obey the "Eikonal equation," mathematically expressed as: in: It is the value of the symbolic vector field at point x.

[0051] It is the gradient of the symbolic vector field at point x.

[0052] The norm (length) of a vector.

[0053] The Eikonal loss function takes the following form: In the symbolic vector field branch, the Eikonal loss function addresses several key issues: ensuring correct distance properties and gradient magnitude. This is a fundamental mathematical property of the distance function. If the gradient magnitude is less than 1, it means the function value changes too slowly, and the estimated distance is farther than the actual distance; if it is greater than 1, it means the change is too fast, and the estimated distance is closer than the actual distance. The Eikonal loss ensures that the sign vector field learns a distance field with accurate measurement; provides smooth and reasonable gradients; stabilizes training and improves convergence.

[0054] The two-dimensional Gaussian splash branch renders the loss function. and These functions are used to supervise and measure the difference between the rendered RGB image and the true value. The loss function is the sum or average of the absolute differences in color values ​​of each pixel between the rendered image output by the model and the real target image. In the loss function, SSIM is a metric used to measure the perceptual quality of two images. Its core idea is that the human visual system is most sensitive to structural information in an image. Within a local window, SSIM compares the following three aspects of the two images: brightness (comparing the average gray levels of the images); contrast (comparing the standard deviation of the images, i.e., the range of pixel value fluctuations); and structure (comparing the correlation of the images after removing the effects of brightness and contrast, i.e., structural information).

[0055] in: and : These are the average pixel values ​​(representing brightness) for image blocks x and y, respectively.

[0056] and : These are the standard deviations of the pixel values ​​of image patches x and y, respectively (representing contrast).

[0057] The covariance of image patches x and y (representing structural correlation).

[0058] : is a constant used to avoid the denominator being zero.

[0059] The purpose of this invention is to overcome the defects and shortcomings of existing 3D neural rendering technology in practical interactive applications. It proposes a 3D scene rendering and editing method and system that integrates 3D Gaussian splashing with the traditional rasterization rendering pipeline to achieve the unification of high-quality visual reconstruction, real-time rendering display, geometrically accurate constraints and interactive editing, thereby significantly improving rendering efficiency and visualization interaction performance while ensuring the quality of neural rendering.

[0060] Addressing the Editability Challenge: In the 3D Gaussian splashing method, each Gaussian primitive in the scene exists only as coordinates, orientation, and scale parameters, lacking explicit geometric structure. This makes it impossible to directly manipulate by traditional 3D modeling or rendering engines, resulting in complex and costly editing and adjustment processes. This invention achieves structural conversion between Gaussian primitives and rasterized triangular meshes, enabling the final trained 3D scene representation to be directly recognized and rendered by the GPU. Simultaneously, a GPU-based Gaussian editing interface is constructed, allowing users to directly adjust Gaussian position, normals, color, and transparency parameters in the real-time rendering view within the UE4 system, achieving visual editing and dynamic updates.

[0061] Addressing the challenge of geometric weakness: 3D Gaussian splashing technology, based solely on color reprojection error optimization, lacks geometric consistency constraints, leading to some Gaussian elements floating outside the object's surface, resulting in problems such as blurred edges, shape damage, and depth discontinuities. This invention introduces a mutual supervision mechanism between a symbolic vector field (SDF) and a vector field, using joint training to prevent Gaussian elements from drifting off the surface, thereby improving geometric stability and structural accuracy.

[0062] Expanding application scope: This invention not only achieves the integration of 3D Gaussian splashing and traditional rasterization pipeline at the algorithm level, but also greatly expands the application scope of 3D neural rendering technology in terms of rendering architecture, geometric expression form and interaction method. This makes the 3D scene representation trained by this invention have comprehensive characteristics of real-time performance, editability and portability. Because the 3D scene representation generated by this invention is compatible with traditional rendering pipelines, it can be quickly integrated with existing VR / AR engines to support real-time dynamic scene rendering in VR headsets and neurorealistic enhancement displays in AR devices. Furthermore, this invention can efficiently express and render high-dimensional lighting and geometric information of realistic scenes, supporting dynamic updates and interactive editing. Therefore, it is particularly suitable for: real-time 3D visualization of digital twin cities and smart factories; interactive simulation systems in industries such as architecture, transportation, and energy; and industrial inspection and state reproduction systems based on 3D representations. Through temporal anti-aliasing rendering and parameter baking mechanisms, this invention allows Gaussian rendering results to be directly used for real-time lighting simulation and special effects production. In film, animation, and game production, it can achieve: hybrid rendering of 3D scenes and materials; high-quality dynamic blur, transparent transitions, and depth-of-field effects; and parametric editing and non-destructive rendering reconstruction of scene assets.

[0063] This invention proposes to flatten 3D Gaussian primitives in a 2D projection form and discretize them into two triangular meshes. The rotation, scale, and normal information of each Gaussian primitive are transformed into a vertex matrix that can be directly rendered by the GPU, enabling the neural rendering results to have an explicit geometric representation.

[0064] This invention encodes the spherical harmonic coefficients and opacity parameters of Gaussian primitives into multi-channel textures and constructs a unified texture atlas. Through texture block indexing and UV mapping rules, it achieves independent storage and efficient sampling of each Gaussian primitive in the texture space. This significantly reduces the memory access overhead during the rendering stage and ensures the parallel rendering performance of multiple Gaussian instances.

[0065] This invention introduces a signed vector field branch and its gradient vector field during the training phase, which are jointly trained and mutually supervised with the two-dimensional Gaussian branch. Through constraints and surface alignment loss, the Gaussian center is kept consistent with the surface of the real object, avoiding Gaussian drift and shape distortion. This results in more geometrically accurate rendering output with higher surface continuity, significantly improving the model's convergence stability.

[0066] Test simulation: Rendering quality: The qualitative results of the 3D scene obtained by training in this invention based on traditional rasterization pipeline rendering are as follows: Figure 3 As shown. The rightmost column displays the real-time rendering results obtained through Unreal Engine 4; the leftmost column shows real-world images captured from the dataset; and the middle column shows the rendering effect of the 3D Gaussian splashing method using its dedicated rasterization pipeline. Overall, no significant difference was observed in the rendering results of the two methods. In some fine-grained areas, this invention can also achieve fine detail rendering. This invention achieves the same reconstruction efficiency and rendering quality as the 3D Gaussian splashing method, while using a traditional rasterization pipeline, extending its application to various commonly used real-time rendering engines such as Unreal Engine.

[0067] Mesh extraction: like Figure 4 As shown, the first row is the mesh obtained by TSDF fusion of the two-dimensional Gaussian sputtering results, and the second row is the mesh directly extracted from the symbolic vector field in this invention. Compared with the mesh obtained by TSDF fusion of the two-dimensional Gaussian sputtering results, the mesh directly extracted from the symbolic vector field in this invention can present superior geometric details.

Claims

1. A view compositing system that fuses three-dimensional Gaussian splottering with a rasterization pipeline, characterized in that, The data input and preprocessing module, the two-dimensional Gaussian splash module, the Gaussian cell conversion grid module, the parameter baking module, the UV mapping and standardization module, the symbolic vector field joint training and constraint module, and the rasterization rendering and editing module; The data input and preprocessing module preprocesses the input data, obtains a data set, and uses the data set as the initialization state of two-dimensional Gaussian cell training; The two-dimensional Gaussian splash module optimizes the preprocessed data set and obtains two-dimensional Gaussian cells; The Gaussian cell conversion grid module, the parameter baking module, and the UV mapping and standardization module jointly convert the two-dimensional Gaussian cells obtained by the two-dimensional Gaussian splash module into polygonal meshes that can be rendered in a GPU rendering pipeline, and map the spherical harmonic coefficients and opacity parameters of the two-dimensional Gaussian cells to texture atlases, thereby establishing a correspondence between the polygonal meshes and the texture atlases, achieving consistency of the two-dimensional standard Gaussian function in the UV space, and facilitating rendering of two-dimensional images and editing of three-dimensional scenes in the rasterization rendering and editing module; The symbolic vector field joint training and constraint module constrains the two-dimensional Gaussian splash module, so that the Gaussian point cloud is closer to the real surface and the geometric precision is improved.

2. The three-dimensional Gaussian spill and rasterization pipeline fused view synthesis system of claim 1, wherein, In the data input and preprocessing module, the input data set is a plurality of view pictures around the same scene, and a method based on a motion recovery structure SfM is used to analyze the plurality of view pictures to recover camera parameters and a simple three-dimensional scene structure sparse point cloud as the initialization state of two-dimensional Gaussian cell training; The two-dimensional Gaussian splash module uses a two-dimensional Gaussian splash algorithm to optimize the point cloud and obtain a two-dimensional Gaussian cell set with color, normal, and scale, and the two-dimensional Gaussian splash uses a flattened Gaussian ellipsoid to represent objects in the scene; The Gaussian cell conversion grid module converts the two-dimensional Gaussian cells trained by the two-dimensional Gaussian splash training module into polygonal meshes that can be rendered in a GPU rendering pipeline; The parameter baking module maps the spherical harmonic coefficients and opacity parameters of the Gaussian cells to a texture map; The UV mapping and standardization module establishes a correspondence between the polygonal meshes and the texture map, and realizes numerical consistency of the two-dimensional standard Gaussian function in the UV space; The symbolic vector field joint training and constraint module jointly optimizes the symbolic vector field and the two-dimensional Gaussian splash training, constrains the Gaussian point cloud to be close to the real surface, improves the geometric precision, and generates a collision mesh; The rendering process of the rasterization rendering and editing module is executed in Unreal Engine 4 through an HLSL shader.

3. A view compositing method for implementing a three-dimensional Gaussian splottering and rasterization pipeline fusion of the system of claim 1 or 2, characterized by, The method comprises the following steps: Step 1: using an original Gaussian splash technology to train the scene and flattening three-dimensional Gaussian cells used to represent the scene to obtain two-dimensional Gaussian cells, each cell containing normal information; Step 2: each trained two-dimensional Gaussian cell is converted into two triangular meshes, and the two triangles are transformed into a world coordinate system as a traditional GPU renderable geometry; Step 3: after being converted into two triangular meshes, the spherical harmonic coefficients and opacity data used to represent the color of each two-dimensional Gaussian cell are baked in the form of a texture. Step 4: Bake both triangles of the grid into the same texture atlas, each Gaussian cell corresponds to a texel in the texture atlas, and the center of the texel is aligned with the Gaussian center. The triangles cover the texel in UV space, and the Gaussian weight is obtained; Step 5: Based on the obtained Gaussian weight, use the time-based dithering anti-aliasing method to simulate the alpha blending behavior in the three-dimensional Gaussian splatting pipeline, so as to visually approach the synthesis effect of the three-dimensional Gaussian splatting technique; Step 6: Train a symbolic vector field SDF branch at the same time, and supervise and constrain each other between the symbolic vector field and the output of the two-dimensional Gaussian to reduce the phenomenon of some Gaussian cells drifting away from the object surface and improve the reconstruction rendering quality.

4. The view synthesis method for fusing three-dimensional Gaussian splashing and rasterized pipelines according to claim 3, characterized in that, In step 1, the operation of "flattening" the three-dimensional Gaussian primitives is used, transforming them into a set of two-dimensional Gaussian planes embedded in the three-dimensional space. Each two-dimensional Gaussian primitive has multiple attributes: center point... Two tangential vectors normal line The two corresponding scaling vectors Opacity Color information represented using spherical harmonic functions and rotation matrix ; By constructing a homogeneous transformation matrix: wherein: , denotes a diagonal matrix; Cut plane coordinates Map to world coordinates: where: represents the center point of the two-dimensional Gaussian cell; are the scaling vectors of the two-dimensional Gaussian cell in the two directions of the UV space; are the two tangent vectors of the two-dimensional Gaussian cell in the UV space; is the homogeneous transformation matrix mentioned before. Flat standard two-dimensional Gaussian: wherein: i.e. tangent plane coordinates of a two-dimensional Gaussian cell .

5. The method of claim 4, wherein the method further comprises: determining a Gaussian blur value for each pixel of the image based on the Gaussian blur value of the pixel and the Gaussian blur value of the pixel's neighboring pixels; and applying the determined Gaussian blur value to each pixel of the image. In step 2, each two-dimensional Gaussian cell is converted into two triangular meshes and transformed into world coordinates, generating GPU renderable geometry for each two-dimensional Gaussian cell, making it a standard Mesh in the traditional rendering pipeline; In the standard normal distribution: is the mean and σ is the standard deviation in this distribution: In the Gaussian tangent plane ( Within the space, taking the range of ±3σ, construct a square and divide it into two triangles using a standardized two-dimensional Gaussian. with the homogeneous transformation matrix Transform the triangle vertices from clip space to world space: For the clip plane point , the world coordinates are .

6. The method of claim 5, wherein the method further comprises: determining a Gaussian blur value for each pixel of the image based on the Gaussian blur value of the pixel and the Gaussian blur value of the pixel's neighboring pixels; and applying the determined Gaussian blur value to each pixel of the image. In step 3, The baking process writes the color-representing spherical harmonic coefficients and opacity of each Gaussian cell into a small texture atlas in the texture atlas. The color of each Gaussian cell is represented by a 48-dimensional spherical harmonic function vector. The 48-dimensional information of the spherical harmonic coefficients is stored in 12 four-channel RGBA textures, and the opacity is stored in a single-channel texture. In the rendering stage, the GPU shader obtains the lighting, color, and opacity information of the Gaussian cell by sampling the texture atlas.

7. The view synthesis method for fusing three-dimensional Gaussian splashing and rasterization pipelines according to claim 6, characterized in that, In step 4, it is ensured that two triangles corresponding to the same Gaussian are sampled to the same texture unit in the texture space, and the Gaussian weight is calculated by the UV distance in the same texture unit; each Gaussian cell is allocated to a texture block in the texture atlas, and the center of the texture block is taken as the Gaussian center; the UV vertices of the two triangles are mapped to cover the texture block; using nearest neighbor sampling, it is ensured that the texture reading of the shader to the block all points to the same texture block; for the picture in the data set, if each Gaussian cell corresponds to one texture block, the corresponding small texture grid of a Gaussian cell is located at The UV vertex coordinates of the corresponding two triangles are: Triangle mesh A: Triangle mesh B: Compute the mapping from UV to gnomonic local coordinates in the GPU shader and compute the difference to the texture block center and finally compute the gnomonic weight as follows: where is the mapping of UV coordinates to Gaussian local coordinates is the difference of the mapping of UV coordinates to Gaussian local coordinates and the center of the texture block, is the standard deviation of the two-dimensional Gaussian distribution.

8. The method of claim 7, wherein the method further comprises: In step 5, the time-based dithering anti-aliasing method discretizes the opacity into a 0 / 1 Boolean mask with a time-based dithering mask per frame or pixel and averages in the temporal dimension over consecutive frames, approximating continuity mixing; For a Gaussian primitive to be rendered, the final opacity of the primitive is calculated on the fragment shader by multiplying the baked opacity with the Gaussian weight with the Gaussian weight​ Write the binary mask to the color buffer if the comparison is true. Write the binary mask to the color buffer if the comparison is true. In Dither Temporal AA and temporal accumulation, the binary writing of consecutive frames is visually restored to continuous alpha blending effect.

9. The method of claim 8, wherein, In step 6, the signed distance field SDF is used to describe the signed distance from any point in space to the surface of an object; SDF is a continuous scalar field that is zero for every point on the surface of the object returns a real value if the point is outside the surface of the object, the value of SDF is positive ​ If the point lies inside the surface of the object, then is negative; if the point lies on the surface of the object, then is zero.

Images