A three-dimensional gaussian reconstruction and rendering method and device for a high light reflection scene

Abstract

The application discloses a three-dimensional Gaussian reconstruction and rendering method and device for a high-light reflection scene. The method first constructs a hybrid scene representation, including a basic Gaussian primitive for representing the basic geometry and appearance of the scene, and a texture environment triangle for representing high-frequency environmental reflection; in the rendering stage, a deferred shading architecture is adopted, the basic Gaussian primitive is rasterized to obtain basic attribute maps and reflection vectors, then the environmental triangle is sampled by using hardware-accelerated ray tracing technology to obtain reflection colors, and finally high-fidelity images are generated by fusion; in the optimization stage, a progressive opacity annealing strategy is adopted, so that the environmental triangle converges from semi-transparent to completely opaque, thereby the rendering speed is greatly improved by using the nearest hit logic during reasoning. By decoupling geometry and appearance, combining UV texture mapping and opaque geometry proxy, high-quality real-time reflection rendering on mobile devices is realized.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics and 3D vision, and particularly relates to a method and apparatus for 3D Gaussian reconstruction and rendering of specular reflection scenes. Background Technology

[0002] With the development of neural radiation fields and 3D Gaussian sputtering technology, the synthesis of new perspectives in static scenes can now achieve real-time and realistic effects. However, existing 3D Gaussian sputtering methods typically rely on spherical harmonic functions to simulate viewpoint-dependent colors, which performs poorly when dealing with high-frequency reflections on smooth surfaces (such as mirrors and metals), often resulting in blurry rendering results. To address the reflection problem, existing technologies attempt to introduce environment mapping or utilize Gaussian spheres for ray tracing. However, environment mapping assumes an infinitely distant light source and cannot handle near-field reflections and parallax; while ray tracing methods based on Gaussian spheres, due to the semi-transparent nature of Gaussian primitives, require sorting and blending the transparency of all intersecting primitives along the ray path during rendering, resulting in huge computational overhead and making it difficult to achieve real-time rendering on mobile devices or resource-constrained devices. Furthermore, while using opaque geometry (such as meshes) can accelerate ray tracing using hardware, expressing high-frequency texture details through vertex colors often requires an extremely large number of triangles, which in turn reduces the speed of traversing the bounding volume hierarchy. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by proposing a 3D Gaussian reconstruction and rendering method and apparatus for specular reflection scenes. By constructing a hybrid representation of the basic Gaussian triangle and the environment triangle, decoupling geometry and appearance using UV texture mapping, and combining an opacity annealing strategy, dynamic reflection scene reconstruction with both high rendering quality and high inference speed is achieved, effectively solving the problems of high computational overhead and difficulty in real-time operation of reflection rendering in existing technologies.

[0004] The objective of this invention is achieved through the following technical solution: a method for three-dimensional Gaussian reconstruction and rendering of specular reflection scenes, the method comprising: (1) Initialize the hybrid scene representation model, which includes a set of basic Gaussian elements for modeling the basic geometry and diffuse appearance of the scene, and a set of environmental triangles for modeling the high-frequency environmental reflection field; (2) Using the deferred shading pipeline, the basic Gaussian elements are first rasterized to generate a geometric property map in screen space; (3) Calculate the reflected light rays based on the geometric attribute graph, perform ray tracing query on the environmental triangle set, and obtain the reflected color; (4) The diffuse color obtained by rasterization and the specular color obtained by ray tracing are weighted and fused to generate the final target view image; (5) During the training process, the parameters of the basic Gaussian primitives and the environment triangle are jointly optimized using the rendering loss function, and the opacity annealing strategy is used to make the environment triangle converge into an opaque surface.

[0005] Furthermore, in step (1), the scene's basic geometry includes normals and depth.

[0006] Furthermore, in step (1), each triangle primitive in the set of environment triangles includes vertex position, UV texture coordinates, texture map and residual spherical harmonic coefficients; learnable UV texture maps are used to represent high-frequency reflection details, and residual spherical harmonic coefficients are used to represent viewpoint-related appearance changes, thereby achieving high-fidelity texture detail expression while maintaining low geometric complexity.

[0007] Further, in step (2), the basic Gaussian elements are represented by 2D Gaussian sputtering, and the rasterization process outputs geometric and appearance attributes including basic color, surface normal, depth, and blending weights; the blending weights are used to indicate the degree to which the pixel region is affected by the reflection component.

[0008] Further, in step (3), the execution of ray tracing query specifically includes: calculating the reflection vector using the surface normal and viewing direction obtained by rasterization, and using the reflection vector as the ray direction; in the inference stage, the hardware ray tracing core is used to execute the nearest hit query, and only the color of the intersection point of the ray and the nearest environment triangle is calculated, avoiding the sorting and mixing of transparent primitives along the way, thereby realizing real-time rendering.

[0009] Further, in step (5), the opacity annealing strategy is as follows: in the early stage of training, the environment triangle is regarded as a semi-transparent medium, and the color is accumulated by volume rendering to ensure gradient propagation; as training progresses, the opacity parameter and softening coefficient are gradually adjusted to force the opacity of the environment triangle to tend towards 0 or 1 binarization, and finally form an opaque geometric surface to adapt to the nearest hit ray tracing algorithm in the inference stage.

[0010] Furthermore, step (5) also includes a gradient-based pruning strategy, which calculates the gradient contribution of the basic Gaussian primitives in the final target view image, comprehensively evaluates their total importance as emitters and reflectors, and removes redundant basic Gaussian primitives accordingly to improve rendering efficiency.

[0011] Secondly, the present invention also provides a three-dimensional Gaussian reconstruction and rendering apparatus for a specular reflection scene, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements any one or more of the above methods.

[0012] Thirdly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the above-mentioned method for three-dimensional Gaussian reconstruction and rendering of specular reflection scenes.

[0013] Fourthly, the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned method for three-dimensional Gaussian reconstruction and rendering of a specular reflection scene.

[0014] The beneficial effects of this invention are as follows: This invention utilizes the combination of opaque triangles and UV textures. On one hand, it leverages the efficient intersection capabilities of the GPU hardware ray tracing core for opaque geometry, avoiding the sorting and blending of semi-transparent Gaussian surfaces required in traditional methods. On the other hand, texture mapping reduces the number of triangles needed, thus reducing the overhead of traversing the bounding volume hierarchy. By introducing a specialized environment field representation, it effectively solves the blurring problem in traditional 3D Gaussian splashing when handling high-frequency reflections. Furthermore, compared to environment mapping methods, this invention can correctly handle near-field reflections and occlusion relationships. Attached Figure Description

[0015] Figure 1 This is a flowchart of a three-dimensional Gaussian reconstruction and rendering method for a specular reflection scene provided by the present invention.

[0016] Figure 2 This is a schematic diagram of the structure represented by the hybrid Gaussian and environmental triangles of this invention.

[0017] Figure 3 This is a structural diagram of a three-dimensional Gaussian reconstruction and rendering device for a specular reflection scene according to the present invention. Detailed Implementation

[0018] The technical details and principles of the present invention will be further described below with reference to the accompanying drawings: like Figure 1 As shown, this invention proposes a 3D Gaussian reconstruction and rendering method for specular reflection scenes, including: This invention jointly optimizes basic Gaussian elements and environment triangles from multi-view images to establish a hybrid scene representation containing high-frequency reflection details. Specifically, it acquires multi-view scene images, initializes the scene using sparse point clouds, constructs a hybrid model composed of basic Gaussian elements and environment triangles with UV textures, and introduces an opacity annealing strategy for efficient modeling of complex reflection scenes.

[0019] The optimized blending model can be used for real-time rendering from any viewpoint. Specifically, based on the input viewpoint, a deferred shading architecture is adopted. First, the basic Gaussian primitives are rasterized to obtain the scene depth, normal, blending weights, and basic colors. Then, hardware-accelerated ray tracing technology is used to sample the opaque environment triangles to obtain the reflected colors. Finally, the models are fused to generate a high-fidelity image, achieving real-time rendering at the mobile device level.

[0020] The overall process of this invention specifically includes the following steps: (1) Use basic Gaussian primitives to explicitly model the basic geometry (normals, depth), basic appearance and blending weights of the scene; at the same time, introduce environment triangles as environment fields and use UV texture maps and residual spherical harmonic coefficients to efficiently store high-frequency reflection details. The specific process of initializing the hybrid scene representation is as follows: The sparse point cloud of the scene is obtained using the Structure for Motion Restoration (SfM) algorithm, which is used to initialize the spatial properties of the basic Gaussian primitives. For environment triangles, a voxel mesh is constructed based on the scene bounding box, and an initial set of triangles is generated by uniform sampling within the mesh. Simultaneously, each environment triangle primitive contains vertex positions, UV texture coordinates, texture maps, and residual spherical harmonic coefficients. The UV textures are used to store high-frequency spatial texture details, and the residual spherical harmonic coefficients are used to compensate for viewpoint-dependent appearance changes, thus forming a decoupled representation of scene geometry and high-frequency reflection environment, achieving high-fidelity texture detail expression while maintaining low geometric complexity. Figure 2 As shown, in the hybrid model of basic Gaussian elements and opaque environment triangles proposed in this invention: For the basic Gaussian elements, a 2D Gaussian sputtering representation is used to generate accurate surface normals. ,depth Basic colors and mixed weights 2DGS provides more accurate geometric surfaces than 3DGS, making it suitable as a source of reflected light. The blending weights are used to indicate the degree to which a pixel region is affected by the reflection component.

[0021] For the environment triangle, define a set. Each triangular facet Includes a learnable UV texture map. and the remaining spherical harmonic coefficients For the intersection of the ray and the triangle, use the centroid coordinates of that intersection point. The formula for calculating color is: in For the direction of light, This indicates a bilinear interpolation operation, which obtains texture color through bilinear interpolation, and the spherical harmonic coefficients supplement viewpoint-dependent specular variations.

[0022] (2) First, the basic Gaussian elements are rasterized to obtain the geometric property map; then, the reflected rays are calculated based on the surface normals, and the opaque environment triangles are sampled using hardware-accelerated ray tracing technology; finally, the final image is synthesized based on the hybrid weights. The specific process of rendering the primitives at the current moment is as follows: First, a deferred shading architecture is used to rasterize the basic Gaussian primitives to screen space, generating a geometric property map containing normals, depth, and blending weights; second, the reflection vector is calculated based on the normals and viewing direction in the geometric property map, and the environment triangles are sampled using hardware ray tracing technology; during the inference phase, the color of the intersection point between the ray and the opaque triangle is directly obtained using the nearest hit logic, avoiding the sorting and blending of transparent primitives on the ray path; finally, the base color and the reflection color are weighted and blended according to the blending weights to output the target image.

[0023] The specific steps of ray tracing include: calculating the reflection vector using the surface normals and view direction obtained from rasterization, and using the reflection vector as the ray direction; during the inference phase, using the hardware ray tracing core to perform the nearest hit query, and only calculating the color of the intersection point of the ray and the nearest environment triangle to achieve real-time rendering.

[0024] During rendering, the present invention employs the following logic for deferred shading: 1. Use rasterization to render basic Gaussian primitives, outputting pixel-level normals, depth, diffuse color, and blend weights.

[0025] 2. Calculate the reflection vector for each pixel. ,in The direction of the light emitted from the camera by this pixel. This is the surface normal output for the corresponding pixel-based Gaussian primitive rendering. Ray tracing is performed on the environment triangles using this vector to obtain the intersection of the nearest triangles and calculate the corresponding color. The specular reflection color of this pixel .

[0026] 3. The final pixel color is .

[0027] (3) An annealing training strategy from semi-transparent to opaque was designed. In the early stage of training, light is allowed to pass through and accumulate gradients to optimize the geometric position; in the later stage of training, the geometry is forced to converge to an opaque surface, so that the nearest hit logic can be directly used to replace the expensive volume rendering sorting during inference, which greatly improves the speed.

[0028] The specific process for optimizing Gaussian elements and triangle parameters is as follows: an opacity annealing strategy is adopted during training. In the early stages of training, the environment triangle is treated as a semi-transparent medium, and volumetric rendering is used to accumulate color contributions on the ray path to ensure effective propagation of geometric gradients. As training progresses, the opacity parameters and softening coefficient are gradually adjusted to force the opacity of the environment triangle to converge towards binary 0 or 1, ultimately forming an opaque geometric surface to adapt to the nearest-hit ray tracing algorithm during the inference phase.

[0029] During training, periodic relocalization and pruning are also implemented. Specifically, probability sampling is performed based on the visible weight and surface area of ​​the environment triangles in the rendered view. Triangles in high-contribution areas are subdivided, and the child triangles inherit the texture features of the parent triangle. At the same time, opacity thresholds and visible weight thresholds are set, and redundant or occluded triangles are automatically pruned to achieve adaptive adjustment of the geometric density of the environment field.

[0030] To enable differentiable optimization of triangle positions and textures, this invention employs a three-stage optimization and annealing process: 1. In the first stage, triangles are treated as semi-transparent, and light passes through multiple triangles and is rendered according to the volumetric rendering formula. Cumulative colors, among which The first one along the direction of light, respectively Transparency blending weight of intersecting triangles and the color of specular reflection Specifically, each vertex of the environment triangle Includes a learnable opacity parameter. The opacity of each triangle This represents the minimum opacity of the three vertices. For the intersection of the ray and the triangle, use the centroid coordinates of that intersection point. Calculate the transparency blending weight at this point. .in, , The opacity threshold. This is the softening coefficient.

[0031] 2. In the second stage, as training progresses, the flexibility coefficient is gradually reduced. And increase the opacity threshold This forces the transparency to blend weights. It tends towards 1, which means completely opaque.

[0032] 3. In the third stage, the geometry is completely fixed as opaque, with only minor texture adjustments. At this point, switch to the most recently hit rendering mode.

[0033] The optimization objectives employed in this invention include RGB reconstruction loss (L1 + SSIM + Perceptual) and normal consistency loss for constraining the underlying geometry. During triangle densification, this invention uses the triangle area as a probability distribution to sample triangles for subdivision, and uses the midpoints of the three sides of the triangle to further subdivide the triangle, ensuring that the segmented triangles retain the texture of the corresponding regions of the original triangle. Based on triangle opacity and visibility thresholds, this invention prunes the triangles, effectively removing redundant geometry.

[0034] Corresponding to the aforementioned embodiments of the three-dimensional Gaussian reconstruction and rendering method for specular reflection scenes, the present invention also provides embodiments of a three-dimensional Gaussian reconstruction and rendering apparatus for specular reflection scenes.

[0035] See Figure 3 The present invention provides a three-dimensional Gaussian reconstruction and rendering device for a specular reflection scene, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a three-dimensional Gaussian reconstruction and rendering method for a specular reflection scene as described in the above embodiment.

[0036] An embodiment of the 3D Gaussian reconstruction and rendering device for a specular reflection scene provided by this invention can be applied to any device with data processing capabilities, such as a computer, graphics workstation, server, or virtual reality device. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is the 3D Gaussian reconstruction and rendering device for a specular reflection scene provided by the present invention. (Except for...) Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware modules depending on the actual function of the device, which will not be described in detail here.

[0037] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0038] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0039] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a three-dimensional Gaussian reconstruction and rendering method for a specular reflection scene as described in the above embodiments.

[0040] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0041] The present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-described method for three-dimensional Gaussian reconstruction and rendering of a specular reflection scene.

[0042] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for 3D Gaussian reconstruction and rendering of specular reflection scenes, characterized in that, The method includes: (1) Initialize the hybrid scene representation model, which includes a set of basic Gaussian elements for modeling the basic geometry and diffuse appearance of the scene, and a set of environment triangles for modeling specular reflection; (2) Using the deferred shading pipeline, the basic Gaussian elements are first rasterized to generate a geometric property map in screen space; (3) Calculate the reflected light rays based on the geometric attribute graph, perform ray tracing query on the environmental triangle set, and obtain the reflected color; (4) The diffuse color obtained by rasterization and the specular color obtained by ray tracing are weighted and fused to generate the final target view image; (5) During the training process, the parameters of the basic Gaussian primitives and the environment triangle are jointly optimized using the rendering loss function, and the opacity annealing strategy is used to make the environment triangle converge into an opaque surface.

2. The method according to claim 1, characterized in that, In step (1), the scene's basic geometry includes normals and depth.

3. The method according to claim 1, characterized in that, In step (1), each triangle primitive in the set of environment triangles includes vertex position, UV texture coordinates, texture map and residual spherical harmonic coefficients; learnable UV texture maps are used to represent specular reflection details, and residual spherical harmonic coefficients are used to represent viewpoint-related appearance changes, thereby achieving high-fidelity texture detail expression while maintaining low geometric complexity.

4. The method according to claim 1, characterized in that, In step (2), the basic Gaussian primitive is represented by 2D Gaussian sputtering, and the rasterization process outputs geometric and appearance attributes including diffuse color, surface normal, depth, and blending weight; the blending weight is used to indicate the degree to which the pixel region is affected by the reflection component.

5. The method according to claim 1, characterized in that, In step (3), the execution of ray tracing query specifically includes: calculating the reflection vector using the surface normal and viewing direction obtained by rasterization, and using the reflection vector as the ray direction; in the inference stage, the hardware ray tracing core is used to execute the nearest hit query, and only the color of the intersection point of the ray and the nearest environment triangle is calculated, avoiding the sorting and mixing of transparent primitives along the way, thereby realizing real-time rendering.

6. The method according to claim 1, characterized in that, In step (5), the opacity annealing strategy is as follows: in the early stage of training, the environment triangle is regarded as a semi-transparent medium, and the color is accumulated by volume rendering to ensure gradient propagation; as training progresses, the opacity parameter and softening coefficient are gradually adjusted to force the opacity of the environment triangle to tend towards 0 or 1 binarization, and finally form an opaque geometric surface to adapt to the nearest hit ray tracing algorithm in the inference stage.

7. The method according to claim 1, characterized in that, Step (5) also includes a gradient-based pruning strategy, which calculates the gradient contribution of the basic Gaussian primitives in the final target view image, comprehensively evaluates their total importance as emitters and reflectors, and removes redundant basic Gaussian primitives accordingly to improve rendering efficiency.

8. A three-dimensional Gaussian reconstruction and rendering apparatus for a specular reflection scene, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a three-dimensional Gaussian reconstruction and rendering method for a specular reflection scene as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a three-dimensional Gaussian reconstruction and rendering method for a specular reflection scene as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement a three-dimensional Gaussian reconstruction and rendering method for a specular reflection scene as described in any one of claims 1-7.

Images