Real-time relighting method based on 2D Gaussian splashing and screen space light stepping

Abstract

The invention discloses a real-time relighting method based on 2D Gaussian splashing and screen space light stepping, and the method comprises the following steps: collecting a multi-view image of an object model and a scene, and carrying out the initial reconstruction of the multi-view image through a motion recovery structure algorithm, so as to obtain an initial 2D Gaussian point cloud; obtaining a 2D Gaussian point cloud with complete geometric and material attributes; the method comprises the following steps: generating a geometric buffer area comprising depth, normal and material attributes based on a 2D Gaussian point cloud with complete geometric and material attributes, calculating a direct illumination result based on the geometric buffer area and an environment illumination map, calculating an indirect illumination result by adopting a screen space light stepping algorithm, and calculating a real-time illumination result. And synthesizing the direct illumination result and the indirect illumination result to output a real-time relighting result. According to the method, the technical problem that the existing relighting technology based on ray tracing cannot be real-time can be solved, and the rendering quality and the reality sense of relighting are improved through the global illumination of the screen space and the optimized illumination sampling strategy on the premise of ensuring the real-time performance.

Description

Technical Field

[0001] This invention relates to the field of graphics rendering technology, specifically to a real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping. Background Technology

[0002] Gaussian splashing is an emerging method for representing and rendering 3D scenes, approximating the geometry and appearance of a scene using a large number of 3D Gaussian distributions. Because the entire rendering process is differentiable, it can achieve high-quality 3D reconstruction results through multi-view image optimization. However, the original Gaussian splashing model couples lighting, material, and other information into spherical harmonic functions, making it unable to support relighting.

[0003] Relighting refers to the technique of modeling the material, geometry, and other information of an object under different lighting conditions, and then re-rendering it to produce a realistic effect using lighting equations. To achieve Gaussian splash-based relighting, inverse rendering is required, which involves decoupling the material, geometry, and lighting information of the scene from the image.

[0004] Existing technologies have attempted to address the relighting problem in Gaussian splatting. For example, IRGS (Inter-Reflective Gaussian Splatting) technology uses 2D Gaussian ray tracing to sample indirect lighting. Although it produces high-quality rendering, the complexity of its ray tracing algorithm is related to the complexity of the scene, resulting in huge computational costs. This makes it impossible to meet the requirements of real-time rendering (typically referring to higher than 30 frames per second), and thus difficult to apply in real-time interactive scenarios such as games and VR / AR.

[0005] Therefore, there is an urgent need in this field for a relighting method that can achieve real-time performance while ensuring high-quality rendering effects. Summary of the Invention

[0006] To address the aforementioned shortcomings in the existing technology, the present invention provides a real-time relighting method based on 2D Gaussian splashing and screen space light stepping.

[0007] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: A real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping includes the following steps: Collect multi-view images of the object model and scene, and use the structure-reconstruction-motion algorithm to perform initial reconstruction to obtain an initial 2D Gaussian point cloud; The properties of the initial 2D Gaussian point cloud are optimized, and the material parameters of the property-optimized 2D Gaussian point cloud are optimized to obtain a 2D Gaussian point cloud with complete geometric and material properties. Based on a 2D Gaussian point cloud with complete geometric and material properties, a geometric buffer including depth, normals, and material properties is generated. Direct lighting results are calculated based on the geometric buffer and ambient light map, and indirect lighting results are calculated using a screen-space ray stepping algorithm. The direct and indirect lighting results are then synthesized to output real-time relighting results.

[0008] Furthermore, the initial 2D Gaussian point cloud is subjected to attribute optimization, and the expression of the loss function used is as follows: in: This is the loss during the attribute optimization phase. For RGB reconstruction loss, The weighting coefficients for the normal consistency loss are: For loss of normal uniformity, The weighting coefficients for deep concentration loss are... For deep concentration loss, These are the weighting coefficients for the normal smoothing loss. For normal smoothing loss, The weighting coefficients for masking loss, To mask out the loss of the mask.

[0009] Furthermore, the material parameters of the attribute-optimized 2D Gaussian point cloud are optimized, and the expression of the loss function used is as follows: in: This is due to losses during the material parameter optimization phase. This is the loss during the attribute optimization phase. The weighting coefficients for the PBR consistency loss are: For PBR consistency loss, These are the weighting coefficients for illumination regularization loss. For illumination normalization loss, The weighting coefficients for albedo smoothing loss are... To smooth the albedo loss, These are the weighting coefficients for roughness smoothing loss. For roughness smoothing loss, The weighting coefficients for the metallicity smoothing loss are: This is to smooth out the loss of metallicity.

[0010] Furthermore, direct lighting results are calculated based on the geometry buffer and environment lightmap. The specific process is as follows: By utilizing the normal, depth, and roughness information in the geometry buffer, the bending normal of each pixel is dynamically calculated; The pre-baked ambient light map is sampled using the curved normal of each pixel to obtain ambient occlusion corrected ambient diffuse color values, and then the direct lighting result is synthesized using the ambient occlusion corrected ambient diffuse color values ​​and physically based rendering equations.

[0011] Furthermore, the bending normal of each pixel is dynamically calculated, specifically as follows: Based on the depth information in the geometric buffer, each pixel is back-projected to the world coordinate system. A set of sampling directions is generated within the hemisphere with the normal direction as the center, and the position of the sampling point is obtained. Project each sampling direction onto the screen space, query the depth of the corresponding pixel in the geometry buffer, and obtain the position of the query point; By comparing the relative positions of the sampling point and the query point in the world coordinate system, it can be determined whether the sampling direction is occluded. Accumulate all unoccluded sampling direction vectors and calculate the average vector to obtain the curvature normal.

[0012] Furthermore, the screen-space ray stepping algorithm is used to calculate the indirect lighting results. The specific process is as follows: For the pixel to be rendered in screen space, multiple sampling rays are generated based on its normal and roughness information; A hierarchical depth buffer is used to perform step and depth intersection detection on each sampling ray. If the sampling ray intersects with a pixel in the screen space, the direct illumination result of the intersecting pixel is included in the indirect illumination contribution. The indirect lighting contributions of all sampled rays are weighted and summed to obtain the indirect lighting results for the pixel to be rendered.

[0013] Furthermore, the method for constructing the hierarchical depth buffer is as follows: taking the original depth in the geometric buffer as the first level, downsampling layer by layer, and taking the maximum depth of the corresponding 2x2 pixel region of the lower layer as the depth of each upper layer pixel to form a depth pyramid, thereby constructing the hierarchical depth buffer.

[0014] Furthermore, a hierarchical depth buffer is used to perform step-by-step depth intersection detection on each sampled ray. The specific process is as follows: starting from the lower layer of the depth pyramid, it is determined whether there is a depth intersection of the sampled rays. If so, the depth intersection is performed step by step in the upper layer. Otherwise, the tracking of the sampled ray is terminated in advance.

[0015] Furthermore, the indirect lighting result is calculated using the screen-space ray stepping algorithm, and its expression is: in: For pixel position And the direction of the line of sight is The indirect lighting results for the corresponding pixels, For sampling index, For the number of samples, For the first Visibility accumulation of sub-screen space sampling for The color of the hit pixel, Along the sampling direction The hit position of the ray step on the depth map. Sampling direction With line of sight The bidirectional reflection distribution function, For the first Sub-screen space sampling direction To take the sign of the maximum value, For the first The solid angle weights corresponding to the sub-screen space sampling direction.

[0016] The beneficial effects of this invention are as follows: (1) This invention generates a geometric buffer including depth, normal and material properties based on a 2D Gaussian point cloud with complete geometric and material properties. Based on the geometric buffer and ambient light map, the direct lighting result is calculated, the indirect lighting result is calculated using the screen space ray stepping algorithm, and the direct lighting result and indirect lighting result are synthesized to output the real-time relighting result. While ensuring visual quality, the rendering frame rate is increased from less than 0.5 fps in the prior art to a stable higher than 30 fps, which meets the needs of real-time interactive applications. (2) This invention utilizes the normal, depth and roughness information in the geometric buffer to dynamically calculate the curved normal of each pixel, and uses the curved normal of each pixel to sample the pre-baked ambient light map to incorporate ambient light occlusion effect into diffuse reflection calculation, thereby enhancing the three-dimensionality and detail of the scene, and optimizing the light sampling direction to make the lighting in the occluded area more realistic. (3) This invention optimizes the properties of the initial 2D Gaussian point cloud and optimizes the material parameters of the 2D Gaussian point cloud after property optimization in order to obtain a 2D Gaussian point cloud with complete geometric and material properties. This process first accurately reconstructs the scene geometry and then decouples the physically real material properties, providing a high-quality data foundation for subsequent arbitrary relighting. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping. Detailed Implementation

[0018] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0019] like Figure 1 As shown, the real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping includes steps S1-S3, as follows: S1. Acquire multi-view images of the object model and scene, and use the motion reconstruction algorithm to perform initial reconstruction to obtain the initial 2D Gaussian point cloud.

[0020] In an optional embodiment of the present invention, the present invention captures a series of multi-view color images with overlapping areas around the object model and scene, and uses the structure of motion recovery algorithm to estimate the camera parameters (position, pose) from these multi-view images of the object model and scene and generate a three-dimensional point cloud; then the present invention uses the three-dimensional point cloud as an initialization seed to create an initial 2D Gaussian point cloud through the spatial position and color information of the three-dimensional point cloud.

[0021] S2. Optimize the properties of the initial 2D Gaussian point cloud, and optimize the material parameters of the property-optimized 2D Gaussian point cloud to obtain a 2D Gaussian point cloud with complete geometric and material properties.

[0022] In an optional embodiment of the present invention, the present invention performs attribute optimization on the initial 2D Gaussian point cloud, and the expression of the loss function used is as follows: in: This is the loss during the attribute optimization phase. For RGB reconstruction loss, The weighting coefficients for the normal consistency loss are: For loss of normal uniformity, The weighting coefficients for deep concentration loss are... For deep concentration loss, These are the weighting coefficients for the normal smoothing loss. For normal smoothing loss, The weighting coefficients for masking loss, To mask out the loss of the mask.

[0023] Specifically, it is the difference between the rendered depth and the actual depth. To render the difference between the normal direction and the actual normal direction, This is the difference between the rendered color and the actual image color.

[0024] This invention utilizes the loss from the attribute optimization stage to optimize the attributes of an initial 2D Gaussian point cloud. The specific process is as follows: the initial 2D Gaussian point cloud is projected onto the image space through differentiable rendering to generate a predicted RGB image, depth map, and normal map. These are then compared with the real RGB data from the input image and the coarse geometric truth obtained by the structure of motion reconstruction algorithm. By minimizing the loss from the attribute optimization stage, backpropagation is used to automatically optimize the position, covariance matrix, normal vector, color, and opacity of each Gaussian element, thereby driving the geometric structure and appearance details of the initial 2D Gaussian point cloud to be consistent with the real scene, thus obtaining the attribute-optimized 2D Gaussian point cloud.

[0025] This invention optimizes the material parameters of a 2D Gaussian point cloud with optimized properties. The expression for the loss function used is as follows: in: This is due to losses during the material parameter optimization phase. This is the loss during the attribute optimization phase. The weighting coefficients for the PBR consistency loss are: This is due to the consistency loss in PBR (Physically Based Rendering). These are the weighting coefficients for illumination regularization loss. For illumination normalization loss, The weighting coefficients for albedo smoothing loss are... To smooth the albedo loss, These are the weighting coefficients for roughness smoothing loss. For roughness smoothing loss, The weighting coefficients for the metallicity smoothing loss are: This is to smooth out the loss of metallicity.

[0026] This invention utilizes the loss during the material parameter optimization stage to optimize the material parameters of a property-optimized 2D Gaussian point cloud, thereby obtaining a 2D Gaussian point cloud with complete geometric and material properties. Specifically, based on the property-optimized 2D Gaussian point cloud, a rendering result is calculated using a physically based rendering equation, combined with the material parameters to be optimized (reflectivity, roughness, metallicity) and fixed geometric properties, and then compared with the actual input image. By minimizing the loss during the material parameter optimization stage, the backpropagation algorithm is used to specifically optimize the material parameters of each Gaussian element, while freezing its position and covariance matrix, thus obtaining a 2D Gaussian point cloud with complete geometric and material properties.

[0027] S3. Based on a 2D Gaussian point cloud with complete geometric and material properties, generate a geometric buffer including depth, normal and material properties. Calculate direct lighting results based on the geometric buffer and ambient light map, calculate indirect lighting results using a screen space ray stepping algorithm, and synthesize the direct and indirect lighting results to output real-time relighting results.

[0028] In an optional embodiment of the present invention, the present invention generates a geometric buffer including depth, normal, and material properties based on a 2D Gaussian point cloud with complete geometric and material properties. The specific process is as follows: the 2D Gaussian point cloud with complete geometric and material properties is projected and rasterized from a specified camera viewpoint using a 2D Gaussian splashing differentiable rasterizer renderer; during this process, the coverage and opacity contribution of each Gaussian point in screen space are calculated based on the pre-optimized spatial position and covariance properties of each Gaussian point, and the depth buffer and normal buffer are accumulated and synthesized based on this; at the same time, its decoupled material properties (reflectivity, roughness, metallicity) are mixed according to the same rasterization weight, and finally the geometric buffer including depth, normal, and material properties is generated in parallel.

[0029] Specifically, the specified camera viewpoint is any viewing angle freely chosen by the user or application as needed. The same rasterization weight applies to each Gaussian pixel covering the current pixel; it is a value between 0 and 1, determined based on the Gaussian function itself and the opacity. In this invention, when rendering a pixel, the corresponding rasterization weight is calculated for all Gaussian pixels covering that pixel. This calculation process for rasterization weights is identical for depth, normals, and material properties.

[0030] This invention calculates direct lighting results based on a geometric buffer and an ambient light map. Specifically, it dynamically calculates the curved normal of each pixel using the normal, depth, and roughness information in the geometric buffer. This innovation replaces the traditional PBR process of directly sampling the IBL (Image-Based Lighting) irradiance map using the original geometric normal. The curved normal of each pixel is used to sample a pre-baked ambient light map to obtain ambient occlusion-corrected diffuse light color values. These ambient occlusion-corrected diffuse light color values ​​are then combined with physically based rendering equations to synthesize the direct lighting result. This direct lighting result is physically accurate and effectively avoids defects such as light leaks and inaccurate shadows. By calculating direct lighting results in the above manner, this invention significantly improves the realism of relighting while ensuring real-time performance.

[0031] This invention dynamically calculates the curvature normal of each pixel. The specific process is as follows: Based on the depth information in the geometry buffer, each pixel is back-projected onto the world coordinate system. A set of sampling directions is generated within a hemisphere with the normal direction as the center, and the position of the sampling point is obtained. Each sampling direction is projected onto the screen space, and the depth of the corresponding pixel is queried in the geometry buffer, and the position of the query point is obtained. By comparing the relative positions of the sampling point and the query point in the world coordinate system, it is determined whether the sampling direction is occluded. All unoccluded sampling direction vectors are accumulated, and the average vector is calculated to obtain the curvature normal.

[0032] Specifically, because for a point on a surface, illumination can only come from its front side (the side where the normal points), and light from the back side is blocked by the object itself, this invention only samples within a hemisphere centered on the direction of the normal.

[0033] The present invention obtains the position of sampling points through the following process: Based on the depth value in the geometric buffer and the camera intrinsic parameters, the screen coordinates of the current pixel are back-projected to the world coordinate system to obtain its three-dimensional spatial position Pcurrent; then, a set of sampling direction vectors Dsample are generated within the corresponding hemisphere with the normal direction of the point as the axis; finally, each sampling direction is multiplied by a preset sampling radius R and then added to the world coordinates Pcurrent of the current point to calculate the position of each sampling point.

[0034] The present invention obtains the location of the query point through the following process: the location of each sampling point is transformed to the screen space through the camera projection matrix to obtain the corresponding screen coordinates; then, the depth value at the screen coordinates is queried in the depth buffer of the geometry buffer; finally, the screen coordinates and depth value are back-projected back to the world coordinate system using the camera intrinsic parameters and the projection matrix, thereby obtaining the location of the query point.

[0035] This invention determines whether a sampling direction is occluded by comparing the relative positions of a sampling point and a query point in the world coordinate system. Specifically, the process involves calculating the distance from the sampling point to the current pixel and the distance from the query point to the current pixel, and determining whether the distance from the query point to the current pixel is less than the distance from the sampling point to the current pixel. If so, the sampling direction is determined to be occluded; otherwise, the sampling direction is determined not to be occluded.

[0036] The sampling direction that is not occluded takes a value of 1, and the sampling direction that is occluded takes a value of 0.

[0037] This invention employs a screen-space ray stepping algorithm to calculate indirect lighting results. The specific process is as follows: For a pixel to be rendered in screen space, multiple sampling rays are generated based on its normal and roughness information; a hierarchical depth buffer is used to perform stepping and depth intersection detection on each sampling ray. If a sampling ray intersects with a pixel in screen space, the direct lighting result of the intersecting pixel is included in the indirect lighting contribution; the indirect lighting contributions of all sampling rays are weighted and summed to obtain the indirect lighting result of the pixel to be rendered.

[0038] The method for constructing a hierarchical depth buffer is as follows: taking the original depth in the geometric buffer as the first level, downsampling layer by layer, and taking the maximum depth of the corresponding 2x2 pixel region of the lower layer for each upper pixel to form a depth pyramid, thereby constructing a hierarchical depth buffer.

[0039] This invention utilizes a hierarchical depth buffer to perform step-by-step depth intersection detection on each sampled ray. The specific process is as follows: starting from the lower layer of the depth pyramid, it is determined whether there is a depth intersection between the sampled rays. If so, the ray is gradually moved to the upper layer for depth intersection detection; otherwise, the tracking of the sampled ray is terminated in advance.

[0040] This invention employs a screen-space ray stepping algorithm to calculate indirect lighting results, the expression of which is: in: For pixel position And the direction of the line of sight is The indirect lighting results for the corresponding pixels, For sampling index, For the number of samples, For the first Visibility accumulation of sub-screen space sampling for The color of the hit pixel, Along the sampling direction The hit position of the ray step on the depth map. Sampling direction With line of sight The bidirectional reflection distribution function, For the first Sub-screen space sampling direction To take the sign of the maximum value, For the first The solid angle weights corresponding to the sub-screen space sampling direction.

[0041] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping, characterized in that, Includes the following steps: Collect multi-view images of the object model and scene, and use the structure-reconstruction-motion algorithm to perform initial reconstruction to obtain an initial 2D Gaussian point cloud; The properties of the initial 2D Gaussian point cloud are optimized, and the material parameters of the property-optimized 2D Gaussian point cloud are optimized to obtain a 2D Gaussian point cloud with complete geometric and material properties. Based on a 2D Gaussian point cloud with complete geometric and material properties, a geometric buffer including depth, normals, and material properties is generated. Direct lighting results are calculated based on the geometric buffer and ambient light map, and indirect lighting results are calculated using a screen-space ray stepping algorithm. The direct and indirect lighting results are then synthesized to output real-time relighting results.

2. The real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping according to claim 1, characterized in that, The initial 2D Gaussian point cloud is optimized for its properties using the following loss function: in: This is the loss during the attribute optimization phase. For RGB reconstruction loss, The weighting coefficients for the normal consistency loss are: For loss of normal uniformity, The weighting coefficients for deep concentration loss are... For deep concentration of losses, These are the weighting coefficients for the normal smoothing loss. For normal smoothing loss, The weighting coefficients for masking loss, To mask out the loss of the mask.

3. The real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping according to claim 2, characterized in that, The material parameters of the attribute-optimized 2D Gaussian point cloud are then optimized using the following loss function: in: This is due to losses during the material parameter optimization phase. This is the loss during the attribute optimization phase. The weighting coefficients for the PBR consistency loss are: For PBR consistency loss, These are the weighting coefficients for illumination regularization loss. For illumination normalization loss, The weighting coefficients for albedo smoothing loss are... To smooth the albedo loss, These are the weighting coefficients for roughness smoothing loss. For roughness smoothing loss, The weighting coefficient for the metallicity smoothing loss is... This is to smooth out the loss of metallicity.

4. The real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping according to claim 1, characterized in that, The direct lighting results are calculated based on the geometry buffer and environment lightmap. The specific process is as follows: By utilizing the normal, depth, and roughness information in the geometry buffer, the bending normal of each pixel is dynamically calculated; The pre-baked ambient light map is sampled using the curved normal of each pixel to obtain ambient occlusion corrected ambient diffuse color values, and then the direct lighting result is synthesized using the ambient occlusion corrected ambient diffuse color values ​​and physically based rendering equations.

5. The real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping according to claim 4, characterized in that, The bending normal of each pixel is dynamically calculated. The specific process is as follows: Based on the depth information in the geometric buffer, each pixel is back-projected to the world coordinate system. A set of sampling directions is generated within the hemisphere with the normal direction as the center, and the position of the sampling point is obtained. Project each sampling direction onto the screen space, query the depth of the corresponding pixel in the geometry buffer, and obtain the position of the query point; By comparing the relative positions of the sampling point and the query point in the world coordinate system, it can be determined whether the sampling direction is occluded. Accumulate all unoccluded sampling direction vectors and calculate the average vector to obtain the curvature normal.

6. The real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping according to claim 1, characterized in that, The indirect lighting results are calculated using the screen-space ray stepping algorithm. The specific process is as follows: For the pixel to be rendered in screen space, multiple sampling rays are generated based on its normal and roughness information; A hierarchical depth buffer is used to perform step and depth intersection detection on each sampling ray. If the sampling ray intersects with a pixel in the screen space, the direct illumination result of the intersecting pixel is included in the indirect illumination contribution. The indirect lighting contributions of all sampled rays are weighted and summed to obtain the indirect lighting results for the pixel to be rendered.

7. The real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping according to claim 6, characterized in that, The method for constructing a hierarchical depth buffer is as follows: taking the original depth in the geometric buffer as the first level, downsampling layer by layer, and taking the maximum depth of the corresponding 2x2 pixel region of the lower layer for each upper pixel to form a depth pyramid, thereby constructing a hierarchical depth buffer.

8. The real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping according to claim 7, characterized in that, A hierarchical depth buffer is used to perform step-by-step depth intersection detection on each sampled ray. The specific process is as follows: starting from the lower layer of the depth pyramid, it is determined whether there is a depth intersection of the sampled rays. If so, the depth intersection is performed step by step in the upper layer. Otherwise, the tracking of the sampled ray is terminated in advance.

9. The real-time relighting method based on 2D Gaussian splashing and screen-space ray stepping according to claim 1, characterized in that, The indirect lighting result is calculated using the screen-space ray stepping algorithm, and its expression is as follows: in: For pixel position And the direction of the line of sight is The indirect lighting result of the corresponding pixel, For sampling index, For the number of samples, For the first Visibility accumulation of sub-screen space sampling for The color of the hit pixel, For sampling direction The hit position of the ray step on the depth map. Sampling direction With line of sight The bidirectional reflection distribution function, For the first Sub-screen space sampling direction To determine the sign of the maximum value, For the first The solid angle weights corresponding to the sub-screen space sampling direction.

Images