Novel expression, drawing and reconstruction method of radiation field

Abstract

The invention discloses a novel representing, drawing and reconstructing method of a radiation field. According to the method, Gaussian enhancement surface elements are used as representation of a radiation field, and for a given view angle, rasterization is firstly carried out on opaque surface elements through a standard graph pipeline to generate color and depth maps; and then projecting the three-dimensional gauss to a screen space and executing a depth test, and weighting and accumulating gauss colors on each pixel of a color image rendered by the surface element according to the opacity of the gauss to obtain a final rendering result. The invention further provides an optimization process from coarse granularity to fine granularity. Compared with the prior art, the method has the advantages that high-quality new-view-angle images and videos meeting visual consistency can be synthesized at an extremely high rendering speed. In addition, the novel representation provided by the invention is combined with the existing method, so that anti-aliasing rendering can be realized, the rendering speed is further improved, the storage overhead is reduced, and more accurate scene geometry can be reconstructed by replacing three-dimensional gauss with two-dimensional gauss.

Description

Technical Field

[0001] The present invention relates to the technical field of new perspective synthesis, and in particular to a novel representation, rendering and reconstruction method of a radiation field. Background Art

[0002] New view synthesis has a long history in computer graphics and vision. It primarily utilizes multi-view images of static scenes to synthesize images from arbitrary new viewpoints. Traditional scene reconstruction and rendering methods typically construct light fields from densely sampled images to perform new view synthesis. With the development of Structure from Motion (SfM) and Multi-View Stereo (MVS) techniques, complete 3D reconstruction from photo collections has become possible. Numerous view synthesis methods based on these two approaches have been proposed. Among these methods, surface light fields (SLFs) are the most relevant representation for this study. Surface light fields represent the radiance emitted from any point on a surface and can capture radiance information from different directions. Constructing surface light fields requires a high-resolution geometric model and densely sampled images. Our method, however, uses bins with view-dependent color to approximate the surface light field, using a 3D Gaussian to supplement scene details that the bins fail to fully capture. Unlike SLFs, our method does not require a high-resolution geometric model.

[0003] Neural Radiance Field (NeRF) is an emerging class of neural rendering methods that has been widely studied for novel perspective synthesis. NeRF generates high-quality view synthesis effects by learning a volumetric representation of the scene, and has spawned a large number of subsequent studies. For example, Instant-NGP (Instant neural graphics primitives with a multiresolution hash encoding. ToG 2022) accelerates the training and rendering process of NeRF by replacing deep MLP with multi-resolution hash encoding, and training can be completed in minutes. In addition, NeRF and its variants, such as Mip-NeRF (Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR 2022) and MegaNeRF (Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs. CVPR 2022), have made significant progress in rendering speed and quality.

[0004] Point-based rendering technology has also been studied for decades, mainly used to render disconnected and unstructured point clouds. To improve rendering quality, a common technique is to render points into larger primitive shapes, such as disks, ellipses, or surfaces. With the development of differentiable point-based rendering, point-based rendering technology has also been applied to new perspective synthesis. Among these technologies, Pulsar (Pulsar: Efficient Sphere-based Neural Rendering, CVPR 2021) uses a deferred neural rendering method to generate screen space feature maps from point clouds and convert them into color images through a pixel generation network. However, such methods usually require a long training time, and temporal instability may occur when using CNN for rendering.

[0005] 3D Gaussian Splatting for Real-Time Radiance Field Rendering (3DGS, ToG 2023) represents the radiation field as sparsely distributed 3D Gaussians and renders them with view-dependent colors. Due to its fast rendering speed and high image quality, 3DGS has inspired a large amount of follow-up research. For example, 2D Gaussian Splatting for Geometrically Accurate Radiance Fields (2DGS, ToG2024) introduces a perspective-accurate 2D Gaussian splatting process and combines depth and normal consistency to optimize the 2D Gaussians so that they are more accurately distributed around the surface, thereby obtaining a more geometrically accurate radiation field. However, 2DGS is still a volumetric representation method, and all Gaussians still need to be depth-sorted and alpha-blended when rendering.

[0006] However, the 3DGS method suffers from the problem of "jumping" artifacts caused by changes in view angle. This is caused by its approximation of the exact depth order of each pixel through global depth sorting. To address this problem, StopThePop (Stopthepop: Sorted Gaussian splatting for view-consistent real-time rendering, ToG 2024) proposed a hierarchical sorting strategy to reduce the impact of this artifact through global pre-sorting and local sorting. However, this strategy still cannot guarantee that the order of each pixel is completely correct, and "jumping" artifacts may still appear in the synthesis of new perspectives.

[0007] Unlike 3DGS and subsequent research that use Gaussians to represent volumetric radiation fields, this method uses two-dimensional bins with view-dependent color to construct coarse-grained surface radiation fields with respect to geometry and appearance, and enhances the detail of these fields with three-dimensional Gaussians. The entire rendering process is completely independent of sorting, resulting in extremely fast rendering speeds and visually consistent video results without "jumping" artifacts. Summary of the Invention

[0008] To address the shortcomings of existing methods, this paper designs a novel method for representing, rendering, and reconstructing radiation fields. This method uses multi-view images of a static scene to learn a representation of the surface radiation field based on two-dimensional facets and three-dimensional Gaussians. Given a new perspective, this method can render visually consistent images or videos from that perspective at high speed and high quality. Furthermore, this paper provides an extension method based on this novel representation, enabling anti-aliasing, higher-speed rendering with low storage overhead, while also reconstructing a relatively accurate geometric representation of the surface mesh.

[0009] The specific technical solutions of the present invention are as follows:

[0010] The present invention provides a novel method for representing, rendering and reconstructing a radiation field, comprising the following steps:

[0011] (1) Radiation field representation: Gaussian enhanced surface elements are used as the representation of the radiation field. At coarse granularity, a set of two-dimensional opaque elliptical surface elements with view-dependent colors constitutes an approximate representation of the scene surface radiation field. Each two-dimensional surface element is a unit disk defined in the local coordinate system and located on the XY plane. It has four properties: three-dimensional position, rotation, scale, and spherical harmonic coefficients. At fine granularity, a volumetric radiation field is formed by the three-dimensional Gaussian surrounding the opaque surface element to supplement the scene appearance information that the surface radiation field cannot well represent. Each three-dimensional Gaussian has five properties: position, rotation, scale, opacity, and spherical harmonic coefficients.

[0012] (2) Radiation field rendering: For a given viewing angle, the radiation field rendering of the Gaussian enhanced surface element in step (1) is divided into two steps, specifically: in the first step, the opaque surface element is rasterized through the standard graphics pipeline. The opaque surface element will be transformed into the world coordinate system through scaling, rotation and translation in turn. The surface element fragments that pass the depth test will write their colors to the color buffer. Supersampling is enabled in the rendering of the surface element. After the rendering is completed, the surface element color map and depth map will be obtained, thereby realizing anti-aliasing rendering; in the second step, the three-dimensional Gaussian is projected into the screen space, and the color and weight of the Gaussian are accumulated for each pixel; during the accumulation process, the depth map of the surface element rendering is used to perform depth testing on the Gaussian, and those Gaussians whose center depth fails the depth test and is blocked by the surface are ignored; the final rendering result is calculated by the weighted combination of the color map of the surface element rendering and the color map of the Gaussian accumulation;

[0013] (3) Radiation field reconstruction: For a given set of multi-view images with known camera parameters, a coarse-grained to fine-grained method is used for optimization. In the coarse-grained optimization stage, opaque facets are optimized to reconstruct the coarse-grained geometric structure and appearance. In the fine-grained optimization stage, the geometric properties of the facets are fixed and a three-dimensional Gaussian is added. The geometric properties of the facets include position, rotation and scale. The three-dimensional Gaussian is then jointly optimized with the spherical harmonic coefficient properties of the facets to reconstruct the radiation field details.

[0014] Specifically, the fine-grained optimization stage in step (3) optimizes the loss function of the entire process including the distance between images.

[0015] The weighted sum of L1 and structural similarity loss D-SSIM is expressed as follows:

[0016] L rgb =(1-λ)L1+λL D-SSIM ;

[0017] Where λ is the weight of the energy term, which ranges from 0 to 1.

[0018] Furthermore, the step (3) includes the following sub-steps:

[0019] (3.1) In the facet optimization stage, the facet is initialized by the point cloud obtained by motion structure recovery (SfM), where the position and color of the facet are set to the position and color of the point cloud, the scaling value of the facet is set to the distance to its nearest neighbor, and the rotation is randomly generated; since opaque faces are not differentiable, an opacity control parameter w is introduced. During the optimization process, by increasing w, the semi-transparent facets that rely on sorting and are rendered based on alpha blending are gradually converted to opaque facets that are rendered based on order-independent depth buffer; thus, a set of coarse-grained opaque facets that express the geometry and appearance of the scene are finally obtained; and after w>30, quadruple supersampling rendering is enabled to achieve anti-aliasing rendering; when w>30, after optimizing the block-based sorting, for each pixel, the precise pixel-level depth of the facet covering the pixel is calculated; the facet with the smallest depth is selected for the first blending calculation, and the blending order of other facets is not adjusted; and the maximum pixel coverage of the facet is used as the basis for removing redundant facets during training;

[0020] (3.2) In the joint optimization stage of Gaussians and facets, the parameters of the 3D Gaussians are jointly optimized together with the spherical harmonic coefficients of the facets. During the joint optimization process, the 3D Gaussians need to be periodically added and deleted. For each specified number of iterations, the squared error map between the rendered image and the real image at all training viewpoints is calculated, and the error of each pixel is normalized. The normalized error is used as the sampling probability of the pixel. A fixed number of pixels are then selected from each image based on the sampling probability, and a point cloud is generated using the corresponding facet depth map and camera parameters. The point cloud is used as the initial position of the newly added 3D Gaussians. The properties of the remaining Gaussians are initialized using the same method as the 3D Gaussian splashing. The maximum contribution value of the Gaussian to the pixel at different viewpoints is calculated, and Gaussians with a maximum contribution value less than 0.02 are eliminated.

[0021] Specifically, the semi-transparent surface element in step (3.1) is also a disk defined on the XY plane of the local coordinate system, and its opacity α i The value at coordinate (x,y) is defined as:

[0022] α i (x,y)=min(1,w i G(x,y)),

[0023]

[0024] where α i (x,y) will be truncated to 0 when min(α i (x,y),G(x,y))<1 / 255; opacity control parameter w i ∈[0,255] is used to establish the conversion relationship between semi-transparent surface elements and opaque surface elements; when w i <1, the surface elements show a two-dimensional Gaussian distribution; as w i As w increases, the Gaussian distribution gradually becomes opaque from the center outward; i When it reaches 255, the opacity is 1 within the range of G(x,y)>1 / 255, and it is 0 outside this range, thus forming an opaque disk; this mechanism ensures that the surface element can smoothly transition from a semi-transparent state to a stable opaque state during the optimization process.

[0025] Furthermore, the maximum pixel coverage of the surface element in step (3.1) is used as a basis to delete redundant surface elements during the training process, specifically: for each surface element S in the training view j i , calculate S i The number of pixels n that are the frontmost covered pixels in the bin i,j The front is defined as the minimum depth, and the coverage score of the surface element is defined as Where T is the total number of training views; remove all i <n thr Surface elements to remove redundant surface elements; for real scenes, set the threshold n thr =16.

[0026] Furthermore, the step (3.2) is to calculate the maximum contribution value of Gaussian to the pixel at different viewing angles, and remove Gaussian with a maximum contribution value less than 0.02, specifically: by calculating each Gaussian G i Contribution score s in the jth training view i,j The calculation formula is as follows:

[0027]

[0028] in is the pixel covered by the Gaussian, is the opacity at the jth training view, and the color contribution c i,j Take the maximum value of the three-channel color value, that is If the maximum contribution of a Gaussian in all T training views If it is less than 0.02, the Gaussian will be eliminated.

[0029] Furthermore, it also includes the optimization of anti-aliasing rendering. Specifically, when drawing the radiation field, the radiation field representation is combined with the Gaussian sputtering improvement method. By applying the world space filter and the approximate screen space box filter to the three-dimensional Gaussian, and keeping the surface element using supersampling rendering, the best anti-aliasing effect rendered at different scales is obtained.

[0030] Furthermore, it also includes accelerated rendering. Specifically, in the fine-grained optimization stage of radiation field reconstruction, the method of removing Gaussians based on the maximum pixel contribution value can be replaced by a removal strategy based on the Hessian score, which is used to further remove Gaussians and obtain the best rendering frame rate under certain rendering quality conditions.

[0031] Furthermore, it also includes optimization of compressed storage, specifically: by querying the spherical harmonic coefficients of color, replacement pixels and Gaussian from the hash grid, and quantizing the scaling and rotation parameters of pixels and Gaussian using residual vector quantization, more than 20 times of storage compression is achieved.

[0032] Furthermore, it also includes the optimization of geometric reconstruction, specifically: when rendering opaque surface elements, in addition to obtaining color and depth maps, a normal map can also be obtained according to the orientation of the surface element; by replacing the three-dimensional Gaussian in step (1) with a two-dimensional Gaussian, the method of rendering color in step (2) can be used to accumulate Gaussian normals by weight on the basis of the surface element normal map, thereby obtaining a final smoother normal map; replacing the normal with depth and rendering in the same way can obtain a final smoothed depth map; using the truncated signed distance function method TSDF to reconstruct the object surface mesh from the rendered multi-view depth map; in order to optimize and obtain accurate geometry, in addition to using the loss function of the optimization process in step (3), a loss function of mutual supervision between depth and normal is added, requiring the normal map derived from the depth map to be the same as the normal map obtained by rendering. Figure 1 At the same time, in the facet optimization stage, a method is added to supervise the optimization of the depth and normal maps in the later stage by optimizing the normal map of the early rendering.

[0033] The beneficial effects of the present invention are as follows:

[0034] After the present invention uses multi-view images of static scenes to train the three-dimensional radiation field, for a given one or a group of continuous new perspectives, it can render high-fidelity, visually consistent (without "jumping" artifacts) images or videos at an extremely high real-time frame rate (reaching over 600 frames per second at 1080p resolution). An extended method based on this method can achieve higher-quality anti-aliasing rendering, high-speed rendering of over 1000 frames per second, compression of space overhead by about 20 times, and more accurate object geometry reconstruction. The training time of this method is usually less than one hour, and it still supports real-time rendering at 2160p, i.e. 4K resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the novel radiation field representation and rendering process of the present invention;

[0036] Figure 2 This is an example diagram of the surface element rendering, Gaussian rendering and complete rendering results obtained by the present invention on an outdoor scene;

[0037] Figure 3 Schematic diagram of the rendering results during the optimization process of the present invention as the number of iterations changes. DETAILED DESCRIPTION

[0038] The present invention discloses a novel representation, rendering and reconstruction method of radiation field, the core of which lies in a novel representation of radiation field, which can render high-fidelity images at extremely high frame rates for a given new perspective.

[0039] like Figure 1As shown, the radiation field representation of the present invention is a representation using three-dimensional Gaussian enhanced two-dimensional surface elements.

[0040] At coarse granularity, a set of two-dimensional opaque elliptical surface elements with view-dependent colors constitutes an approximate representation of the scene surface radiation field. Each two-dimensional surface element is a unit disk defined in the local coordinate system and located on the XY plane, with position p i , rotate r i , Zoom i and spherical harmonic color coefficient SH i Four attributes. The color of the entire pixel c i and from the center of the surface element p i The spherical harmonic colors pointing in the direction of the camera position o are consistent, that is, they satisfy: c i =Y(∥op i ∥,SH i ), where Y(·) represents the spherical harmonics.

[0041] At fine granularity, some 3D Gaussians around the 2D surface element form a volume radiation field to supplement the scene appearance information that the surface radiation field fails to represent well. Each 3D Gaussian has a position p i , rotate r i , Zoom i , opacity σ i With spherical harmonic color SH i The shape of a 3D Gaussian is controlled by rotation and scale, and its definition is exactly the same as that of a 3D Gaussian splatter (3DGS).

[0042] like Figure 1 As shown, the radiation field rendering of the present invention includes two steps:

[0043] The first step is to rasterize the opaque pixels through the standard graphics pipeline. The opaque pixels will be transformed to the world coordinate system through scaling, rotation and translation. The pixel fragments that pass the depth test will have their colors written to the color buffer. In order to achieve anti-aliasing rendering, 4x multisampling is enabled in the pixel rendering. After the rendering is completed, the pixel color map C will be obtained. S With the depth map D S .

[0044] In the second step, the 3D Gaussian is projected into screen space using the same method as 3D Gaussian splattering (3DGS), and the color and weight of the Gaussian are accumulated for each pixel. During the accumulation process, the depth map of the surface rendering is used to perform depth testing on the Gaussian, and those Gaussians whose center depth fails the depth test, that is, are occluded by the surface, are ignored. The final rendering result is calculated by a weighted combination of the color map of the surface rendering and the color map of the Gaussian accumulation. Pixel Gaussian cumulative color C at GWith cumulative weight W G The calculation method is as follows:

[0045]

[0046] in is the indicator function, d i is the Gaussian center depth, d s is the depth of the pixel position corresponding to the face depth map, Σ is the Gaussian covariance matrix obtained by the rotation and scaling parameters of the three-dimensional Gaussian, is the pixel position corresponding to the Gaussian center in screen space, α i is the weighted weight, σ i is the maximum weight of the Gaussian center, K is the number of pixels covered The number of all Gaussians in ∈ is five times the average of the major and minor axes of the facet, which is a small value to avoid truncation of Gaussians near the facet. The final rendering result C is calculated by weighted combination of the facet rendering color map and the Gaussian accumulation color map:

[0047]

[0048] Where W s is the weight of the face color, which has a fixed value of 1. Figure 2 The figure shows the surface element rendering results, Gaussian rendering results and complete rendering results, showing the contribution of surface elements and Gaussian to the final result in coarse and fine granularity.

[0049] The optimization process of this invention is a coarse-grained to fine-grained optimization process. Given a set of multi-view images of a static scene, and the corresponding cameras have been calibrated using the Structure from Motion (SfM) method, the goal of this invention is to optimize a set of 2D bins and 3D Gaussians to achieve high-fidelity free-view synthesis.

[0050] The present invention adopts a coarse-to-fine optimization strategy. In the coarse-grained optimization stage, the coarse-grained geometric structure and appearance information of the scene are reconstructed by optimizing the opaque facets. In the fine-grained optimization stage, the geometric properties of the surface elements (including position, rotation and scale) are fixed, and on this basis, a three-dimensional Gaussian is introduced and jointly optimized with the spherical harmonic coefficients of the surface elements to further enhance the detailed appearance of the scene. The loss function of the entire optimization process includes the weighted sum of the L1 distance between images and the structural similarity loss D-SSIM: L rgb =(1-λ)L1+λL D-SSIM , where λ = 0.2. Figure 3 The rendering results at different optimization iterations are shown, where the first 20,000 iterations are for surfel optimization and the last 30,000 iterations are for surfel-Gaussian joint optimization.

[0051] The first stage is the facet optimization. The facets are initialized by the point cloud obtained by motion structure recovery (SfM). The position and color of the facet are set to the position and color of the SfM point, the scaling value of the facet is set to the distance to the nearest neighbor, and the rotation is randomly generated. In order to solve the problem of non-differentiability of opaque facets, in this stage, semi-transparent facets are optimized and the opacity control parameter w is used. During the optimization process, by increasing w, the semi-transparent facets that rely on sorting and are based on alpha blending rendering are gradually converted into opaque facets that are order-independent and based on depth buffer rendering, thus finally obtaining a set of coarse-grained opaque facets that express the scene geometry and appearance. The semi-transparent facet is also a disk defined on the XY plane of the local coordinate system, and its opacity α i The value at (x,y) is defined as:

[0052] α i (x,y)=min(1,w i G(x,y)),

[0053]

[0054] where α i (x,y) will be truncated to 0 when min(α i (x,y),G(x,y))<1 / 255. Opacity control parameter w i ∈[0,255] is used to establish the conversion relationship between semi-transparent surface elements and opaque surface elements. i <1, the surface elements show a two-dimensional Gaussian distribution. i As w increases, the Gaussian distribution becomes increasingly opaque from the center outward. i When it reaches 255, the opacity is 1 within the range of G(x,y)>1 / 255, and it is 0 outside this range, thus forming an opaque disk. This mechanism ensures that the surface element can smoothly transition from a semi-transparent state to a stable opaque state during the optimization process, thereby improving the accuracy of scene representation and rendering quality. In order to obtain more effective gradient information at the edge of the surface element and achieve anti-aliasing rendering of the surface element, w i >30 will enable 4x super sampling rendering; in order to improve the accuracy of block sorting in the optimization phase, when w i When θ>30, after block-based sorting, for each pixel, the exact pixel-level depth of the bins covering the pixel is calculated. The bin with the smallest depth is selected for the first blending calculation, and the blending order of the other bins is not adjusted. In the first 10,000 iterations of the optimization, the present invention uses the same strategy as the two-dimensional Gaussian splash (2DGS) method to add and delete semi-transparent bins. At the 10,000th iteration, all opacity parameters w are discarded. i<0.8 surface elements (but record the positions of these surface elements for the initialization of the subsequent three-dimensional Gaussian), and at the same time, w of the remaining surface elements i Increase to no less than 30, and stop adding and deleting operations. At the 18000th and 19000th iterations, further increase the w of all surface elements i are raised to no less than 60 and 90 respectively. At the 20000th iteration, the w of all surface elements is i Set to 255, thus completing the optimization of surface elements. After the 15000th iteration, the present invention adopts a new strategy to further delete surface elements. Specifically, for each surface element S in the training view j i , calculate S i The number of pixels n that are the frontmost (with minimum depth) covered pixels in the bin i,j The coverage score of the facet is defined as Where T is the total number of training views. Next, remove all i <n thr Surface elements of the image are used to remove extremely small or almost invisible surface elements. For real scenes, set the threshold n thr =16.

[0055] The second stage is the joint optimization of Gaussian and facets. At the 10000th iteration, the facets that were discarded during the optimization process and that meet w i Surfactant positions with a value <0.8 are used to initialize the 3D Gaussian. During the joint optimization process, this method will periodically add and delete 3D Gaussians. Every 1000 iterations, the squared error map between the rendered image and the real image under all training viewpoints is calculated, and the error of each pixel is normalized. The normalized error is used as the sampling probability of the pixel. Then, a fixed number of pixels are selected from each image according to the sampling probability, and a point cloud is generated using the corresponding surfel depth map and camera parameters. The point cloud is used as the initial position of the newly added 3D Gaussian. The properties of the remaining Gaussians are initialized using the same method as 3D Gaussian splashing (3DGS). For the deletion of 3D Gaussians, by calculating each Gaussian G i Contribution score s in the jth training view i,j , that is, the maximum color contribution of Gaussian to the rendering result is achieved, and its calculation formula is as follows:

[0056]

[0057] in is the pixel covered by the Gaussian, is the opacity at the jth training view, and the color contribution c i,j Take the maximum value of the three-channel color value, that is If the maximum contribution of a Gaussian in all T training views If it is less than 0.02, the Gaussian will be culled to remove invisible or low-contributing Gaussians.

[0058] The radiation field representation proposed in this invention can be combined with existing technologies to form a variety of extended methods, including:

[0059] Anti-aliased rendering: This method applies the MipSplat (Mip-Splatting: Alias-free 3D Gaussian Splatting, CVPR 2024) world space filter and an approximate screen space box filter to the 3D Gaussian, and keeps the pixels rendered with 4x multisampling, which improves the anti-aliasing effect of this method at different scales.

[0060] Accelerated rendering: The method of removing Gaussians based on the maximum pixel contribution value in the surface-Gaussian joint optimization process can be replaced with the Hessian score-based removal strategy in SpeedySplat (Speedy-Splat: Fast 3D Gaussian Splatting with SparsePixels and Sparse Primitives) to further remove a large number of Gaussians, in exchange for a higher rendering frame rate at the expense of slightly sacrificing rendering quality.

[0061] Compressed Storage: By querying the color of the facets and Gaussians from a hash grid, instead computing the color from the spherical harmonic coefficients of each Gaussian, and quantizing the scale and rotation parameters of the facets and Gaussians using residual vector quantization, more than 20x storage compression can be achieved.

[0062] Geometry reconstruction: When rendering opaque surfaces, in addition to obtaining color and depth maps, you can also obtain a normal map based on the surface orientation. By replacing the 3D Gaussian with a 2D Gaussian, you can calculate the smoothed normal and depth maps as follows:

[0063]

[0064] Among them D s Depth map for pixel rendering, N s Normal map of the rendered surface element, d i With n iThe depth and normal of the pixel position covered by the two-dimensional Gaussian correspond to the two-dimensional Gaussian plane. The normal of each face element is the normal of the plane on which it is located, and the depth of any point on the face element is the depth of the corresponding plane. The Truncated Signed Distance Function (TSDF) method can be used to reconstruct the surface mesh of the object from the rendered multi-view depth map. In order to optimize the accurate normal and depth, after replacing the three-dimensional Gaussian with a two-dimensional Gaussian, the loss function of the face element Gaussian joint optimization stage is modified to L = λ n L n +λ gd L gd +L rgb , where λ n =0.05,λ gd =0.1, where L n To ensure that the normal derived from the depth is consistent with the rendered normal, the loss function is calculated in the same way as described in the two-dimensional Gaussian splattering (2DGS) method. The loss function in the facet optimization phase is modified to L = λ n L n +L rgb (When the number of iterations is less than 10,000) and L = λ sn L sn +λ sd L sd +L rgb (When the number of iterations is greater than 10000), where λ n =λ sn =λ sd =0.05, is the operator that calculates the normal map from the depth map, N supv is the normal map rendered at the 10,000th iteration. During the surface optimization process, since there are still semi-transparent surfaces, the rendering of the depth map D and the normal map N needs to be obtained through Alpha blending, and the calculation method is as follows:

[0065]

[0066] In order to achieve two-dimensional Gaussian anti-aliasing rendering, this method uses EWA (Elliptically Weighted Average) filtering in the local coordinate system to improve the rendering quality.

[0067] Example

[0068] The inventors implemented an example implementation of the present invention on a desktop computer equipped with an i7-13700KF CPU, 32GB of RAM, and an NVIDIA RTX 4090 graphics processor. For a real-life dataset, the average training time was 43 minutes. On a novel perspective synthesis task, the method achieved a rendering frame rate of 675 frames per second at a 1080p rendering resolution.

[0069] The inventors tested this method on different data sets. It was verified that this method can achieve high-quality rendering results in new perspective image synthesis tasks in various indoor and outdoor scenes. Compared with previous methods, this method not only has advantages in rendering quality and speed, but also can achieve continuous video generation with visual consistency. At the same time, for the extended method of radiation field expression proposed by the present invention: the anti-aliasing rendering method can achieve higher rendering quality; the accelerated rendering method can achieve rendering of more than 1000 frames per second with slightly reduced rendering quality; the compressed storage method can reduce the storage overhead of saving the radiation field to 47MB; the geometric reconstruction method can extract high-quality surface meshes from multi-perspective images.

[0070] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed in this application.

[0071] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A novel method for representing, rendering and reconstructing radiation fields, characterized in that: The following steps are involved: (1) Radiation field representation: Gaussian enhanced surface elements are used as the representation of the radiation field. At coarse granularity, a set of two-dimensional opaque elliptical surface elements with view-dependent colors constitutes an approximate representation of the scene surface radiation field. Each two-dimensional surface element is a unit disk defined in the local coordinate system and located on the XY plane. It has four properties: three-dimensional position, rotation, scale, and spherical harmonic coefficients. At fine granularity, a volumetric radiation field is formed by the three-dimensional Gaussian surrounding the opaque surface element to supplement the scene appearance information that the surface radiation field cannot well represent. Each three-dimensional Gaussian has five properties: position, rotation, scale, opacity, and spherical harmonic coefficients. (2) Radiation field rendering: For a given viewing angle, the radiation field rendering of the Gaussian enhanced surface element in step (1) is divided into two steps, specifically: in the first step, the opaque surface element is rasterized through the standard graphics pipeline. The opaque surface element will be transformed into the world coordinate system through scaling, rotation and translation in turn. The surface element fragments that pass the depth test will write their colors to the color buffer. Supersampling is enabled in the rendering of the surface element. After the rendering is completed, the surface element color map and depth map will be obtained, thereby realizing anti-aliasing rendering; in the second step, the three-dimensional Gaussian is projected into the screen space, and the color and weight of the Gaussian are accumulated for each pixel; during the accumulation process, the depth map of the surface element rendering is used to perform depth testing on the Gaussian, and those Gaussians whose center depth fails the depth test and is blocked by the surface are ignored; the final rendering result is calculated by the weighted combination of the color map of the surface element rendering and the color map of the Gaussian accumulation; (3) Radiance field reconstruction: For a given set of multi-view images with known camera parameters, a coarse-grained to fine-grained approach is used for optimization; In the coarse-grained optimization stage, opaque surfaces are optimized to reconstruct the coarse-grained geometric structure and appearance; In the fine-grained optimization stage, the geometric properties of the surface element are fixed and a three-dimensional Gaussian is added. The geometric properties of the surface element include position, rotation and scale. It is jointly optimized with the spherical harmonic coefficient properties of the surface element to reconstruct the radiation field details.

2. A novel method for representing, rendering and reconstructing radiation fields according to claim 1, characterized in that: In the fine-grained optimization stage in step (3), the loss function of the entire optimization process includes the weighted sum of the distance L1 between images and the structural similarity loss D-SSIM, which is expressed as follows: L rgb =(1-λ)L1+λL D-SSIM ; Where λ is the weight of the energy term, which ranges from 0 to 1.

3. A novel method for representing, rendering and reconstructing radiation fields according to claim 1, characterized in that: The step (3) includes the following sub-steps: (3.1) In the facet optimization stage, the facets are initialized from the point cloud obtained by structure-from-motion (SfM). The facet position and color are set to the point cloud position and color, the facet scale is set to the distance to its nearest neighbor, and the rotation is randomly generated. Since opaque faces are not differentiable, an opacity control parameter w is introduced. During the optimization process, by increasing w, the semi-transparent faces that rely on sorting and are rendered based on alpha blending are gradually converted to opaque faces that are rendered based on the depth buffer and are order-independent. This ultimately results in a set of coarse-grained opaque faces that represent the scene geometry and appearance. When w>30, quadruple supersampling rendering is enabled to achieve anti-aliasing rendering. When w>30, after optimizing the block-based sorting, for each pixel, the precise pixel-level depth of the surface covering the pixel is calculated. The surface with the smallest depth is selected for the first blending calculation, while the blending order of other surfaces is not adjusted. The maximum pixel coverage of the surface is used as the basis to delete redundant surfaces during training. (3.2) In the joint optimization stage of Gaussians and surface elements, the parameters of the three-dimensional Gaussians and the spherical harmonic coefficients of the surface elements are jointly optimized. During the joint optimization process, the three-dimensional Gaussians need to be periodically added and deleted. After a specified number of iterations, the squared error map between the rendered image and the real image at all training viewpoints is calculated, and the error of each pixel is normalized. The normalized error is used as the sampling probability of the pixel. Then, a fixed number of pixels are selected from each image according to the sampling probability, and a point cloud is generated using the corresponding surface depth map and camera parameters. This point cloud is used as the initial position of the newly added 3D Gaussian. The properties of the remaining Gaussians are initialized using the same method as the 3D Gaussian splashing. The maximum contribution value of the Gaussian to the pixel at different viewing angles is calculated, and the Gaussian with a maximum contribution value less than 0.02 is eliminated.

4. A novel method for representing, rendering and reconstructing radiation fields according to claim 3, characterized in that: The semi-transparent surface element in step (3.1) is also a disk defined on the XY plane of the local coordinate system, and its opacity α i The value at coordinate (x,y) is defined as: α i (x,y)=min(1,w i G(x,y)), where α i (x,y) will be truncated to 0 when min(α i (x,y),G(x,y))<1 / 255; opacity control parameter w i ∈[0,255] is used to establish the conversion relationship between semi-transparent surface elements and opaque surface elements; when w i <1, the surface elements show a two-dimensional Gaussian distribution; as w i As w increases, the Gaussian distribution gradually becomes opaque from the center outward; i When it reaches 255, the opacity is 1 within the range of G(x,y)>1 / 255, and it is 0 outside the range, thus forming an opaque disk; This mechanism ensures that the surface element can smoothly transition from a semi-transparent state to a stable opaque state during the optimization process.

5. A novel method for representing, rendering and reconstructing radiation fields according to claim 3, characterized in that: In the step (3.1), the maximum pixel coverage of the surface element is used as the basis, and redundant surface elements are deleted during the training process. Specifically, for each surface element S in the training view j, i , calculate S i The number of pixels n that are the frontmost covered pixels in the bin i,j The front is defined as the minimum depth, and the coverage score of the surface element is defined as Where T is the total number of training views; Remove all the i <n thr Surface elements to remove redundant surface elements; for real scenes, set the threshold n thr =16.

6. A novel method for representing, rendering and reconstructing radiation fields according to claim 3, characterized in that: The step (3.2) is to calculate the maximum contribution value of Gaussian to the pixel at different viewing angles, and remove Gaussian with a maximum contribution value less than 0.

02. Specifically, by calculating each Gaussian G i Contribution score s in the jth training view i,j The calculation formula is as follows: in is the pixel covered by the Gaussian, is the opacity at the jth training view, and the color contribution c i,j Take the maximum value of the three-channel color value, that is If the maximum contribution of a Gaussian in all T training views If it is less than 0.02, the Gaussian will be eliminated.

7. A novel method for representing, rendering and reconstructing radiation fields according to claim 1, characterized in that: It also includes optimizations for anti-aliasing rendering. Specifically, when drawing the radiation field, the radiation field representation is combined with an improved Gaussian sputtering method. By applying a world space filter and an approximate screen space box filter to the three-dimensional Gaussian, and keeping the surface elements using supersampling rendering, the best anti-aliasing effect rendered at different scales is obtained.

8. A novel method for representing, rendering and reconstructing radiation fields according to claim 1, characterized in that: It also includes accelerated rendering. Specifically, in the fine-grained optimization stage of radiation field reconstruction, the method of removing Gaussians based on the maximum pixel contribution value can be replaced by a removal strategy based on the Hessian score, which is used to further remove Gaussians and obtain the best rendering frame rate under certain rendering quality conditions.

9. A novel method for representing, rendering and reconstructing radiation fields according to claim 1, characterized in that: It also includes optimizations for compressed storage, specifically: by querying the spherical harmonic coefficients of color, replacement pixels and Gaussians from the hash grid, and quantizing the scaling and rotation parameters of pixels and Gaussians using residual vector quantization, storage compression of more than 20 times is achieved.

10. A novel method for representing, rendering and reconstructing radiation fields according to claim 1, characterized in that: It also includes the optimization of geometric reconstruction, specifically: when rendering opaque surface elements, in addition to obtaining color and depth maps, a normal map can also be obtained according to the orientation of the surface element; by replacing the three-dimensional Gaussian in step (1) with a two-dimensional Gaussian, the method of rendering color in step (2) can be used to accumulate Gaussian normals by weight on the basis of the surface element normal map, thereby obtaining a final smoother normal map; replacing the normal with depth and rendering in the same way can obtain the final smoothed depth map; The truncated signed distance function method TSDF is used to reconstruct the surface mesh of the object from the multi-view depth map obtained by rendering. In order to optimize and obtain accurate geometry, in addition to using the loss function of the optimization process in step (3), a loss function of mutual supervision between depth and normal is added during the optimization process, requiring that the normal map derived from the depth map is consistent with the normal map obtained by rendering. At the same time, in the facet optimization stage, a method is added to use the normal map of the early rendering to supervise the optimization of the depth and normal maps in the later stage.

Images