A reconstruction method of planar transmission and reflection based on three-dimensional gauss

Abstract

The application discloses a reconstruction method of planar transmission and reflection based on three-dimensional Gauss, which is based on a joint representation of three-dimensional Gauss representing planar transmission and reflection and a reflection plane modeling reflection model. For a given view angle, the three-dimensional Gauss is first projected to a screen space and color is accumulated to obtain a transmission color map; then the three-dimensional Gauss is symmetric according to the reflection plane, and a reflection color map is rendered in the same way; finally, a reflection coefficient is calculated according to a Fresnel reflection model on the reflection plane, and the transmission color map and the reflection color map are combined to obtain a final rendering result. The method also designs a multi-stage optimization strategy to realize accurate separation and reconstruction of transmission and reflection, and can realize real-time high-fidelity new view synthesis in a scene containing complex planar transmission and reflection.

Description

Technical Field

[0001] This invention relates to the field of novel perspective synthesis, and more particularly to a method for reconstructing planar transmission and reflection based on three-dimensional Gaussian. Background Technology

[0002] Transparent flat glass (such as windows and display cases) is very common in interior scenes, and its surface exhibits complex transmission and reflection optical phenomena. Accurately modeling these optical phenomena is crucial for achieving high-fidelity Novel View Synthesis (NVS) in interior scenes. In recent years, NVS technology has developed rapidly, and the 3DGaussian Splatting (3DGS) method, as an advanced technology in this field, has demonstrated excellent rendering quality in various scenes. However, 3DGS still faces difficulties when processing interior scenes containing flat glass. The fundamental reason is that the 3DGS method overfits transmission and reflection using only low-opacity Gaussians, failing to correctly distinguish and model the two. This results in better fitting performance under training views, while reflection under testing views often exhibits noise or is missing.

[0003] Currently, there is no work based on 3DGS specifically researching high-fidelity rendering of indoor transparent flat glass. The closest work is Mirroraussian (J. Liu, X. Tang, F. Cheng, R. Yang, Z. Li, J. Liu, Y. Huang, J. Lin, S. Liu, X. Wu et al. Mirrorgaussian: Reflecting 3dgaussians for reconstructing mirror reflections[C] / / European Conference on Computer Vision (ECCV). Berlin : Springer, 2024: 377–393.), but it only focuses on mirrors with pure reflection and does not consider transmission phenomena. Some methods based on Neural Radiance Field (NeRF) attempt to solve the problem of mixed transmission and reflection, but such methods have limitations in terms of viewpoint range and handling of complex scenes, and lack robustness. In addition, NeRF methods generally suffer from low computational efficiency, making real-time rendering difficult. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a method for reconstructing planar transmission and reflection based on three-dimensional Gaussian methods. The specific technical solution is as follows:

[0005] A method for reconstructing planar transmission and reflection based on three-dimensional Gaussian methods includes the following steps:

[0006] Step 1: Plane transmission and reflection are jointly represented using a 3D Gaussian plane and a reflecting plane. Plane transmission uses a set of master scene Gaussian representations to model the realistic scene; plane reflection uses a set of mirror Gaussian representations, each mirror Gaussian derived symmetrically from the master scene Gaussian planes based on the reflecting planes; the transparent glass that causes transmission and reflection is represented by a reflecting plane carrying a Fresnel reflection model to calculate the weights of plane transmission and reflection in the final rendering result; the shape of the transparent glass is modeled using a glass Gaussian plane.

[0007] Step 2: Given a viewpoint, render planar transmission and planar reflection, specifically including:

[0008] S2.1: Project the main scene Gaussian and glass Gaussian onto the screen space, and accumulate the color and reflectivity of the Gaussian for each pixel; when accumulating the color map, ignore the color of the glass Gaussian; finally obtain the transmission color map and the reflection mask map; and use the reflection plane to calculate the initial reflection coefficient map, and multiply it with the reflection mask map to obtain the reflection coefficient map;

[0009] S2.2: The Gaussian of the main scene is symmetrically arranged along the reflection plane to obtain a mirror Gaussian; the mirror Gaussian is projected onto the screen space, and the color and reflection intensity of the mirror Gaussian are accumulated for each pixel to obtain an initial reflection color map and a reflection intensity map, and then the two are multiplied to obtain the final reflection color map;

[0010] S2.3: The final rendering result is obtained by weighting the transmission color map and the final reflection color map according to the reflection coefficient map;

[0011] For scenes with more than two reflective planes, the total reflectance coefficient map and reflectance color map are obtained by merging them through multiple rendering passes.

[0012] Step 3: Based on a set of multi-view images with known camera parameters and some manually marked reflection mask images of glass areas, a three-stage optimization method is used to reconstruct planar transmission and planar reflection, specifically including:

[0013] In the initial stage, only the Gaussian plane of the main scene is optimized to reconstruct the rough scene geometry and appearance; after optimization, the reflecting plane and the glass Gaussian plane are initialized.

[0014] During the plane adjustment phase, the main scene Gaussian, glass Gaussian, and reflection plane are optimized simultaneously, but the parameter gradient of the reflection color map back to the main scene Gaussian is interrupted to avoid affecting the plane position adjustment when optimizing the reflection color. In addition, during the optimization process, depth variance loss, gradient conflict loss, and reflection mask loss are added to decompose the transmission and reflection components, and an opacity perturbation mechanism is activated to improve the robustness of the decomposition.

[0015] During the joint optimization phase, all gradient backpropagation is initiated to jointly optimize the main scene Gaussian plane, the glass Gaussian plane, and the reflection plane, further improving the visual quality.

[0016] Furthermore, in step one, the main scene Gaussian, the mirror Gaussian, and the glass Gaussian all have seven attributes: position, rotation, scaling, opacity, spherical harmonic coefficient, reflection intensity, and reflection visibility; the reflecting plane includes five attributes: plane center, plane normal, plane width, plane height, and basic reflectivity.

[0017] Furthermore, the calculation of the reflection coefficient diagram in S2.1 specifically includes the following sub-steps:

[0018] Given a training viewpoint and a reflection plane P, starting from the current viewpoint, emit a series of rays that pass through the screen pixels. Calculate the intersection of the current viewing direction with the reflection plane to obtain the intersection mask H. If the ray emitted by a pixel intersects with the reflection plane, the value of H is 1, otherwise it is 0.

[0019] Additionally, to calculate an initial reflection coefficient map, if the light emitted from a pixel does not intersect with the reflecting plane, its initial reflection coefficient is 0; if there is an intersection, the formula for calculating its initial reflection coefficient is as follows:

[0020]

[0021] Where R0 is the fundamental reflectivity, n p is the plane normal, and d is the line-of-sight vector;

[0022] Based on the initial reflection coefficient, an initial reflection coefficient map is obtained. Multiplying this map with the reflection mask map yields the final reflection coefficient map.

[0023] Furthermore, the parameters of the mirror Gaussian in S2.2 are calculated as follows:

[0024] By symmetrically positioning the main Gaussian about the reflection plane, we obtain the position of the mirror Gaussian:

[0025]

[0026] in, It is the position vector of the mirror Gaussian vector, u iIt is the position vector of the Gaussian in the main scene, u p It is the center vector of the reflecting plane, n p It is the normal vector of the reflecting plane;

[0027] By symmetrically representing the rotation matrix of the Gaussian in the main scene with respect to the reflection plane, we obtain the rotation matrix of the mirror Gaussian.

[0028]

[0029]

[0030] in, The rotation matrix Q of the Gaussian in the main scene. i The three main axes, The rotation matrix is ​​a mirror Gaussian matrix;

[0031] The color of the Gaussian symmetry of the main scene is used to obtain the color of the mirrored Gaussian symmetry:

[0032]

[0033]

[0034] Among them, c i () indicates the color of the main scene calculated by Gaussian based on spherical harmonics. The direction of the line of sight after symmetry with the reflecting plane;

[0035] The scaling, opacity, spherical harmonics, reflection intensity, and reflection visibility of the mirrored Gaussian are consistent with those of the main scene Gaussian.

[0036] Furthermore, for scenes with more than two reflective planes, the overall reflectance coefficient map and reflectance color map are merged through multiple rendering passes, specifically including:

[0037] (1) Calculate the intersection mask and reflection coefficient map of a single reflection plane respectively, and obtain the mirror Gaussian by symmetrically dividing the main scene Gaussian along the reflection plane and rendering the corresponding reflection color map;

[0038] (2) Based on the intersecting mask diagrams of each reflection plane, the final reflection coefficient diagram and reflection color diagram are obtained by combining them:

[0039]

[0040]

[0041] Where, N p H represents the number of reflecting planes. k R k These represent the intersecting mask diagram and reflection coefficient diagram of the k-th reflecting plane, respectively. Let R represent the reflection color map of the k-th reflecting plane, and C be the final reflection coefficient map. r This is the final reflection color map.

[0042] Furthermore, in the initialization phase of step three, optimizing the Gaussian distribution of the main scene specifically includes:

[0043] The main scene Gaussian is initialized with point clouds obtained using the motion structure recovery method, and the transmissive color map rendered by the main scene Gaussian is used as the final rendering result, optimized using a color loss function; the specific initialization of the parameters of the main scene Gaussian is as follows:

[0044] The position and color are set to the position and color of the point cloud, the scaling value is set to the distance to the nearest neighbor, the rotation is randomly generated, the reflection intensity is set to 1.0, and the reflection visibility is set to 0.

[0045] Furthermore, the initialization phase, specifically initializing the reflective plane and the glass Gaussian plane, includes:

[0046] (1) The artificially marked reflective mask M a Edge detection is performed, and the edge is expanded by 20 pixels. The area containing these 20 pixels is considered the glass edge area.

[0047] (2) Next, the main scene Gaussian is projected onto all viewpoints containing artificially marked reflection masks. Those main scene Gaussian whose centers fall on the glass edge area are selected as representatives to fit the reflection plane.

[0048] (3) Next, the RANSAC algorithm is used to fit the parameters of the plane representing the Gaussian plane to obtain the plane normal n. p And calculate the center of the interior points obtained by the RANSAC algorithm, and use it as the plane center u. p ;

[0049] (4) Project these interior points onto the estimation plane to obtain a series of two-dimensional points; use the rotating caliper method to calculate the minimum bounding rectangle of these two-dimensional points, and use the width and height of the rectangle as the width w and height h of the reflection plane; after obtaining the above parameters, initialize the reflection plane P=(u p , n p , w, h, R0), where if the reflecting plane is transparent glass, then R0 is initialized to 0.2, and if it is a pure mirror, then it is initialized to 1.0;

[0050] (5) After the reflection plane is initialized, 1000 points are randomly sampled within its range as the position of the glass Gaussian, and the reflection visibility of the glass Gaussian is initialized to 1.0. The initialization method of the other 5 parameters of the glass Gaussian is the same as that of the main scene Gaussian.

[0051] Furthermore, the color loss function is a weighted sum of the L1 distance between the rendered image and the real image and the structural loss D-SSIM.

[0052] Furthermore, the depth variance loss during the planar adjustment phase is specifically as follows:

[0053] First, the Gaussian of the main scene is projected onto the screen space, the center depth of the Gaussian of the main scene is accumulated and normalized to obtain the rendered depth map; then, the difference between the center depth of each Gaussian of the main scene and the corresponding depth value on the rendered depth map is calculated, and the depth difference is accumulated in the same way to obtain a depth variance map. The mean of the depth variance map is the depth variance loss.

[0054] The gradient conflict loss is specifically calculated by using the Sobel operator to calculate the gradients of the transmission color map and the reflection color map, multiplying the two gradient maps and taking the average value, which is the gradient conflict loss.

[0055] The reflection mask loss is the L1 difference between the rendered reflection mask and the manually labeled reflection mask.

[0056] Furthermore, the specific implementation of the opacity perturbation mechanism is as follows:

[0057] Every 1000 optimization rounds, select the main scene Gaussians located behind the reflection plane, add random noise of -0.4 to 0.4 to their opacity, and crop them to the range of 0 to 1; if the number of iterations is a multiple of 3000, crop the opacity of all main scene Gaussians to the range of 0 to 0.01.

[0058] The beneficial effects of this invention are as follows:

[0059] This invention enables accurate separation and modeling of transmission and reflection, significantly improving the realism and detail of flat glass rendering; at the same time, this method has high rendering efficiency (more than 200 frames per second at 720p resolution), achieving real-time high-fidelity rendering. Attached Figure Description

[0060] Figure 1 This is a schematic diagram illustrating the representation and rendering of planar transmission and reflection based on three-dimensional Gaussian in this invention.

[0061] Figure 2 The image shows a transmission color map, a reflection color map, a reflection coefficient map, and a complete rendering result obtained from one embodiment of the present invention.

[0062] Figure 3 This is the rendering result of each sub-stage in the rendering process of another embodiment of the present invention. Detailed Implementation

[0063] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0064] The core of this invention lies in representing, rendering, and reconstructing planar reflection and transmission based on three-dimensional Gaussian, with the reconstruction results supporting real-time, high-quality rendering.

[0065] like Figure 1 As shown, the planar transmission and reflection reconstruction method based on three-dimensional Gaussian of the present invention includes the following steps:

[0066] Step 1: Use a 3D Gaussian plane and a reflecting plane to jointly represent planar transmission and reflection. Planar transmission is represented by a set of master scene Gaussian planes to model the realistic scene; planar reflection is represented by a set of mirror Gaussian planes, and each mirror Gaussian plane is obtained symmetrically from the master scene Gaussian planes based on the reflecting planes; the transparent glass that brings about transmission and reflection is represented by a reflecting plane, which carries a Fresnel reflection model to calculate the weights of planar transmission and reflection in the final rendering result; the shape of the transparent glass is modeled using a glass Gaussian plane.

[0067] Each main scene Gaussian and mirror Gaussian has seven attributes: position, rotation, scaling, opacity, spherical harmonic coefficient, reflection intensity, and reflection visibility. The seven attributes of the i-th Gaussian are represented by position u. i Rotation R i Scaling i Opacity i spherical harmonic coefficient SH i Reflection intensity a i and reflective visibility m i .

[0068] Each mirror Gaussian is obtained by symmetry of the main scene Gaussian with respect to the reflection plane. This representation is based on the geometric property of planar reflection: the virtual scene formed by the reflection is symmetric to its corresponding real scene with respect to the reflection plane.

[0069] The reflecting plane includes the center u of the plane. p Plane normal n p The five attributes are plane width w, plane height h, and basic reflectivity R0.

[0070] The definition of the glass Gaussian is the same as that of the main scene Gaussian. The difference is that the reflection visibility of the main scene Gaussian is 0, while the reflection visibility of the glass Gaussian is initialized to 1 and can be freely optimized.

[0071] Step Two: Given a viewpoint, render planar transmission and planar reflection; for example... Figure 1 As shown, the rendering of this invention comprises two parts:

[0072] The first step, given a viewpoint, is to project the main scene Gaussian and glass Gaussian onto screen space using the same method as 3D Gaussian Splatter (3DGS), and accumulate the color and reflectance visibility of the Gaussian for each pixel to obtain the transmission color map C. t And the reflection mask diagram M.

[0073] Transmission color diagram C t The calculation formula is as follows:

[0074] ;

[0075] ;

[0076] Where d represents the vector of the line of sight; T i Represents the cumulative transmittance up to the i-th Gaussian; u is the screen coordinate, c i Let SH represent the spherical harmonic function of the i-th Gaussian. i Calculated Gaussian color; G i This represents the two-dimensional Gaussian weights projected onto the screen space; during the accumulation process, the color of the glass Gaussian does not contribute to the total transmitted color.

[0077] This formula can not only accumulate Gaussian colors, but also... i Replace with other Gaussian properties to calculate the corresponding image, such as the cumulative Gaussian reflectance m. i The reflection mask image M is obtained. Under the supervision of reflection mask loss, the reflection mask image has a value of 1.0 in the reflection region and 0.0 in the other regions.

[0078] Next, a reflection coefficient map is calculated based on the reflecting plane: Starting from the current viewpoint, a series of rays are emitted through the screen pixels, and their intersection with the reflecting plane is determined to obtain an intersection mask map H. If a ray emitted by a pixel intersects with the reflecting plane, its value is 1; otherwise, it is 0. In addition, an initial reflection coefficient map is calculated. If a pixel does not intersect with the reflecting plane, its initial reflection coefficient is 0; if it does intersect, its initial reflection coefficient is calculated according to the Schlick reflection model.

[0079] ;

[0080] This formula is an approximation of the Fresnel reflection model, used to calculate the proportion of reflected light in the incident light, where d is the line-of-sight vector, R0 is the fundamental reflectivity, and n... p It is the normal vector of the reflecting plane.

[0081] The initial reflection coefficient map R is obtained after the calculation is completed. rawThis will be multiplied with the reflection mask map M to filter out non-reflective areas, resulting in the final reflection coefficient map R.

[0082] The second step is to obtain a mirror Gaussian by symmetrically mirroring the main scene Gaussian along the reflection plane. First, the positions are symmetric about the reflection plane:

[0083] ;

[0084] in, It is the position vector of the mirror Gaussian vector, u i It is the position vector of the Gaussian in the main scene, u p It is the center vector of the reflecting plane, n p It is the normal vector of the reflecting plane.

[0085] Next, the Gaussian rotation matrix of the main scene is symmetric about the reflection plane. A symmetric rotation matrix is ​​equivalent to symmetrically rotating its three principal axes. To ensure that the three principal axes remain in a right-handed coordinate system after symmetry, the first principal axis is symmetrically flipped, calculated as follows:

[0086] ;

[0087] ;

[0088] in, The rotation matrix Q of the Gaussian in the main scene. i The three main axes, It is the rotation matrix of the mirror Gaussian.

[0089] Finally, there's the Gaussian color for the symmetrical main scene. The Gaussian color of the main scene is calculated based on the spherical harmonic coefficients, according to the viewing direction. To calculate the mirror Gaussian color, we first need to symmetrically represent the current viewing direction along the reflection plane, using this as input to calculate the output of the Gaussian spherical harmonic function for the main scene. The calculation formula is as follows:

[0090] ;

[0091] ;

[0092] Among them, c i () indicates the color of the main scene calculated by Gaussian based on spherical harmonics. The direction of the line of sight after symmetry with the reflecting plane.

[0093] This approach avoids the large computational burden associated with direct symmetric spherical harmonics. Except for position, rotation, and color attributes, the mirrored Gaussian maintains the same scaling, opacity, spherical harmonics, reflection intensity, and reflection visibility as the main scene Gaussian.

[0094] After obtaining the mirror Gaussian, it is projected onto the screen space, and the color and reflection intensity of the mirror Gaussian are accumulated for each pixel to obtain the initial reflection color map C. raw Multiplying the reflection intensity map A and the reflection color map C yields the final reflection color map C. r Finally, the transmission color map and the final reflection color map are weighted according to the reflection coefficient map to obtain the final rendering result:

[0095] ;

[0096] If there are multiple reflecting planes, the number N is... p If the value is greater than 1, then the reflection coefficient map and reflection color map corresponding to each reflection plane are obtained through multiple rendering passes and finally combined. Specifically: for the k-th reflection plane, its intersection mask map H is calculated respectively. k and reflection coefficient diagram R k The Gaussian plane of the main scene is symmetrically symmetrically arranged along the k-th reflection plane to obtain a mirror Gaussian plane, and the corresponding reflection color map is rendered. Based on the intersecting mask diagrams of each reflection plane, the overall reflection coefficient diagram R and reflection color diagram C are obtained by combining them. r :

[0097] ;

[0098] ;

[0099] Step 3: Based on a set of multi-view images with known camera parameters and some manually marked reflection mask images of glass areas, a three-stage optimization method is used to reconstruct planar transmission and planar reflection.

[0100] The optimization of this invention is a multi-stage optimization process. Given a set of multi-view images containing planar transmission and reflection that have been calibrated by the Structure from Motion (SFM) method, and a reflection mask for some manually annotated glass areas, the purpose of this invention is to optimize the three-dimensional Gaussian plane and the reflection plane to achieve high-fidelity real-time synthesis of new viewpoints for planar transmission and reflection.

[0101] This invention employs a multi-stage optimization strategy. In the initial stage, only the main scene Gaussian is optimized to reconstruct the approximate geometry and appearance of the scene. After optimization, a coarse reflection plane and glass Gaussian are estimated and initialized using a manually labeled reflection mask. In the plane adjustment stage, the reflection plane and glass Gaussian are jointly optimized, but the parameter gradients propagated back from the reflection image to the main scene Gaussian are interrupted, focusing instead on adjusting the reflection plane position. In the joint optimization stage, the learning rate for the reflection plane position and normal is reduced, and the main scene Gaussian, glass Gaussian, and reflection plane are jointly optimized. All gradient backpropagation is initiated to further improve the scene's appearance. In the initialization stage, the loss function only includes the color loss function. ,in .

[0102] Starting from the plane adjustment stage, a depth variance loss L is also added. d Gradient conflict loss L c and reflection mask loss L m The parameters are used to decompose transmission and reflection, with weights of 0.005, 0.2, and 0.5, respectively. An opacity perturbation mechanism is also added to prevent the optimization from getting trapped in local optima. Figure 3 The results of the optimized rendering were shown in different iterations, with the first 3,000 iterations being the initialization phase, the next 3,000 iterations being the planar adjustment phase, and the last 24,000 iterations being the joint optimization phase.

[0103] The first stage is the initialization stage. The main scene Gaussian is initialized using the point cloud obtained from SFM (Structured Motion Recovery). The position and color of the Gaussian are set to the position and color of the point cloud, the scaling value of the Gaussian is set to its distance to the nearest neighbor, rotation is randomly generated, reflection intensity is set to 1.0, and reflection visibility is set to 0. This stage uses the transmission color map rendered by the main scene Gaussian as the final rendering result, and optimizes it using a color loss function. After the initialization stage, the main scene Gaussian has become a coarse representation of the scene. Then, combined with manually labeled reflection masks, the position of the reflection plane can be estimated: for each manually labeled reflection mask M... a Edge detection is performed, and the edges are expanded by 20 pixels; this 20-pixel range is considered the glass edge region. Next, the main scene Gaussian is projected onto all viewpoints containing artificially marked reflection masks. Those main scene Gaussian masks whose centers fall within the glass edge region are selected as representatives to fit the reflection plane. Then, the RANSAC algorithm is used to fit the plane parameters of these representative Gaussian masks to obtain the plane normal n. p And calculate the center of the interior points obtained by the RANSAC algorithm, and use it as the plane center u. pProjecting these interior points onto the estimation plane yields a series of two-dimensional points. The minimum bounding rectangle of these two-dimensional points is calculated using the rotating caliper method, and the width and height of the rectangle are used as the width w and height h of the reflecting plane. After obtaining these parameters, the reflecting plane P=(u p , n p For the reflective plane (r, w, h, R0), if it is transparent glass, R0 is initialized to 0.2; if it is a pure mirror, it is initialized to 1.0. After the reflective plane is initialized, 1000 points are randomly sampled within its range as the positions of the glass Gaussian, and its reflective visibility is initialized to 1.0. The other 5 parameters are initialized in the same way as the Gaussian initialization method of the main scene.

[0104] The second stage is the planar adjustment stage. This involves adjusting the transmissive color map C of the main scene's Gaussian rendering. t Reflection color map C rendered by mirror Gaussian r The final rendering result C is a weighted average of the reflection coefficient map R calculated from the reflection plane, while simultaneously optimizing the Gaussian distribution of the main scene, the glass Gaussian distribution, and the reflection plane. In addition to the color loss function, a depth variance loss L is added. d Gradient conflict loss L c and reflection mask loss L m Used to separate transmission and reflection.

[0105] Depth variance loss L d Based on an observation in 3D space: Original 3DGS tends to use a large number of low-opacity Gaussian vectors far from the real surface to fit reflections, which often results in reflection artifacts from new perspectives. To eliminate these artifacts, depth variance loss minimizes the distance between the Gaussian center depth and the rendered depth map, eliminating a large number of low-opacity Gaussian vectors far from the real surface to promote transmission and reflection decomposition. First, the center depth of the Gaussian vectors in the main scene is accumulated to screen space and normalized to obtain the depth map D. t The calculation formula is as follows:

[0106] ;

[0107] Where T t (u) represents the cumulative transmittance, z i Let represent the center depth of the Gaussian of the i-th main scene.

[0108] The difference between the center depth of each Gaussian in the main scene and the corresponding depth map is accumulated to obtain the depth variance loss L. d :

[0109] ;

[0110] Here, H is the intersecting mask image, and multiplying by H is to avoid the loss of depth variance affecting the image quality of non-reflective areas.

[0111] Gradient conflict loss L c This is based on an observation of two-dimensional images: a significant image gradient appearing in the reflectance color map is less likely to also appear in the transmittance color map. Gradient conflict loss suppresses gradient similarity between the transmittance and reflectance color maps by minimizing the product of their gradients, thus achieving a finer decomposition. The calculation formula is as follows:

[0112] ;

[0113] in, It is the Sobel operator used to calculate the image gradient. sg() means to interrupt gradient backpropagation to avoid contamination of the reflection color map.

[0114] Reflection mask loss L m It uses a manually labeled reflection mask to supervise the rendering of the reflection mask, ensuring that the value is 1 in the reflection area and 0 in other areas. The calculation is as follows:

[0115] ;

[0116] Where M(u) is the rendered reflection mask, M a This is a manually labeled reflection mask.

[0117] In addition to the loss function, an opacity perturbation operation is performed to further improve the robustness of the decomposition. Specifically, every 1000 optimization iterations, the main scene Gaussians located behind the reflection plane are selected, and random noise of -0.4 to 0.4 is added to their opacity, which is then clipped to the range of 0 to 1. If the number of iterations is a multiple of 3000, the opacity of all main scene Gaussians is clipped to the range of 0 to 0.01. This simple operation simultaneously perturbs the depth variance loss and gradient conflict loss, allowing the optimization to escape local optima and improving the robustness of the decomposition.

[0118] Meanwhile, to avoid conflicts caused by the joint optimization of the reflection color and the position of the reflection plane, the parameter gradients back from the reflection color map to the Gaussian of the main scene are interrupted in the second stage, and the focus is on optimizing the plane position.

[0119] The third stage is the joint optimization stage. Having obtained the accurate planar position, the center of the reflection plane, the plane normal, and the learning rate are reduced to the initial 0.1. The rendering method, loss function, and opacity perturbation operation in this stage are consistent with the second stage, and backpropagation of all gradients is initiated to jointly optimize all parameters to further improve rendering quality.

[0120] Implementation Example 1

[0121] The inventors implemented an example of this invention on a desktop computer equipped with an i9-13900KF CPU and an NVIDIA RTX 4090 GPU. For real-world datasets, the average training time was 33 minutes. On the novel perspective synthesis task, the method achieved a rendering frame rate of 225 frames per second at a 720p rendering resolution, with a peak signal-to-noise ratio (PSNR) of 31.21.

[0122] Figure 2 In the figure, (a) is the transmission color map, (b) is the reflection color map, (c) is the reflection coefficient map, and (d) is the combined image. As can be seen from the figure, this method can achieve accurate decoupling of transmission and reflection, and synthesize high-fidelity new perspective images in real time at 225 FPS with a resolution of 960×720.

[0123] Implementation Example 2

[0124] The inventors implemented an example of the invention on a desktop computer equipped with an i9-13900KF central processing unit and an NVIDIA RTX 4090 graphics processor. Figure 3 The various sub-stages of the multi-stage optimization of this method are demonstrated. Figure 3 (a) in the text refers to the initialization phase. Figure 3 (b) in the diagram represents the planar adjustment stage. Figure 3 In the figure, (c) represents the joint optimization stage, and the number in the upper left corner of each image indicates the optimization iteration round. As can be seen from the figure, the multi-stage optimization strategy of this method can effectively reconstruct high-quality planar transmission and reflection.

[0125] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for reconstructing planar transmission and reflection based on three-dimensional Gaussian methods, characterized in that, Includes the following steps: Step 1: Plane transmission and reflection are jointly represented using a 3D Gaussian plane and a reflecting plane. Plane transmission uses a set of master scene Gaussian representations to model the realistic scene; plane reflection uses a set of mirror Gaussian representations, each mirror Gaussian derived symmetrically from the master scene Gaussian planes based on the reflecting planes; the transparent glass that causes transmission and reflection is represented by a reflecting plane carrying a Fresnel reflection model to calculate the weights of plane transmission and reflection in the final rendering result; the shape of the transparent glass is modeled using a glass Gaussian plane. Step 2: Given a viewpoint, render planar transmission and planar reflection, specifically including: S2.1: Project the main scene Gaussian and glass Gaussian onto the screen space, and accumulate the color and reflectivity of the Gaussian for each pixel; when accumulating the color map, ignore the color of the glass Gaussian; finally obtain the transmission color map and the reflection mask map; and use the reflection plane to calculate the initial reflection coefficient map, and multiply it with the reflection mask map to obtain the reflection coefficient map; S2.2: The Gaussian of the main scene is symmetrically arranged along the reflection plane to obtain a mirror Gaussian; the mirror Gaussian is projected onto the screen space, and the color and reflection intensity of the mirror Gaussian are accumulated for each pixel to obtain an initial reflection color map and a reflection intensity map, and then the two are multiplied to obtain the final reflection color map; S2.3: The final rendering result is obtained by weighting the transmission color map and the final reflection color map according to the reflection coefficient map; For scenes with more than two reflective planes, the total reflectance coefficient map and reflectance color map are obtained by merging them through multiple rendering passes. Step 3: Based on a set of multi-view images with known camera parameters and some manually marked reflection mask images of glass areas, a three-stage optimization method is used to reconstruct planar transmission and planar reflection, specifically including: In the initial stage, only the Gaussian plane of the main scene is optimized to reconstruct the rough scene geometry and appearance; after optimization, the reflecting plane and the glass Gaussian plane are initialized. During the plane adjustment phase, the main scene Gaussian, glass Gaussian, and reflection plane are optimized simultaneously, but the parameter gradient of the reflection color map back to the main scene Gaussian is interrupted to avoid affecting the plane position adjustment when optimizing the reflection color. In addition, during the optimization process, depth variance loss, gradient conflict loss, and reflection mask loss are added to decompose the transmission and reflection components, and an opacity perturbation mechanism is activated to improve the robustness of the decomposition. During the joint optimization phase, all gradient backpropagation is initiated to jointly optimize the main scene Gaussian plane, the glass Gaussian plane, and the reflection plane, further improving the visual quality.

2. The reconstruction method for planar transmission and reflection based on three-dimensional Gaussian as described in claim 1, characterized in that, In step one, the main scene Gaussian, the mirror Gaussian, and the glass Gaussian all have seven attributes: position, rotation, scaling, opacity, spherical harmonic coefficient, reflection intensity, and reflection visibility; the reflection plane includes five attributes: plane center, plane normal, plane width, plane height, and basic reflectivity.

3. The reconstruction method for planar transmission and reflection based on three-dimensional Gaussian as described in claim 1, characterized in that, The calculation of the reflection coefficient diagram in S2.1 specifically includes the following sub-steps: Given a training viewpoint and a reflection plane P, starting from the current viewpoint, emit a series of rays that pass through the screen pixels. Calculate the intersection of the current viewing direction with the reflection plane to obtain the intersection mask H. If the ray emitted by a pixel intersects with the reflection plane, the value of H is 1, otherwise it is 0. Additionally, to calculate an initial reflection coefficient map, if the light emitted from a pixel does not intersect with the reflecting plane, its initial reflection coefficient is 0; if there is an intersection, the formula for calculating its initial reflection coefficient is as follows: ； Where R0 is the fundamental reflectivity, n p is the plane normal, and d is the line-of-sight vector; Based on the initial reflection coefficient, an initial reflection coefficient map is obtained. Multiplying this map with the reflection mask map yields the final reflection coefficient map.

4. The reconstruction method for planar transmission and reflection based on three-dimensional Gaussian as described in claim 2, characterized in that, The parameters of the mirror Gaussian described in S2.2 are calculated as follows: By symmetrically positioning the main Gaussian about the reflection plane, we obtain the position of the mirror Gaussian: ； in, It is the position vector of the mirror Gaussian vector, u i It is the position vector of the Gaussian in the main scene, u p It is the center vector of the reflecting plane, n p It is the normal vector of the reflecting plane; By symmetrically representing the rotation matrix of the Gaussian in the main scene with respect to the reflection plane, we obtain the rotation matrix of the mirror Gaussian. ； ； in, The rotation matrix Q of the Gaussian in the main scene. i The three main axes, The rotation matrix is ​​a mirror Gaussian matrix; The color of the Gaussian symmetry of the main scene is used to obtain the color of the mirrored Gaussian symmetry: ； Among them, c i () indicates the color of the main scene calculated by Gaussian based on spherical harmonics. The direction of the line of sight after symmetry with the reflecting plane; The scaling, opacity, spherical harmonics, reflection intensity, and reflection visibility of the mirrored Gaussian are consistent with those of the main scene Gaussian.

5. The reconstruction method for planar transmission and reflection based on three-dimensional Gaussian as described in claim 1, characterized in that, For scenes with more than two reflective planes, the overall reflectance coefficient map and reflectance color map are obtained by merging them through multiple rendering passes. Specifically, these include: (1) Calculate the intersection mask and reflection coefficient map of a single reflection plane respectively, and obtain the mirror Gaussian by symmetrically dividing the main scene Gaussian along the reflection plane and rendering the corresponding reflection color map; (2) Based on the intersecting mask diagrams of each reflection plane, the final reflection coefficient diagram and reflection color diagram are obtained by combining them: ； Where, N p H represents the number of reflecting planes. k R k These represent the intersecting mask diagram and reflection coefficient diagram of the k-th reflecting plane, respectively. Let R represent the reflection color map of the k-th reflecting plane, and C be the final reflection coefficient map. r This is the final reflection color map.

6. The method for reconstructing planar transmission and reflection based on three-dimensional Gaussian as described in claim 2, characterized in that, The initialization phase of step three, specifically optimizing the Gaussian of the main scene, includes: The main scene Gaussian is initialized with point clouds obtained using the motion structure recovery method, and the transmissive color map rendered by the main scene Gaussian is used as the final rendering result, optimized using a color loss function; the specific initialization of the parameters of the main scene Gaussian is as follows: The position and color are set to the position and color of the point cloud, the scaling value is set to the distance to the nearest neighbor, the rotation is randomly generated, the reflection intensity is set to 1.0, and the reflection visibility is set to 0.

7. The method for reconstructing planar transmission and reflection based on three-dimensional Gaussian as described in claim 6, characterized in that, The initialization phase specifically includes initializing the reflective plane and the glass Gaussian plane: (1) The artificially marked reflective mask M a Edge detection is performed, and the edge is expanded by 20 pixels. The area containing these 20 pixels is considered the glass edge area. (2) Next, the main scene Gaussian is projected onto all viewpoints containing artificially marked reflection masks. Those main scene Gaussian whose centers fall on the glass edge area are selected as representatives to fit the reflection plane. (3) Next, the RANSAC algorithm is used to fit the parameters of the plane representing the Gaussian plane to obtain the plane normal n. p And calculate the center of the interior points obtained by the RANSAC algorithm, and use it as the plane center u. p ; (4) Project these interior points onto the estimation plane to obtain a series of two-dimensional points; use the rotating caliper method to calculate the minimum bounding rectangle of these two-dimensional points, and use the width and height of the rectangle as the width w and height h of the reflection plane; after obtaining the above parameters, initialize the reflection plane P=(u p , n p , w, h, R0), where if the reflecting plane is transparent glass, then R0 is initialized to 0.2, and if it is a pure mirror, then it is initialized to 1.0; (5) After the reflection plane is initialized, 1000 points are randomly sampled within its range as the position of the glass Gaussian, and the reflection visibility of the glass Gaussian is initialized to 1.

0. The initialization method of the other 5 parameters of the glass Gaussian is the same as that of the main scene Gaussian.

8. The method for reconstructing planar transmission and reflection based on three-dimensional Gaussian as described in claim 7, characterized in that, The color loss function is a weighted sum of the L1 distance between the rendered image and the real image and the structural loss D-SSIM.

9. The method for reconstructing planar transmission and reflection based on three-dimensional Gaussian as described in claim 8, characterized in that, The depth variance loss during the plane adjustment phase is specifically as follows: First, the Gaussian of the main scene is projected onto the screen space, the center depth of the Gaussian of the main scene is accumulated and normalized to obtain the rendered depth map; then, the difference between the center depth of each Gaussian of the main scene and the corresponding depth value on the rendered depth map is calculated, and the depth difference is accumulated in the same way to obtain a depth variance map. The mean of the depth variance map is the depth variance loss. The gradient conflict loss is specifically calculated by using the Sobel operator to calculate the gradients of the transmission color map and the reflection color map, multiplying the two gradient maps and taking the average value, which is the gradient conflict loss. The reflection mask loss is the L1 difference between the rendered reflection mask and the manually labeled reflection mask.

10. The method for reconstructing planar transmission and reflection based on three-dimensional Gaussian as described in claim 1, characterized in that, The specific mechanism for initiating opacity perturbation is as follows: Every 1000 optimization rounds, select the main scene Gaussian located behind the reflection plane, add random noise of -0.4 to 0.4 to its opacity, and crop it to the range of 0 to 1; If the number of iterations is a multiple of 3000, then the opacity of all main scene Gaussians will be cropped to the range of 0 to 0.01.

Images