Reflecting object inverse rendering method, system and equipment based on two-dimensional Gaussian sputtering and multi-mode diffusion prior and medium

Abstract

The invention discloses a reflecting object inverse rendering method, system and device based on two-dimensional Gaussian sputtering and multi-mode diffusion prior and a medium. The method comprises the steps that images and camera parameters of a reflecting object under multiple visual angles are acquired; calculating an observation angle according to the camera parameters; performing rasterization rendering on the images of the reflecting object under the multiple visual angles by using the trained two-dimensional Gaussian primitives under the observation visual angle to obtain a rendered image under the new visual angle; according to the method, two-dimensional Gaussian is adopted as a scene representation element, surface characteristics are better fitted, and geometric reconstruction precision can be remarkably improved; the trained two-dimensional Gaussian primitive is closer to a real three-dimensional scene, and experimental tests of multiple data sets show that the method not only can accurately reconstruct geometric and material distribution of a high-reflection area, but also can recover clear environment illumination with high-frequency details, realizes efficient and real reflection inverse rendering, and has a good application prospect. The method can be widely applied to downstream tasks such as three-dimensional reconstruction, material editing and relighting.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and graphics, and in particular to a method, system, device and medium for inverse rendering of reflective objects based on two-dimensional Gaussian splatting (2DGS) and multimodal diffusion prior, which is used to recover the geometric structure, material properties and lighting information of reflective objects from multi-view images. Background Art

[0002] Inverse rendering is an important technology at the intersection of computer vision and graphics, aiming to recover the geometry, material, and lighting information of a scene from multi-view images. In recent years, the emergence of neural rendering technologies such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) has promoted the development of inverse rendering. However, existing methods still face the following problems when dealing with highly reflective objects: (1) The computational overhead of implicit representation: NeRF relies on dense ray sampling and high-dimensional function fitting, resulting in excessively long training time; (2) Lighting-material ambiguity: Highly reflective surfaces cause the incident light to be highly coupled with the material properties, making it difficult to decouple; (3) Insufficient geometric reconstruction accuracy: 3D Gaussian Splatting (3DGS) has difficulty in accurately modeling the surface details of specular reflective objects due to the conflict between volumetric and surface properties; (4) Lack of secondary ray modeling: 3DGS, based on rasterization, lacks a ray tracing mechanism and cannot explicitly model ambient occlusion and interreflection effects.

[0003] While existing 3DGS-based methods improve rendering efficiency, their geometric reconstruction ambiguity and multi-view inconsistencies limit the realistic rendering of reflective objects. Therefore, an efficient and high-precision inverse rendering method is urgently needed to address these issues. Summary of the Invention

[0004] In response to the shortcomings of existing inverse rendering methods for reflective objects in terms of reconstruction accuracy and efficiency, the present invention proposes an inverse rendering method, system, equipment and medium for reflective objects that integrates two-dimensional Gaussian sputtering and multimodal diffusion priors. This method can significantly improve the accuracy of geometric and material reconstruction while maintaining rendering efficiency, and achieve efficient and high-fidelity inverse rendering results.

[0005] To achieve the above object, the present invention is implemented through the following technical solutions:

[0006] A first aspect of the present invention provides a method for inverse rendering of reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior, comprising the following steps:

[0007] Obtain images and camera parameters of reflective objects at multiple viewing angles;

[0008] The observation angle is calculated based on the camera parameters;

[0009] Under the observation perspective, the trained two-dimensional Gaussian primitives are used to rasterize and render the images of the reflective object under multiple perspectives to obtain the rendered images under the new perspective;

[0010] The trained two-dimensional Gaussian primitive is obtained through the following process:

[0011] The dense point cloud is restored by the motion structure recovery method using the images and camera parameters of the reflective object at multiple perspectives; a two-dimensional Gaussian primitive is initialized at each point cloud position, and then the initialized two-dimensional Gaussian primitive is trained, and the position, orientation, shape and appearance attributes of the two-dimensional Gaussian primitive are continuously optimized and adjusted, so that the two-dimensional Gaussian primitive gradually fits the real three-dimensional scene until the preset training stop condition is reached, and the trained two-dimensional Gaussian primitive is obtained.

[0012] Furthermore, a two-dimensional Gaussian primitive is initialized at each point cloud position using the following formula:

[0013] P(u,v)=p k +vs u t u u+s v t v v=H(u,v,1,1) (1)

[0014] Where P(u,v) is the mathematical expression of the two-dimensional Gaussian basis element under the local tangent plane, (u,v) is the local tangent plane, u and v are the coordinates of the first and second tangent planes respectively, is the homogeneous transformation matrix representing the two-dimensional Gaussian geometric structure, p k represents the position of the two-dimensional Gaussian basis element in three-dimensional space, s u is a scalar that controls the variance of the first vector direction, t u is the first vector, u is the first tangent plane coordinate, s v is a scalar that controls the variance of the first vector direction, t v is the second vector, v is the second tangent plane coordinate, and H represents the homogeneous transformation matrix of the two-dimensional Gaussian geometric structure.

[0015] Furthermore, the preset training stop condition is to train N times, where Then, step 1) is used to obtain the first rendered image. Step 2) is performed again to obtain a second rendered image; a total loss function including a rendering error and a regularization term is constructed based on the first rendered image or the second rendered image;

[0016] Step 1): Calculate the color value of each two-dimensional Gaussian basis element in the three-dimensional scene represented by the two-dimensional Gaussian using a physically based rendering function;

[0017] Map each 2D Gaussian basis element to the image plane;

[0018] Accumulating color values of the two-dimensional Gaussian primitives mapped to the image plane in a front-to-back order using an Alpha blending method to obtain a first rendered image;

[0019] Step 2): Map each 2D Gaussian basis element to the image plane;

[0020] Performing raster rendering on the normals and material properties of the two-dimensional Gaussian primitives projected onto the image plane, and accumulating the normals and material properties of the two-dimensional Gaussian primitives projected onto the image plane from front to back to obtain a two-dimensional normal map and material property map;

[0021] According to the two-dimensional normal map and material attribute map, the normal and material attribute of each pixel of the camera plane are obtained, and the color value of each pixel of the camera plane is calculated using a physically based rendering function to generate a second rendered image.

[0022] Furthermore, the total loss function is:

[0023]

[0024] Among them, λ d ,λ n ,λ np ,λ mp are the first, second, third, and fourth hyperparameters, is the total loss function, is the RGB reconstruction loss, is the depth distortion loss, is the normal consistency loss, is the normal supervision loss of the diffusion prior, Monitor loss for materials.

[0025] Furthermore, RGB reconstruction loss Calculated by the following formula:

[0026]

[0027] Among them, the hyperparameter λ=0.2, is the absolute error loss, is the D-SSIM loss;

[0028] Depth distortion loss Calculated by the following formula:

[0029]

[0030] Among them, ω k is the mixing weight of the kth intersection, ω l is the mixing weight of the lth intersection, z k is the depth of the k-th intersection, z l is the depth at the l-th intersection, k and l represent the indices of different Gaussian basis points;

[0031] Normal consistency loss Calculated by the following formula:

[0032]

[0033] Among them, n k is the local 2D Gaussian normal, N d is the depth normal;

[0034] Normal Supervision Loss with Diffusion Prior Calculated by the following formula:

[0035]

[0036] in, represents the rendered normal map, and N represents the normal prior pseudo label.

[0037] Furthermore, the physically based rendering function is:

[0038] c(ω o )=a(1-m)L diffuse +M specular L specular +c r (ω r ) (3)

[0039] where c(ω o ) is the color observed from the viewing angle of the two-dimensional Gaussian basis element, a is the albedo, m is the metalness, L diffuse and L specular are the integrals of diffuse and specular lighting, M specular is the integral of BRDF bidirectional reflectance distribution function and pure white ambient light, c r (ω r ) is the residual term of reflected indirect light, ω r is the reflection direction.

[0040] Furthermore, the color value c(x) of each pixel of the first rendered image is:

[0041]

[0042] Where K is the total number of Gaussian primitives, k is the index of Gaussian primitives, and ck represents the color of the k-th Gaussian primitive, w k is the weight;

[0043] Weight w k Calculated by the following formula:

[0044]

[0045] where α k Indicates the transparency of the k-th Gaussian primitive, j is the index of the Gaussian primitive before the k-th Gaussian primitive, α j Indicates the transparency of the j-th Gaussian primitive.

[0046] A second aspect of the present invention provides a reflective object inverse rendering system based on two-dimensional Gaussian sputtering and multimodal diffusion prior, comprising:

[0047] Image and parameter acquisition module, used to obtain images and camera parameters of reflective objects at multiple viewing angles;

[0048] An observation angle calculation module, used to calculate the observation angle according to camera parameters;

[0049] A rasterization rendering module is used to perform rasterization rendering on the images of the reflective object at multiple viewing angles using the trained two-dimensional Gaussian primitives under the observation angle to obtain a rendered image at a new viewing angle;

[0050] The trained two-dimensional Gaussian primitive is obtained through the following process:

[0051] The dense point cloud is restored by the motion structure recovery method using the images and camera parameters of the reflective object at multiple perspectives; a two-dimensional Gaussian primitive is initialized at each point cloud position, and then the initialized two-dimensional Gaussian primitive is trained, and the position, orientation, shape and appearance attributes of the two-dimensional Gaussian primitive are continuously optimized and adjusted, so that the two-dimensional Gaussian primitive gradually fits the real three-dimensional scene until the preset training stop condition is reached, and the trained two-dimensional Gaussian primitive is obtained.

[0052] A third aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the inverse rendering method of reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior is implemented.

[0053] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the inverse rendering method for reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] Compared to 3D Gaussian-based scene representation methods, this method uses 2D Gaussians as scene representation primitives, which better fit surface characteristics and significantly improve geometric reconstruction accuracy. The trained 2D Gaussian primitives in this method are closer to the actual 3D scene. Experimental testing on multiple datasets demonstrates that this method not only accurately reconstructs the geometry and material distribution of highly reflective areas, but also restores clear ambient lighting with high-frequency details. This method enables efficient and realistic inverse reflection rendering and is widely applicable to downstream tasks such as 3D reconstruction, material editing, and relighting.

[0056] Furthermore, compared to the inverse rendering method of reflective objects using only forward shading or deferred shading, the two-stage hybrid shading strategy adopted in the present invention can, on the one hand, achieve stable geometric optimization with the help of forward shading, and on the other hand, use deferred shading to align the reflection direction by aggregating normals, thereby restoring high-frequency ambient lighting and enhancing the reflection effect.

[0057] Furthermore, compared with existing inverse rendering methods, the present invention realizes the joint optimization of materials and normals through diffusion prior, reduces the ambiguity of illumination and material decomposition in the inverse rendering problem, and optimizes the decoupling capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of the inverse rendering method of reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior in the present invention;

[0059] Figure 2 : This is a comparison diagram of the effects of the present invention and the existing method in the new perspective synthesis task of reflective objects; wherein, (a) is a real image of the "bell" scene, (b) is a GShader-rendered image of the "bell" scene, (c) is a rendered image of the "bell" scene by the present invention, (d) is a real image of the "tbell" scene, (e) is a GShader-rendered image of the "tbell" scene, (f) is a rendered image of the "tbell" scene by the present invention, (g) is a real image of the "car" scene, (h) is a GShader-rendered image of the "car" scene, and (i) is a rendered image of the "car" scene by the present invention;

[0060] Figure 3 The effects of new perspective synthesis, geometric reconstruction, material decomposition, and illumination restoration in the "teapot" scene of the present invention are demonstrated in the figure; (a) is the real image, (b) is the rendered image, (c) is the diffuse reflection image, (d) is the specular reflection image, (e) is the normal image, (f) is the albedo image, (g) is the roughness image, (h) is the metalness image, and (i) is the ambient lighting;

[0061] Figure 4 Schematic diagram of the inverse rendering system for reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior in the present invention. DETAILED DESCRIPTION

[0062] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present invention. However, the present invention may be implemented in a variety of different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present invention.

[0063] This paper presents an inverse rendering method for reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion priors. This method uses a flat 2D Gaussian as a scene representation primitive to achieve improved geometric reconstruction accuracy. Combined with a hybrid forward-deferred shading strategy and a multimodal diffusion prior, it achieves efficient geometric reconstruction of reflective objects and decouples material illumination from the underlying image. Experimental testing on multiple datasets demonstrates the effectiveness of this method, which not only accurately reconstructs the geometry and material distribution of highly reflective areas, but also restores clear ambient lighting with high-frequency details.

[0064] like Figure 1 As shown, the present invention proposes a method for inverse rendering of a three-dimensional scene reflective object based on a two-dimensional Gaussian representation, which specifically includes the following steps:

[0065] Step 1: Data Acquisition and Preprocessing. Images of reflective objects from multiple viewpoints and their corresponding camera parameters are collected to construct a reflective object dataset. This dataset is then divided into training and test sets for subsequent model training and evaluation.

[0066] Step 2: Construct a three-dimensional scene under the two-dimensional Gaussian representation. This method uses two-dimensional Gaussian as the representation primitive for reflective objects in three-dimensional scenes, which has both geometric and appearance description capabilities. Before the start of training, the dense point cloud is restored by the Structure from Motion (SFM) method using the images of reflective objects in the training set under multiple perspectives and the corresponding camera parameters. Subsequently, a two-dimensional Gaussian basis element is initialized at each point cloud position. The initialized two-dimensional Gaussian basis element is then trained. As the training process progresses, the position, direction, shape and appearance attributes of the two-dimensional Gaussian basis element are continuously optimized and adjusted so that the two-dimensional Gaussian basis element gradually fits the real three-dimensional scene until the preset training stop condition is reached. The construction of the three-dimensional scene under the two-dimensional Gaussian representation is completed, and the trained two-dimensional Gaussian basis element is obtained.

[0067] The specific steps to initialize a two-dimensional Gaussian primitive at each point cloud position are:

[0068] Each 2D Gaussian primitive is defined on the local tangent plane (u, v). Each 2D Gaussian primitive is geometrically defined by a center point p k , two orthogonal tangent vectors (the first vector t u , the second vector t v ) and scaling vector s=(s u ,s v ) parameterized. u is a scalar that controls the variance of the first vector direction, s v A scalar that controls the variance of the second vector direction.

[0069] Among them, the center point p k Represents the position of the two-dimensional Gaussian primitive in three-dimensional space; the orthogonal tangent vector (the first vector t u , the second vector t v ) describes the orientation of the two-dimensional Gaussian primitive on the local tangent plane; the scaling vector (s u ,s v ) respectively control the two-dimensional Gaussian distribution in (the first vector t u , the second vector t v ) direction; the tangent plane normal vector is given by t w =t u ×t v It is calculated that it represents the direction of the steepest density change on the local surface, thereby more accurately describing the surface characteristics of the thin layer structure.

[0070] Define the rotation matrix R = [t u ,t v ,t w ], scaling factor S = diag(s u ,s v ,0), the two-dimensional Gaussian basis element can be expressed as:

[0071] P(u,v)=p k +s u t u u+s v t v v=H(u,v,1,1) (1)

[0072] Where P(u,v) is the mathematical expression of the two-dimensional Gaussian basis element under the local tangent plane (u,v), u and v are the coordinates of the first tangent plane and the second tangent plane respectively. is a homogeneous transformation matrix representing a two-dimensional Gaussian geometry:

[0073]

[0074] Each two-dimensional Gaussian primitive also has parameters such as transparency α, albedo a, metalness m and roughness r, which together constitute the appearance characteristics of each two-dimensional Gaussian primitive and participate in subsequent rasterization rendering and physically based rendering processes.

[0075] The initialized two-dimensional Gaussian primitive includes the geometric features and appearance features (transparency α, albedo a, metalness m and roughness r) of the two-dimensional Gaussian primitive.

[0076] Step 3: Use the camera parameters in the training set to rasterize and render the 3D scene under the 2D Gaussian representation to generate a rendered image that is consistent with the data perspective in the training set.

[0077] In order to effectively model the rendering effect of reflective objects, this paper proposes a two-stage hybrid shading strategy to perform raster rendering on three-dimensional scenes under two-dimensional Gaussian representation, including the first stage and the second stage. The preset training stop condition is 50,000 training times, of which the first 20,000 times are performed using the first stage and the last 30,000 times are performed using the second stage. Figure 1 As shown in the figure, the first stage is forward shading. In this stage, the color value of each two-dimensional Gaussian primitive in the three-dimensional scene under the two-dimensional Gaussian representation is calculated through a physically based rendering function. The Split Sum Approximation (SSA) method is used to solve the physically based rendering function to improve rendering efficiency. The physically based rendering function is:

[0078] c(ω o )=a(1-m)L diffuse +M specular L specular +c r (ω r ) (3)

[0079] where c(ω o ) is a two-dimensional Gaussian basis element with direction ω o Observed color (direction ω o is the viewing angle, calculated using camera parameters, the calculation process is well known in the art), a is albedo, m is metalness, L diffuse and L specular are the integrals of diffuse and specular lighting, respectively, which can be calculated in the form of a multi-level environment map, M specular is the integral of BRDF (Bidirectional Reflectance Distribution Function) and pure white ambient light, which is a scalar that can be precalculated. r (ω r ) is the residual term of reflected indirect light, which is about the reflection direction ωr function.

[0080] After calculating the color value of each 2D Gaussian primitive in the 3D scene represented by the 2D Gaussian, rasterization rendering is used to map each 2D Gaussian primitive to the image plane. The specific process is as follows: First, the world-to-camera view projection matrix W (obtained using camera parameters, the calculation process is well known in the art) is combined with the geometric transformation matrix H to project each 2D Gaussian primitive to the screen space to obtain the projection coordinates x = (xy, yz, z, z) T =WH(u,v,1,1) T , where (u,v) is the local tangent plane.

[0081] Then, the color values of the two-dimensional Gaussian primitives mapped to the screen space are accumulated in a front-to-back order using the Alpha (Alpha-Blending, transparency) blending method to obtain the first rendered image. The color value c(x) of each pixel in the first rendered image is:

[0082]

[0083] Where K is the total number of Gaussian primitives, k is the index of Gaussian primitives, and c k represents the color of the k-th Gaussian primitive, w k is the weight, weight w k Calculated by the following formula:

[0084]

[0085] where α k Indicates the transparency of the k-th Gaussian primitive, j is the index of the Gaussian primitive before the k-th Gaussian primitive, α j Indicates the transparency of the j-th Gaussian primitive.

[0086] The second stage is the deferred shading stage. In this stage, first, as in the forward shading stage, the world-to-camera view projection matrix W (obtained using the camera parameters) is combined with the geometric transformation matrix H to project each two-dimensional Gaussian primitive into the screen space to obtain the projection coordinates. , where (u,v) is the local tangent plane.

[0087] Subsequently, the normals and material attributes of the two-dimensional Gaussian primitives projected into the screen space are rasterized and rendered. That is, according to formulas (4) and (5), the normals and material attributes of the two-dimensional Gaussian primitives projected into the screen space are accumulated from front to back, thereby obtaining a two-dimensional normal map and material attribute map. The difference from the forward shading stage is that in formula (4), the forward shading stage accumulates color values, while the deferred shading stage accumulates normals and material attributes.

[0088] According to the two-dimensional normal map and material attribute map, the normal and material attribute of each pixel on the camera plane are obtained. Then, according to formula (3), the color value of each pixel on the camera plane is calculated using the physically based rendering function to generate the second rendered image.

[0089] Step 4: Design of loss function and parameter optimization for training. A pixel-by-pixel comparison is performed between the first rendered image and a real image in the training set that has the same perspective as the first rendered image, or a pixel-by-pixel comparison is performed between the second rendered image and a real image in the training set that has the same perspective as the second rendered image. This constructs a total loss function that includes rendering error and a regularization term. This function is used to guide the joint optimization of the geometric and appearance parameters (including position, orientation, shape, transparency, and material properties) of the two-dimensional Gaussian. A backpropagation algorithm is used during training to iteratively update the parameters of the two-dimensional Gaussian by minimizing the loss function until the preset iteration termination condition is met.

[0090] The total loss function including rendering error and regularization term used in the training process includes RGB reconstruction loss Depth distortion loss Normal consistency loss and diffusion prior normal supervision loss and material supervision loss

[0091] RGB reconstruction loss Consistent with 3DGS, the absolute error loss and D-SSIM loss Two components:

[0092]

[0093] Among them, the hyperparameter λ=0.2.

[0094] Absolute error loss Measuring the difference in pixel-level color values between the rendered image and the real image, absolute error loss Calculated by the following formula:

[0095]

[0096] in, is the color value of the i-th pixel in the rendered image, I i is the color value of the i-th pixel in the real image, and N is the total number of pixels.

[0097] D-SSIM is based on the structural similarity index (SSIM), which makes up for the deficiency of pixel-level error by quantifying the difference of local structure of the image. Calculated by the following formula:

[0098]

[0099] The depth distortion term minimizes the distance between the ray and the disk intersection, so that the weight is concentrated along the ray, solving the problem of 3DGS rendering ignoring the distance between the intersecting Gaussian primitives, and the depth distortion loss Calculated by the following formula:

[0100]

[0101] in, is the mixing weight of the kth intersection, ω l is the mixing weight of the lth intersection, z k is the depth of the k-th intersection, z l is the depth at the l-th intersection, k and l represent the indices of different Gaussian primitives.

[0102] The normal consistency term ensures that the 2DGS is locally aligned with the actual surface, and reduces the surface normal estimation error and normal consistency loss by aligning the normal with the depth map gradient. Calculated by the following formula:

[0103]

[0104] Among them, n k is the local 2D Gaussian normal, N d is the depth normal, which is calculated by the cross product of horizontal and vertical finite differences of adjacent points after back-projecting the depth map onto the point cloud.

[0105] During the training process, the normal and material priors generated by the diffusion model (Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations, IDArb) are added as pseudo labels, and the absolute error loss is used to calculate the normal and material priors. And cosine loss to supervise the rendered normal map, diffusion prior normal supervision loss Calculated by the following formula:

[0106]

[0107] in, represents the rendered normal map, and N represents the normal prior pseudo label.

[0108] Introducing an absolute error loss between the rendered texture map and the pseudo labels Regularization term, material supervision loss Calculated by the following formula:

[0109]

[0110] in, represents the rendered albedo, metalness and roughness map, and A, M, R are the prior pseudo labels of the rendered albedo, metalness and roughness map.

[0111] The total loss function of the training process is:

[0112]

[0113] Among them, λ d ,λ n ,λ np ,λ mp are the first, second, third, and fourth hyperparameters, is the total loss function.

[0114] Step 5: Downstream Task Support. After training is complete, the resulting 2D Gaussian representation of the 3D scene can be used to adjust the camera position or material and lighting parameters and re-render the image. This allows for inverse rendering of various reflective objects, such as new perspective synthesis, material editing, and relighting.

[0115] Step 6: Test and Evaluation. Synthesize the new perspective on the test set to obtain a rendered image. The specific process is as follows: Obtain the corresponding camera parameters based on the test set, calculate the observation perspective, and then use the trained 2D Gaussian primitives to rasterize and render the images of the reflective object at multiple perspectives within the observation perspective. This results in a rendered image at the new perspective, achieving the rendering goal.

[0116] The real images in the test set are compared with the rendered images under the new perspective for evaluation. The evaluation indicators include PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index) and LPIPS (Learned Perceptual Image Patch Similarity).

[0117] The present invention is tested in multiple scenes of two reflective object datasets, ShinyBlender and GlossySynthetic, and compared with representative models for reconstruction and inverse rendering based on 3DGS, including 3DGS, 2DGS, GaussianShader and Relightable Gaussian (R3DG).

[0118] This paper uses PSNR, SSIM, and LPIPS as evaluation metrics. Tables 1 and 2 show the test results of the novel perspective synthesis performance of the present invention and comparison methods on the ShinyBlender and GlossySynthetic datasets, respectively. The proposed method achieves leading performance in most scenarios, with particular strength in the GlossySynthetic dataset, which is dominated by highly reflective surfaces. Compared to 3DGS and GaussianShader, the proposed method more accurately captures highlight variations and effectively suppresses noise, ensuring that the generated image maintains reflective effects while exhibiting higher visual quality.

[0119] Table 1 Comparison of new perspective synthesis results under ShinyBlender dataset

[0120]

[0121]

[0122] Table 2 Comparison of new perspective synthesis results under the GlossySynthetic dataset

[0123]

[0124]

[0125] Figure 2 (a)-(i) further presents the comparison of the rendering results of the existing reflective object inverse rendering method GShader and the method proposed in this invention. Figure 2 As can be seen from the figure, GShader faces great difficulties in dealing with high-frequency and perspective-dependent reflection effects. In many scenes, GShader's rendering is significantly blurred. For example, in the "bell" and "tbell" scenes shown in Figure (b) and Figure (e), the reflection of the metal surface of the object is blurred. For example, in the "car" scene shown in Figure (h), the reflection on the non-metallic car window is also extremely blurred, and the boundary is obviously missing. In sharp contrast, in Figure 2 In (c), (f), and (i), the proposed method successfully overcomes these challenges. Whether it's metal or non-metal, the proposed method demonstrates excellent rendering performance, accurately capturing the subtle features of reflective surfaces and achieving high-quality rendering results.

[0126] Figure 3(a)-(i) show the new perspective synthesis and inverse rendering effects of some reflective scenes in the test set, which include rendered images, diffuse color images, specular color images, normal images, material images (specifically including albedo, roughness, and metalness) and restored environment maps. These visualization results can intuitively and comprehensively reflect the capabilities of the present invention in terms of geometric reconstruction, material decomposition, and ambient light estimation. Figure 3 From the rendered image in (b), the method proposed in this invention can accurately capture the reflection characteristics under complex lighting conditions, so that the synthesized rendered image highly restores the real scene. Figure 3 As shown in (e), the method proposed in this invention can not only accurately restore the continuous normal distribution of smooth surfaces, maintain stable normal estimation in strong reflection areas, but also effectively reconstruct the detailed features of complex geometric structures. Figure 3 The material decomposition results of (b), (c) and (f)-(h) show that the method proposed in this invention can reasonably distinguish the diffuse reflection and specular reflection components and accurately restore the physical properties such as albedo, roughness and metalness. At the same time, the method proposed in this invention can also restore the physical properties such as the following in the case of strong specular reflection. Figure 3 A clear and accurate environment map like in (i).

[0127] This paper constructs a 3D scene representation based on a 2D Gaussian, using flattened 2D Gaussian primitives to enhance multi-view consistency and geometric reconstruction accuracy. A two-stage hybrid shading strategy is employed: in the first stage, forward shading optimizes geometric primitives, followed by deferred shading in the second stage to more accurately model specular reflections and improve reflection modeling accuracy. A pre-trained diffusion model (IDArb) is constructed to generate pseudo-labels for normals and materials to mitigate lighting-material ambiguity, guide geometric optimization and material decomposition, and thus improve the reconstruction quality of reflective surfaces. A total loss function is used to achieve high-quality reconstruction of geometry, materials, and lighting. This method achieves leading performance on multiple reflective object datasets, enabling efficient and realistic inverse rendering of reflections. It can be widely applied to downstream tasks such as 3D reconstruction, material editing, and relighting.

[0128] See also Figure 4 In another embodiment of the present invention, a reflective object inverse rendering system based on two-dimensional Gaussian sputtering and multimodal diffusion prior is provided, comprising:

[0129] Image and parameter acquisition module, used to obtain images and camera parameters of reflective objects at multiple viewing angles;

[0130] An observation angle calculation module, used to calculate the observation angle according to camera parameters;

[0131] A rasterization rendering module is used to perform rasterization rendering on the images of the reflective object at multiple viewing angles using the trained two-dimensional Gaussian primitives under the observation angle to obtain a rendered image at a new viewing angle;

[0132] The trained two-dimensional Gaussian primitive is obtained through the following process:

[0133] The dense point cloud is restored by the motion structure recovery method using the images and camera parameters of the reflective object at multiple perspectives; a two-dimensional Gaussian primitive is initialized at each point cloud position, and then the initialized two-dimensional Gaussian primitive is trained, and the position, orientation, shape and appearance attributes of the two-dimensional Gaussian primitive are continuously optimized and adjusted, so that the two-dimensional Gaussian primitive gradually fits the real three-dimensional scene until the preset training stop condition is reached, and the trained two-dimensional Gaussian primitive is obtained.

[0134] In one embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for evaluating the multi-level safety performance of a battery is implemented.

[0135] In one embodiment of the present invention, a computer-readable storage medium is provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium herein may include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. In addition, the storage space also stores one or more instructions suitable for being loaded and executed by a processor. These instructions may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium herein may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage. The processor may load and execute one or more instructions stored in the computer-readable storage medium to implement the multi-level safety performance level evaluation method for batteries in the above-mentioned embodiment.

[0136] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

[0141] The above description is merely a description of the preferred embodiment of the present invention and is not to be construed as limiting the claims. The present invention is not limited to the above embodiment, and variations in the specific structure are permitted. Any variations made within the scope of the independent claims of the present invention are also within the scope of protection of the present invention.

[0142] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

Claims

1. A method for inverse rendering of reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior, characterized in that: The following steps are involved: Obtain images and camera parameters of reflective objects at multiple viewing angles; The observation angle is calculated based on the camera parameters; Under the observation perspective, the trained two-dimensional Gaussian primitives are used to rasterize and render the images of the reflective object under multiple perspectives to obtain the rendered images under the new perspective; The trained two-dimensional Gaussian primitive is obtained through the following process: The dense point cloud is restored by the motion structure recovery method using the images and camera parameters of the reflective object at multiple perspectives; a two-dimensional Gaussian primitive is initialized at each point cloud position, and then the initialized two-dimensional Gaussian primitive is trained, and the position, orientation, shape and appearance attributes of the two-dimensional Gaussian primitive are continuously optimized and adjusted, so that the two-dimensional Gaussian primitive gradually fits the real three-dimensional scene until the preset training stop condition is reached, and the trained two-dimensional Gaussian primitive is obtained.

2. The inverse rendering method for reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior according to claim 1, characterized in that: Initializing a two-dimensional Gaussian primitive at each point cloud position is performed using the following formula: P(u,v)=p k +s u t u u+s v t v v=H(u,v,1,1) (1) Where P(u,v) is the mathematical expression of the two-dimensional Gaussian basis element under the local tangent plane, (u,v) is the local tangent plane, u and v are the coordinates of the first and second tangent planes respectively, is the homogeneous transformation matrix representing the two-dimensional Gaussian geometric structure, p k represents the position of the two-dimensional Gaussian basis element in three-dimensional space, s u is a scalar that controls the variance of the first vector direction, t u is the first vector, u is the first tangent plane coordinate, s v is a scalar that controls the variance of the first vector direction, t v is the second vector, v is the second tangent plane coordinate, and H represents the homogeneous transformation matrix of the two-dimensional Gaussian geometric structure.

3. The inverse rendering method for reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior according to claim 1, characterized in that: The preset training stop condition is to train N times, where Then, step 1) is used to obtain the first rendered image. Step 2) is performed again to obtain a second rendered image; Constructing a total loss function including a rendering error and a regularization term according to the first rendered image or the second rendered image; Step 1): Calculate the color value of each two-dimensional Gaussian basis element in the three-dimensional scene under the two-dimensional Gaussian representation through a physically based rendering function; Map each 2D Gaussian basis element to the image plane; Accumulating color values of the two-dimensional Gaussian primitives mapped to the image plane in a front-to-back order using an Alpha blending method to obtain a first rendered image; Step 2): Map each 2D Gaussian basis element to the image plane; Performing raster rendering on the normals and material properties of the two-dimensional Gaussian primitives projected onto the image plane, and accumulating the normals and material properties of the two-dimensional Gaussian primitives projected onto the image plane from front to back to obtain a two-dimensional normal map and material property map; According to the two-dimensional normal map and material attribute map, the normal and material attribute of each pixel of the camera plane are obtained, and the color value of each pixel of the camera plane is calculated using a physically based rendering function to generate a second rendered image.

4. The inverse rendering method for reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior according to claim 1 or 3, characterized in that: The total loss function is: Among them, λ d ,λ n ,λ np ,λ mp are the first, second, third, and fourth hyperparameters, is the total loss function, is the RGB reconstruction loss, is the depth distortion loss, is the normal consistency loss, is the normal supervision loss of the diffusion prior, Monitor loss for materials.

5. The inverse rendering method for reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior according to claim 1, characterized in that: RGB reconstruction loss Calculated by the following formula: Among them, the hyperparameter λ=0.2, is the absolute error loss, is the D-SSIM loss; Depth distortion loss Calculated by the following formula: Among them, ω k is the mixing weight of the kth intersection, ω l is the mixing weight of the lth intersection, z k is the depth of the k-th intersection, z l is the depth at the l-th intersection, k and l represent the indices of different Gaussian basis points; Normal consistency loss Calculated by the following formula: Among them, n k is the local 2D Gaussian normal, N d is the depth normal; Normal Supervision Loss with Diffusion Prior Calculated by the following formula: in, represents the rendered normal map, and N represents the normal prior pseudo label.

6. The inverse rendering method for reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior according to claim 1, characterized in that: The physically based rendering function is: c(ω o )=a(1-m)L diffuse +M specular L specular +c r (oh r ) (3) where c(ω o ) is the color observed from the viewing angle of the two-dimensional Gaussian basis element, a is the albedo, m is the metalness, L diffuse and L specular are the integrals of diffuse and specular lighting, M specular is the integral of BRDF bidirectional reflectance distribution function and pure white ambient light, c r (ω r ) is the residual term of reflected indirect light, ω r is the reflection direction.

7. The inverse rendering method for reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior according to claim 1, characterized in that: The color value c(x) of each pixel of the first rendered image is: Where K is the total number of Gaussian primitives, k is the index of Gaussian primitives, and c k represents the color of the k-th Gaussian primitive, w k is the weight; Weight w k Calculated by the following formula: where α k Indicates the transparency of the k-th Gaussian primitive, j is the index of the Gaussian primitive before the k-th Gaussian primitive, α j Indicates the transparency of the j-th Gaussian primitive.

8. A reflective object inverse rendering system based on two-dimensional Gaussian sputtering and multimodal diffusion prior, characterized in that: include: Image and parameter acquisition module, used to obtain images and camera parameters of reflective objects at multiple viewing angles; An observation angle calculation module, used to calculate the observation angle according to camera parameters; A rasterization rendering module is used to perform rasterization rendering on the images of the reflective object at multiple viewing angles using the trained two-dimensional Gaussian primitives under the observation angle to obtain a rendered image at a new viewing angle; The trained two-dimensional Gaussian primitive is obtained through the following process: The dense point cloud is restored by the motion structure recovery method using the images and camera parameters of the reflective object at multiple perspectives; a two-dimensional Gaussian primitive is initialized at each point cloud position, and then the initialized two-dimensional Gaussian primitive is trained, and the position, orientation, shape and appearance attributes of the two-dimensional Gaussian primitive are continuously optimized and adjusted, so that the two-dimensional Gaussian primitive gradually fits the real three-dimensional scene until the preset training stop condition is reached, and the trained two-dimensional Gaussian primitive is obtained.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the inverse rendering method for reflective objects based on two-dimensional Gaussian sputtering and multimodal diffusion prior is implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the inverse rendering method of a reflective object based on two-dimensional Gaussian sputtering and multimodal diffusion prior is implemented as described in any one of claims 1 to 7.

Images