Inverse rendering attribute decomposition method based on few view angle input

Abstract

The invention relates to the field of computer vision and graphics, in particular to a multi-view image inverse rendering decomposition method under a limited view condition. According to the method, under the condition that only a small number of different view angle images need to be input, inverse rendering decomposition can be carried out on the appearance of the target object, and editable rendering attributes including albedo, roughness, metal degree, illumination and the like are obtained. Compared with the prior art, the method has the advantages that the requirement for the number of multi-view collection is lowered, effective decomposition of materials and illumination can still be achieved in the scene with insufficient data, the decomposition result can be directly used for downstream tasks such as editing, re-lighting and re-rendering, and therefore the rendering flexibility and the actual application value are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and computer graphics, and particularly relates to a multi-view image inverse rendering decomposition method under a condition of a small number of view angles. BACKGROUND

[0002] The part provided in this section is merely background information related to the present disclosure, which does not necessarily have to be prior art.

[0003] With the development of computer vision and computer graphics, inverse rendering gradually becomes a key technology in the fields of virtual reality, augmented reality, film special effects, digital asset production, cultural relic digitization and industrial detection, and its goal is to decompose physical properties such as albedo, roughness, metallicity, normal and illumination from image data, thereby providing a basis for re-rendering, relighting and material editing. However, the existing multi-view image inverse rendering method usually relies on a large number of view angle inputs, and often needs dozens to hundreds of images to ensure the stability of geometry and material decomposition. For example, the method based on multi-view stereo reconstruction (MVS) needs a high-density camera array to collect data to reconstruct geometry and separate material properties, and the inverse rendering method based on neural radiation field (NeRF) also needs large-scale multi-angle image input to obtain more accurate results through volume rendering optimization. However, in many practical application scenarios, it is difficult to obtain a large number of view angle images. For example, in the protection of cultural relics, limited by the collection conditions, only a limited number of angle images can be obtained. In the production of film special effects, due to cost and shooting cycle considerations, it is difficult to arrange enough cameras. In the industrial detection scene, it is often limited by space and hardware conditions and can only be collected from a small number of view angles. These limitations make it difficult for the existing inverse rendering method relying on a large amount of data to be directly applicable. At the same time, when the input view angle is insufficient, the existing method often faces the problem of incomplete reconstruction of geometric structure, which further affects the decomposition accuracy of material and illumination, and due to the lack of sufficient multi-view constraints, the material properties such as albedo, roughness and metallicity are easy to be mixed with illumination information, thereby making the physical consistency of the decomposition result insufficient, and it is difficult to guarantee the realism of subsequent re-rendering and editing. Therefore, the existing technology urgently needs an inverse rendering method that can still achieve physical consistent and high-precision decomposition under the condition of a small number of view angle inputs, so as to reduce the data collection requirements and improve the applicability and practical value of the inverse rendering method. SUMMARY

[0004] The present application aims at the problem that the existing multi-view image inverse rendering method relies on a large number of view angle inputs and has insufficient decomposition accuracy under the condition of a small number of view angles, and proposes a multi-view image inverse rendering decomposition method based on a small number of view angle inputs.

[0005] A method for inverse rendering attribute decomposition based on a small number of view angle inputs, comprising the following steps:

[0006] Step 1: Obtain input images of different perspectives of the target object, and obtain the camera intrinsic and extrinsic parameters corresponding to each perspective.

[0007] Step 2: Model the spatial geometric structure of the target object using a two-dimensional Gaussian representation as the geometric carrier of inverse rendering decomposition, and add material properties to the two-dimensional Gaussian representation.

[0008] Step 3: Project and weight the two-dimensional Gaussian representation obtained in step 2 at a given perspective using the Gaussian splatting algorithm, and fuse the material properties of each two-dimensional Gaussian distribution to the two-dimensional image plane to obtain a set of two-dimensional attribute maps.

[0009] Step 4: Input the two-dimensional attribute maps obtained in step 3 and external lighting parameters into a bidirectional reflectance distribution function based on a microfacet model, calculate the diffuse reflection component and the specular reflection component respectively, and synthesize a physically consistent rendering image.

[0010] Step 5: Use the multi-perspective images obtained in step 1 as the true reference, use the synthesized image obtained in step 4 as the prediction result, and construct a loss function based on the difference between the synthesized image and the input image. Perform iterative optimization on the material properties of the two-dimensional Gaussian representation through backpropagation until the model converges.

[0011] In some embodiments, step 1 includes the following specific steps:

[0012] Step 1-1: Obtain target object images at different perspectives, which cover the target object comprehensively, and the number of images is generally less than 20;

[0013] Step 1-2: Obtain the camera intrinsic and extrinsic parameters corresponding to each perspective image, which are used to provide geometric consistency constraints for subsequent steps; the intrinsic parameters represent the focal length, principal point position and distortion coefficient of the camera, and the extrinsic parameters represent the position and attitude of the camera in the world coordinate system.

[0014] In some embodiments, step 2 specifically includes the following steps:

[0015] Step 2-1: Model the spatial geometric features of the target object in the form of a series of two-dimensional Gaussian distributions using a two-dimensional Gaussian representation as the geometric carrier of inverse rendering decomposition; the spatial features are the geometric information of the original scene, including spatial position, which is specifically the spatial position and shape parameter (variance of Gaussian) of the target object on the two-dimensional Gaussian.

[0016] Step 2-2: Add the material properties of the target object to each two-dimensional Gaussian distribution using the gaussian splatting method, including albedo, roughness, metallicity and normal.

[0017] Each of the two-dimensional Gaussian representation includes a center position parameter and a variance matrix for describing the geometric distribution, to model the position and shape distribution of the target object surface in space; includes color, transparency and albedo parameters for characterizing the surface appearance, to depict the inherent color and material characteristics of the object under non-illumination conditions; includes normal direction, roughness and metallicity parameters for modeling the surface illumination response, to model the reflection characteristics of the object surface when interacting with different light sources.

[0018] The center position and the variance matrix are used to depict the geometric shape of the object surface, the color, transparency and albedo parameters can represent the inherent material information of the object, and the normal direction, roughness and metallicity parameters can be combined with the illumination model, so that the joint modeling of the geometric structure and the material attribute is realized in the inverse rendering decomposition process, and the decomposition result not only has geometric consistency but also can truly reflect the material performance under the action of illumination.

[0019] In some embodiments, step 3 specifically includes the following steps:

[0020] Step 3-1: Based on the given view angle, the two-dimensional Gaussian representation in three-dimensional space is projected onto the two-dimensional image plane using the Gaussian splatting algorithm;

[0021] Step 3-2: The material attributes of multiple two-dimensional Gaussian distributions that exist on the image plane after projection are weighted, fused and accumulated;

[0022] Step 3-3: A set of two-dimensional attribute maps consistent with the size of the input image are generated, including normal map, albedo map, roughness map and metallicity map.

[0023] Each attribute map corresponds to the geometric direction distribution, inherent color information, surface roughness characteristics and metallicity distribution of the target object under the view angle; the attribute map not only maintains the geometric consistency of the input view angle and the target object, but also provides fine sub-attribute input for the physical-based rendering step, so as to realize high-fidelity reproduction of the object appearance and accurate simulation of the lighting effect in the subsequent rendering process.

[0024] In some embodiments, step 4 specifically includes the following steps:

[0025] Step 4-1: The two-dimensional attribute maps generated in step 3 are used as input micro-surface model together with the externally defined illumination parameters, and the bidirectional reflectance distribution function is used for pixel-by-pixel rendering calculation, wherein:

[0026] The diffuse reflection component Ld is calculated by the following formula:

[0027]

[0028] wherein m refers to metallic, a refers to albedo;

[0029] The specular reflection component Ls is calculated by the following formula:

[0030]

[0031] wherein n represents a surface normal, l represents an illumination vector of a scene light source to a target point, v represents a view line vector of an eye to the target point, D, F, and G are normal distribution function, Fresnel reflection function, and geometric occlusion function, respectively;

[0032] Step 4-2: Add the calculated diffuse reflection component and the specular reflection component to synthesize an image containing complete lighting interaction effects at a specified viewing angle.

[0033] By adding the diffuse reflection component and the specular reflection component, a final rendering result containing complete lighting interaction effects at a specified viewing angle can be obtained, thereby truly restoring the appearance performance of an object surface under multiple lighting conditions in the inverse rendering decomposition task, and ensuring that the generated result is both physically consistent and visually realistic, and can be directly used in downstream application scenarios such as re-rendering, relighting, and virtual reality, digital asset production, etc.

[0034] In some embodiments, step 5 specifically includes:

[0035] Step 5-1: Take the input multi-view image as real data, and take the image synthesized in step 4 as a prediction result, and calculate the mean square error between the two as an optimization objective function;

[0036] Step 5-2: Through a back propagation mechanism, iteratively update the two-dimensional Gaussian representation of the material properties, including albedo, roughness, metallic, and normal, using the optimization objective function, so that the synthesized image can maintain high consistency with the input real image at different viewing angles, and gradually converge to an appearance performance consistent with physical reality under multi-view constraints, and finally achieve high-precision recovery of inverse rendering decomposition results such as albedo, roughness, metallic, and lighting of the target object.

[0037] In some embodiments, in step 5-2: Take external lighting parameters as an optimizable variable, and jointly iteratively update the material properties in the same optimization framework, thereby realizing joint optimization of material properties and lighting parameters.

[0038] Step 5-2 specifically includes:

[0039] Step 5-2-1: Render the properties carried by the current two-dimensional Gaussian representation into a two-dimensional attribute map through Gaussian sputtering technology, and synthesize an RGB prediction image through physically-based rendering calculation of the two-dimensional attribute map;

[0040] Step 5-2-2: Calculate the mean square error loss between the RGB predicted image and the real RGB image of the corresponding view in the dataset;

[0041] Step 5-2-3: Calculate the gradient of the mean square error loss with respect to each attribute of the two-dimensional Gaussian representation by backpropagation algorithm;

[0042] Step 5-2-4: Iteratively update the attributes of the two-dimensional Gaussian representation by Adam optimizer according to the gradient.

[0043] The present application first acquires less than 20 multi-view input images and their corresponding camera parameters. A two-dimensional Gaussian representation (2D Gaussian, 2DGS) is used as a geometric carrier, and material-related attributes such as albedo, roughness, metalness, and normal are added to the two-dimensional Gaussian to model the geometric shape and surface characteristics of the object simultaneously. By Gaussian Splatting algorithm, the two-dimensional Gaussian is projected and accumulated under the given view to generate two-dimensional attribute maps such as normal map, albedo map, roughness map, and metalness map consistent with the size of the input image. The two-dimensional attribute maps and lighting parameters are input into the physically-based rendering model, and the rendering calculation is performed through the bidirectional reflectance distribution function (BRDF) of the micro-surface model to obtain the synthesized image containing diffuse and specular reflection components. Finally, the synthesized image is compared with the input real multi-view image, an optimization objective function based on mean square error is constructed, and the material attributes of the two-dimensional Gaussian and the learnable lighting parameters are iteratively optimized by backpropagation, so that the decomposition result gradually approximates the appearance of the real object under multi-view constraints.

[0044] Compared with the prior art, the present application can achieve high-precision decomposition of albedo, roughness, metalness, and lighting attributes under the condition of a small number of views while significantly reducing the number of multi-view image acquisition requirements.

[0045] The method combines image processing, three-dimensional reconstruction, and physically-based rendering (PBR) and other related technologies, aiming to solve the technical problem of accurately decomposing the appearance of the target object under the condition of limited number of view inputs. The present application realizes the decomposition and reconstruction of albedo, roughness, metalness, and normal and other material attributes and lighting parameters by introducing two-dimensional Gaussian representation (2D Gaussian) as a geometric carrier and combining Gaussian Splatting rendering algorithm and physically-based rendering model. This method not only reduces the number of multi-view image acquisition requirements, but also maintains the physical consistency and visual fidelity of inverse rendering decomposition in the scene with limited input views. BRIEF DESCRIPTION OF DRAWINGS

[0046] The above and / or other aspects of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings, in which:

[0047] Figure 1 is a flowchart of the method described in the embodiments of the present application;

[0048] Figure 2 is an input multi-view image;

[0049] Figure 3 is a normal map decomposed by inverse rendering;

[0050] Figure 4 is a roughness map decomposed by inverse rendering;

[0051] Figure 5 is an albedo map decomposed by inverse rendering;

[0052] Figure 6 is a result map based on physical rendering; DETAILED DESCRIPTION

[0053] The overall flowchart of the present scheme is shown in Figure 1 , specifically:

[0054] Step 1: Input image acquisition. A small number of different view input images of the target object are acquired, preferably less than twenty different views, and the camera intrinsic and extrinsic parameters corresponding to each view are also acquired, wherein the intrinsic parameters are used to represent the focal length, principal point position and distortion coefficient of the camera, and the extrinsic parameters are used to represent the position and attitude of the camera in the world coordinate system. The original multi-view pictures are acquired by camera shooting, and after the multi-view pictures are acquired, the camera intrinsic and extrinsic parameters of each picture are calculated by using the colmap algorithm, and an example of a multi-view image is shown in Figure 2 .

[0055] Step 2: Two-dimensional Gaussian representation modeling. The two-dimensional Gaussian representation (2D Gaussian, 2DGS) is used as the geometric carrier of inverse rendering decomposition, the spatial geometric features of the target object are modeled in the form of a series of two-dimensional Gaussian distributions, and the material-related properties including albedo, roughness, metallic and normal are attached to each two-dimensional Gaussian distribution.

[0056] Step 3: Gaussian Splatting rendering. Project and weight accumulate the above two-dimensional Gaussian representation at a given view angle using Gaussian Splatting algorithm, fuse the material properties of each two-dimensional Gaussian distribution to the two-dimensional image plane, the attribute map includes normal map, albedo map, roughness map and metallicity map, the example is plastic, the metallicity is decomposed to 0, and the results of the remaining attribute maps are as shown in Figures 3-5 .

[0057] Step 4: Physically-based rendering calculation. Input the two-dimensional attribute map and external lighting parameters into the bidirectional reflectance distribution function (BRDF) based on the micro-surface model for rendering calculation, wherein the diffuse reflection part is represented by , which is used to describe the contribution of albedo to the diffuse reflection component under different metallicity; the specular reflection part is represented by , wherein D, F, G are normal distribution function, Fresnel reflection function and geometric occlusion function, respectively, which are used to model the specular reflection component. By adding the diffuse reflection component and the specular reflection component, a composite image containing complete lighting interaction effects is obtained, and the example rendering result is as shown in Figure 6 .

[0058] Step 5: Optimization and iterative update. Take the input multi-view image as the ground truth and the composite image as the prediction result, construct an optimization objective function based on the mean square error between the two, and iteratively optimize the attributes of the two-dimensional Gaussian representation through the back propagation mechanism, including albedo, roughness, metallicity and normal; At the same time, maintain the optimizable lighting parameters and update them together with the material properties, so as to gradually approach the appearance of the real image under the constraint of multi-view, realize the physically consistent inverse rendering decomposition of the target object.

[0059] The present application provides a kind of based on the thought and method of multi-view image inverse rendering decomposition method of small amount of view input, the method and approach for specifically realizing this technical solution are many, above-mentioned only preferred embodiment of the present application, it should be pointed out, for the ordinary skill in the art person of this technical field, under the premise of not departing from the principle of the present application, can also make several improvements and refinements, these improvements and refinements also should be regarded as the protection scope of the present application. The components not explicitly described in the embodiment can be realized by existing technology.

Claims

1. A method for inverse rendering attribute decomposition based on a small number of viewpoint inputs, characterized in that, Includes the following steps: Step 1: Obtain input images of the target object from different perspectives, and obtain the camera intrinsic and extrinsic parameters for each perspective; Step 2: Using the two-dimensional Gaussian representation as the geometric carrier for inverse rendering decomposition, model the spatial geometry of the target object and attach material properties to the two-dimensional Gaussian representation. Step 3: Project and weighted accumulate the two-dimensional Gaussian representation obtained in Step 2 from a given viewpoint using the Gaussian sputtering algorithm, and fuse the material properties of each two-dimensional Gaussian distribution into a two-dimensional image plane to obtain a set of two-dimensional property maps. Step 4: Input the two-dimensional attribute map obtained in Step 3 and the external lighting parameters into the bidirectional reflection distribution function based on the micro-surface model, calculate the diffuse reflection component and the specular reflection component respectively, and synthesize a rendering image with physical consistency. Step 5: Using the multi-view image obtained in Step 1 as the real reference and the synthetic image obtained in Step 4 as the prediction result, construct a loss function based on the difference between the synthetic image and the input image, and iteratively optimize the material properties represented by the two-dimensional Gaussian through backpropagation until the model converges.

2. The inverse rendering attribute decomposition method based on a small number of viewpoint inputs according to claim 1, characterized in that, Step 1 includes the following specific steps: Step 1-1: Acquire images of the target object from different perspectives, ensuring that the images fully cover the target object; Step 1-2: Obtain the camera intrinsic and extrinsic parameters corresponding to each viewpoint image to provide geometric consistency constraints for subsequent steps; the intrinsic parameters represent the camera's focal length, principal point position, and distortion coefficient, and the extrinsic parameters represent the camera's position and orientation in the world coordinate system.

3. The inverse rendering attribute decomposition method based on a small number of viewpoint inputs according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2-1: Using two-dimensional Gaussian representation as the geometric carrier for inverse rendering decomposition, the spatial geometric features of the target object are modeled in the form of a series of two-dimensional Gaussian distributions. Step 2-2: Attach the material properties of the target object to each two-dimensional Gaussian distribution. The material properties include albedo, roughness, metallicity, and normal.

4. The inverse rendering attribute decomposition method based on a small number of viewpoint inputs according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3-1: Based on the given viewpoint, use the Gaussian sputtering algorithm to project the two-dimensional Gaussian representation in three-dimensional space onto the two-dimensional image plane; Step 3-2: Perform weighted fusion and accumulation on the material properties of multiple overlapping two-dimensional Gaussian distributions projected onto the image plane; Step 3-3: Generate a set of two-dimensional attribute maps with the same size as the input image. The two-dimensional attribute maps include a normal map, an albedo map, a roughness map, and a metallicity map.

5. The inverse rendering attribute decomposition method based on a small number of viewpoint inputs according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4-1: Use the two-dimensional attribute map generated in step 3 and the externally defined lighting parameters as input micro-surface models, and perform pixel-by-pixel rendering calculations using the bidirectional reflection distribution function; Step 4-2: Add the calculated diffuse reflection component and specular reflection component to synthesize an image containing complete lighting interaction effects at a specified viewpoint.

6. The inverse rendering attribute decomposition method based on a small number of viewpoint inputs according to claim 1, characterized in that, Step 5 specifically includes: Step 5-1: Using the input multi-view images as real data and the image synthesized in Step 4 as the prediction result, calculate the mean square error between the two as the optimization objective function; Step 5-2: Through the backpropagation mechanism, the material properties represented by the two-dimensional Gaussian are iteratively updated using the optimization objective function. The material properties include albedo, roughness, metallicity, and normal.

7. The inverse rendering attribute decomposition method based on a small number of viewpoint inputs according to claim 6, characterized in that, In step 5-2: the external lighting parameters are treated as optimizable variables and jointly iterated and updated with the material properties within the same optimization framework.

8. The inverse rendering attribute decomposition method based on a small number of viewpoint inputs according to claim 6 or 7, characterized in that, Step 5-2 specifically involves: Step 5-2-1: Render the attributes carried by the current two-dimensional Gaussian representation into a two-dimensional attribute map using Gaussian sputtering technology, and synthesize the two-dimensional attribute map into an RGB prediction image through physically based rendering calculation; Step 5-2-2: Calculate the mean squared error loss between the predicted RGB image and the corresponding real RGB image in the dataset; Step 5-2-3: Backpropagate the mean squared error loss using the backpropagation algorithm and calculate the gradient of the mean squared error loss relative to each attribute in the two-dimensional Gaussian representation; Step 5-2-4: Iteratively update the properties represented by the two-dimensional Gaussian using the Adam optimizer based on the gradient.

Images