Micro-inverse rendering method based on multi-view image

Abstract

The invention provides a multi-view image-based micro-inverse rendering method, which comprises the following steps of: S1, acquiring a group of multi-view RGB (Red, Green, Blue) shot images shot for a three-dimensional object under a fixed illumination condition, acquiring initial camera parameters of the group of images and an initial geometric structure of the object based on an explicit three-dimensional reconstruction technology, and initializing a rendering parameter set, a plurality of directional light sources and a point light source are initially set in the illumination parameters; s2, constructing a differentiable rendering function which is configured to render and output a multi-view-angle linear image by taking the geometric structure, the rendering parameter set and the view angles as input, and converting the multi-view-angle linear image into a multi-view-angle RGB reconstructed image; and S3, the geometric structure, the material parameters and the illumination parameters are optimized to gradually reduce the difference between the multi-view RGB reconstructed image and the multi-view RGB shot image, and the optimized illumination parameters comprise adaptive illumination control, combination of directional light sources with similar radiation contributions, combination of directional light sources with similar irradiation directions and elimination of invalid directional light sources.

Description

Technical Field

[0001] This invention relates to the fields of generative artificial intelligence and machine learning, specifically to the field of image processing technology, and more specifically, to a differentiable inverse rendering method based on multi-view images. Background Technology

[0002] In fields such as 3D reconstruction, augmented reality, and digital twins, obtaining high-fidelity 3D models of real-world objects is crucial. This includes not only their geometry but also their surface material properties and the lighting environment they are in. Traditional methods typically require expensive specialized equipment and highly controlled laboratory conditions (such as light field environments) for data acquisition, making the process cumbersome and unsuitable for everyday scenarios.

[0003] A 3D model can be represented by its geometric structure, material properties, and texture mapping. Material properties determine how light interacts with the object's surface, such as through reflection, scattering, or transmission. Given a 3D model and its lighting environment, the rendering process converts the 3D model into a 2D image. Inverse rendering, on the other hand, is the reverse process; given a series of images, it reconstructs the geometry, materials, and lighting of the 3D model.

[0004] In recent years, image-based reverse rendering technology has become a research hotspot. Its core idea is to construct a differentiable forward rendering pipeline, comparing the rendered result with the real image and using optimization algorithms such as gradient descent to adjust the rendering parameters (materials and lighting) in reverse until they match. However, this method faces a fundamental "pathological" problem: for a given multi-view image, there are countless combinations of materials and lighting that can produce the same rendering effect. Especially regarding lighting, its quantity, type, direction, and intensity are all unknown, making the optimization process prone to getting trapped in local optima or obtaining physically unreasonable and globally inconsistent results.

[0005] There are three main methods for decoupling lighting parameters in existing technologies: first, pre-annotating the light source (mainly its position and type); second, making strong assumptions about the lighting (such as unidirectional lighting or low-order spherical harmonic lighting); and third, using discretized high-dimensional representations (such as environment mapping). The first two methods are difficult to depict realistic and complex lighting environments, while the third method lacks effective regularization constraints, leading to unstable optimization and ambiguity between lighting and materials.

[0006] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0007] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a reverse rendering method.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] According to a first aspect of the present invention, an inverse rendering method is provided, comprising performing the following operations for each object to be inverse rendered: S1, acquiring a set of multi-view RGB images of a three-dimensional object taken under fixed lighting conditions, acquiring initial camera parameters and initial geometric structure of the object based on explicit three-dimensional reconstruction technology, and initializing a rendering parameter set, wherein the rendering parameter set includes material parameters and lighting parameters, and the lighting parameters initially include multiple directional light sources and one point light source; S2, constructing a differentiable rendering function, which is configured to take the geometric structure, the rendering parameter set, and the viewpoint as input, render a multi-view linear image, and convert the multi-view linear image into a multi-view RGB reconstructed image; S3, using a gradient descent algorithm to optimize the geometric structure, material parameters, and lighting parameters to gradually reduce the difference between the multi-view RGB reconstructed image and the multi-view RGB captured image, and obtaining a final optimized result, wherein optimizing the lighting parameters includes adaptive lighting control, which includes merging directional light sources with similar radiation contributions, merging directional light sources with similar illumination directions, and eliminating invalid directional light sources. This scheme achieves at least the following beneficial technical effects: During the reverse rendering process, multiple directional light sources and one point light source are initially set. During optimization, not only are the geometric structure and material parameters optimized, but the number of directional light sources is also adjusted to merge similar directional light sources, reducing the computational load of reverse rendering and eliminating interference from invalid directional light sources. Ultimately, a simplified, necessary, and physically interpretable lighting representation evolves from the overly complete initial light source set, fundamentally alleviating the pathological problem that "different lighting combinations can produce the same image."

[0010] Optionally, the operation of merging directional light sources with similar radiation contributions includes: for each pair of directional light sources, calculating the difference in illumination contribution between the two light sources in the half-range vector direction of all texels in the material map; obtaining designated light source pairs whose contribution difference is less than a radiation similarity threshold; constructing a merged directional light source from the two similar directional light sources in each designated light source pair; wherein the direction of the merged directional light source is set to be consistent with the directional light source with the largest radiance in the corresponding designated light source pair; and the radiance of the merged directional light source is set to the sum of the radiances of the two directional light sources in the corresponding designated light source pair. This scheme can achieve at least the following beneficial technical effects: this scheme can gradually simplify similar directional light sources during the optimization process by merging them, reducing the number of directional light sources and constructing a more realistic and physically interpretable lighting environment.

[0011] Optionally, merging directional light sources with similar illumination directions includes: for each pair of directional light sources, calculating the cosine similarity of the illumination directions of the two light sources in the pair; selecting light source pairs with a cosine similarity greater than a geometric similarity threshold as selected light source pairs; constructing a fused directional light source from the two directional light sources in each selected light source pair; wherein the direction of the fused directional light source is set as a weighted average of the illumination directions of the two directional light sources in the corresponding selected light source pair; and the radiance of the fused directional light source is set as the sum of the radiances of the two directional light sources in the corresponding selected light source pair.

[0012] Optionally, removing invalid directional light sources includes: deleting directional light sources whose radiance norm is less than a radiance threshold; and / or deleting directional light sources with weak radiance influence. For each directional light source, the number of pixels significantly affected by that directional light source in the most recent complete rendering is counted, and directional light sources with a number less than a threshold are considered as having weak radiance influence. This scheme can achieve at least the following beneficial technical effects: It can gradually remove directional light sources with weak radiance intensity and weak radiance influence during the optimization process, thereby further reducing the number of directional light sources and constructing a more realistic and physically interpretable lighting environment.

[0013] Optionally, S3 employs a phased optimization approach for geometry, material parameters, and lighting parameters. Each phase utilizes a loss function specific to that phase to optimize designated items within these parameters. This approach achieves at least the following beneficial technical effects: phased optimization reduces optimization interference and convergence difficulties caused by simultaneous updates to geometry, material, and lighting parameters, thereby improving the final rendering quality.

[0014] Optionally, material parameters include reflectivity, metallicity, and roughness. S3 includes performing one or more rounds of the following operations: First stage: Using rendering loss to guide the update of reflectivity in the material parameters, without updating metallicity, roughness, and lighting parameters; Second stage: Using a weighted sum of rendering loss and geometric loss as the total loss to guide the update of lighting parameters, roughness, and geometry, without updating reflectivity and metallicity. After this stage reaches convergence, adaptive lighting control is performed once; Third stage: Using a weighted sum of rendering loss, geometric loss, material smoothing loss, and physical plausibility loss as the total loss to guide the update of material parameters, lighting parameters, and geometry. The rendering loss is used to limit the distance between the multi-view RGB reconstructed image and the multi-view RGB captured image; the geometric loss is used to limit the distance between the predicted vertex positions of the facets and the labeled vertex positions of the facets, as well as the distance between the rasterized projection of the reconstructed object onto the multi-view RGB reconstructed image plane and the object mask in the multi-view RGB captured image; the material smoothing loss is used to apply isotropic and anisotropic smoothing constraints to the material parameters; and the physical plausibility loss is used to constrain the material parameters to fall within a reasonable numerical range. This scheme achieves at least the following beneficial technical effects: In the first stage, it updates reflectivity using only rendering loss, allowing material details corresponding to reflectivity to diffuse into the triangle texture, while avoiding roughness instability in the next stage due to unreasonable lighting parameter presets; in the second stage, it simultaneously optimizes lighting parameters, roughness, and geometry while reducing metallicity interference; and in the third stage, it optimizes all learnable parameters to improve the overall rendering effect. Thus, through phased training, it ultimately outputs optimized, globally consistent, high-quality geometry, material parameters, and simplified lighting parameters.

[0015] Optionally, material parameters include reflectivity, metallicity, and roughness. S3 includes performing one or more rounds of the following operations: First stage: updating geometry using geometric loss; Second stage: updating reflectivity in material parameters using rendering loss, without updating metallicity, roughness, and lighting parameters; Third stage: updating lighting parameters, roughness, and geometry using a weighted sum of rendering and geometric losses as the total loss, without updating reflectivity and metallicity. Adaptive lighting control is performed once this stage reaches convergence; Fourth stage: using rendering loss, geometric loss, material smoothing loss, and physical losses... The weighted sum of the rationality loss serves as the total loss to guide the updating of material parameters, lighting parameters, and geometry. Among these, the rendering loss is used to limit the distance between the multi-view RGB reconstructed image and the multi-view RGB captured image; the geometry loss is used to limit the distance between the predicted face vertex positions and the labeled face vertex positions, as well as the distance between the rasterized projection of the reconstructed object on the multi-view RGB reconstructed image plane and the object mask in the multi-view RGB captured image; the material smoothing loss is used to apply isotropic and anisotropic smoothing constraints to the material parameters; and the physical rationality loss is used to constrain the material parameters to fall within a reasonable numerical range.

[0016] According to a second aspect of the present invention, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0017] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the method described in the first aspect by executing the executable instructions. Attached Figure Description

[0018] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0019] Figure 1 This is a flowchart illustrating the inverse rendering method according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of the overall network structure corresponding to the inverse rendering method according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of a portion of the human hand model in the human hand synthesis dataset used in an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram comparing a rendered image generated according to an embodiment of the present invention with an original image;

[0023] Figure 5This is a schematic diagram of a material texture according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0025] As mentioned in the background section, existing technologies mainly employ three methods to decouple lighting parameters: pre-annotation of light sources (primarily their location and type), strong assumptions about lighting (such as single directional light or low-order spherical harmonic lighting), and the use of discretized high-dimensional representations (such as environment mapping). The first two methods struggle to depict realistic and complex lighting environments, while the third method lacks effective regularization constraints, leading to unstable optimization and potential ambiguity between lighting and materials. Therefore, a robust optimization framework is urgently needed to handle lighting uncertainties, ensuring that the decoupled material properties and lighting environment possess physical rationality and global consistency. To this end, this invention proposes an inverse rendering method. During the inverse rendering process, multiple directional light sources and one point light source are first set. In the optimization process, not only are the geometric structure and material parameters optimized, but the number of directional light sources is also adjusted to merge similar directional light sources, reducing the computational load of inverse rendering and eliminating interference from invalid directional light sources. Ultimately, a simplified, necessary, and physically interpretable lighting representation evolved from the overly complete initial light source set, fundamentally alleviating the pathological problem that "different lighting combinations can produce the same image".

[0026] Before describing the embodiments of the present invention in detail, some of the terms used therein are explained as follows:

[0027] A linear image, or radiometrically linear image, is an image in which the pixel value is proportional to the physical light energy (such as radiance or illuminance) of the corresponding point in the scene. For example, a High Dynamic Range (HDR) image.

[0028] According to one embodiment of the present invention, the method introduces an optimization strategy based on adaptive illumination control, effectively solving the technical challenge of illumination uncertainty. See also Figure 1 This invention provides a reverse rendering method, including steps S1, S2, and S3. To better understand this invention, each step is described in detail below with reference to specific embodiments.

[0029] Step S1: Obtain a set of multi-view RGB images of a 3D object under fixed lighting conditions. Based on explicit 3D reconstruction technology, obtain the initial camera parameters and the initial geometric structure of the object in the set of images, and initialize the rendering parameter set. The rendering parameter set includes material parameters and lighting parameters. The lighting parameters initially set multiple directional light sources and one point light source.

[0030] According to one embodiment of the present invention, this step mainly involves data preparation and initialization. Under multi-view, fixed lighting conditions, a set of static RGB images (i.e., multi-view RGB captured images) of the object to be modeled (i.e., the object itself, such as a hand, cabinet, or table) are acquired, and the corresponding initial camera parameters and initial geometric structure are obtained (e.g., through traditional multi-view stereoscopic techniques such as colmap). The set of rendering parameters to be optimized is then initialized. It includes:

[0031] Material parameters Used to define a set of physical material properties for each surface sampling point (or texel) on a geometry (3D model), including at least reflectivity. Specify base color), metallicity ( (specifying whether it is metallic) and roughness ( This indicates the roughness of the microfacet. Optionally, the material properties may also include high reflectivity (…). ) and / or normal map ( (etc.) Before training, the initial reflectivity can be... It is set to purely random. However, to reduce physical inconsistencies, the initial reflectivity is... Set to the average HDR value of the patch vertices across all projected images.

[0032] Lighting parameters : There are one directional light source (directional base light source) and one point light source. The direction of the directional base light source is evenly distributed on a unit sphere (simulating ambient lighting), and the point light source is located at the viewing angle (simulating flash during shooting). Each directional light source... Including the direction of light and radiance .

[0033] According to an example of the invention, see Figure 2Taking a human hand as the object to be inversely rendered as an example, 62 RGB photos taken from different angles around the hand are used as a set of multi-view RGB images. This allows for the generation of camera intrinsic and extrinsic parameters (pose, focal length, principal point). For instance, SFM (Structure of Motion Recovery) is used to recover the camera pose (position and viewpoint) and sparse structure of the hand from the 62 photos. MVS (Multi-View Stereo Vision) is used to estimate the depth maps for each viewpoint, thus obtaining a single-viewpoint point cloud. The point clouds from each viewpoint are then fused, and surface reconstruction is performed to generate the model's geometry (vertices, normals). Finally, a series of facets are obtained, which constitute the mesh of the hand model.

[0034] Using 3D modeling software (such as Maya), each facet of the human hand model is mapped to a... On the texture of the parameters, each texel contains three material properties, corresponding to reflectivity, metallicity, and roughness. Three PBR material maps with a resolution of 2048x2048 are initialized. The initial reflectivity of each texel is set to the average HDR value of the vertices of the patch across all projected images, the initial metallicity is set to 0 (i.e., non-metallic material), and the initial roughness is set to 0.5.

[0035] Initialize 20 directions A directional light source (located at the vertices of a regular icosahedron) uniformly distributed on a unit sphere has an initial radiance. Set to 1. The point light source is located at the camera position, with an initial radiance of 1. It is the square of the average distance from the camera to the surface of the human hand model.

[0036] Step S2: Construct a differentiable rendering function, which is configured to take geometry, rendering parameter set and viewpoint as input, render multi-view linear image, and convert multi-view linear image into multi-view RGB reconstructed image.

[0037] According to one embodiment of the present invention, this step mainly involves constructing a differentiable rendering pipeline. Specifically, this is achieved by constructing a differentiable rendering function. This function renders the parameter set. Geometric structure (Including location) and normal ) and perspective Input: Linear image Before calculating the rendering loss, tone mapping techniques can be used to convert the linear image into RGB form, resulting in a multi-view RGB reconstructed image. Optionally, the linear image can be an HDR image.

[0038] Preferably, the rendering function can perform rendering based on rasterizers, shaders, and reprojection. For example, the rendering function uses a physically based bidirectional reflectance distribution function (such as the Cook-Torrance BRDF) to calculate the shading of each pixel. Alternatively, the rendering function can also use the Blinn-Phong function. For each sampling point on the surface of the 3D model, the bidirectional reflectance distribution function can approximately determine the contribution of each ray to the light ultimately reflected from a plane with given material properties. If a plane has a perfectly smooth surface (such as a mirror), then for all incident rays... (Except for one beam) the BRDF function will always return 0, except for one beam and the outgoing ray. Light rays with the same (reflected) angle will return a value of 1.

[0039] The entire rendering pipeline (including geometric transformations, texture sampling, rasterization or ray tracing, BRDF calculation, and illumination integration) is implemented using a differentiable programming framework, ensuring that gradients can be derived from the output image. Backpropagation to all input rendering parameter sets Θ and geometry and perspective .

[0040] According to one example of the invention, when estimating the HDR value for each surface sampling point (or texel), the material parameters... The solution space is related to the material map resolution and is generally large, so directly optimizing the material parameters is sufficient. This can easily lead to physically inconsistencies. Since the number of vertices in a 3D human hand mesh is much smaller than the number of texels in a material map, we can first estimate the HDR value of the face vertices and then use the face index of each pixel. and the centroid coordinates of each pixel within its corresponding triangle This is mapped onto a reflectivity map as the initial value for reflectivity. Therefore, the HDR prediction image based on patch vertices is calculated as follows.

[0041]

[0042] in, Represents matrix multiplication. This indicates an indexing operation on the HDR attribute tensor. and This is the return result of the rasterizer, with data dimensions of [batch, height, width] and [batch, height, width, 3], where batch represents the batch size, height represents the height, width represents the width, and 3 represents the 3 vertices of the triangle facet corresponding to each pixel.

[0043] According to one example of the invention, PyTorch3d, OpenDR, or Nvdiffrast is used as a differentiable rasterizer. The rendering function employs a bidirectional reflectance distribution function (BRDF), which accepts material parameters such as reflectivity, metallicity, roughness, and lighting direction. and radiance The position of the vertex of the face and normal and perspective The illustrative execution flow of BRDF, as an input parameter to the function, is as follows:

[0044] First, rasterize and project the human hand model onto a certain view plane. To obtain the predicted image , Each valid pixel in the image corresponds to a patch index of the human hand model. and the coordinates of the center of gravity According to the patch index and the coordinates of the center of gravity Reflectivity, metallicity, and roughness are obtained from material texture sampling, and position is obtained from geometry sampling. and normal .

[0045] Then, the effect of each light source on the radiance of each pixel is calculated as follows:

[0046] .

[0047] in, The specular reflection portion in a BRDF is represented as:

[0048]

[0049] in, Let be the normal distribution function. It is a geometric function. For Fresnel equations, Represents a half-range vector. Indicates the basic reflectivity. Indicates the direction of view. Indicates the direction of light.

[0050] Optionally, for the normal distribution function, if an Unreal Engine 4 model is used, then:

[0051]

[0052] Alternatively, if a Disney model is used, then:

[0053] .

[0054] Optionally, the geometric function can be represented as:

[0055] ;

[0056] in, , Indicates the direction of view Or the direction of light Any one of them, This represents the function in BRDF used to calculate the shadowing-masking effect.

[0057] Fresnel equations are expressed as:

[0058] The diffuse component in a BRDF is represented as:

[0059]

[0060] in, For half-range vectors, The basic reflectivity of the plane is as follows:

[0061]

[0062]

[0063] The effect of ambient light on the radiance of each pixel:

[0064]

[0065] Predicted HDR value for each pixel:

[0066] .

[0067] Step S3: Optimize the geometric structure, material parameters, and lighting parameters using the gradient descent algorithm to gradually reduce the gap between the multi-view RGB reconstructed image and the multi-view RGB captured image, and obtain the final optimization result. Among them, the optimization of lighting parameters includes adaptive lighting control, which includes merging directional light sources with similar radiation contributions, merging directional light sources with similar illumination directions, and eliminating invalid directional light sources.

[0068] According to an embodiment of the present invention, the operation of merging directional light sources with similar radiation contributions includes: for each pair of directional light sources, calculating the difference in illumination contribution between the two light sources in the light source pair along the half-range vector direction of all texels in the material map; obtaining designated light source pairs whose light source contribution difference is less than a radiation similarity threshold; and constructing two similar directional light sources in each designated light source pair into a merged directional light source, wherein the direction of the merged directional light source is set to be consistent with the directional light source with the largest radiance in the corresponding designated light source pair, and the radiance of the merged directional light source is set to the sum of the radiances of the two directional light sources in the corresponding designated light source pair. This scheme merges similar light sources based on radiation contribution; for example, for each pair of directional light sources... and Calculate the difference in their illumination contribution along the half-range vector directions of all texels. This reflects the similarity in coloring effects between the two light sources. If the following conditions are met... < (Radiation similarity threshold) then determines the light source and Highly similar, the two are merged into a new directional light source. Its direction is consistent with the original light source with the greatest radiance, and its radiance is the sum of the two.

[0069] According to one embodiment of the present invention, merging directional light sources with similar illumination directions includes: for each pair of directional light sources, calculating the cosine similarity of the illumination directions of the two light sources in the pair; selecting light source pairs with a cosine similarity greater than a geometric similarity threshold as selected light source pairs; constructing a fused directional light source from the two directional light sources in each selected light source pair; wherein the direction of the fused directional light source is set as a weighted average of the illumination directions of the two directional light sources in the corresponding selected light source pair; and the radiance of the fused directional light source is set as the sum of the radiances of the two directional light sources in the corresponding selected light source pair. This scheme merges similar light sources based on geometric differences, for example, calculating the cosine similarity of the illumination directions of each pair of directional light sources. and Cosine similarity of the illumination direction This reflects the positional difference between the two light sources. If the following conditions are met and > (Geometric similarity threshold) then determines the light source and High overlap, merging the two into a new light source. Its direction is a weighted average of the illumination directions of the two original light sources, and its radiance is the sum of the two.

[0070] According to one embodiment of the present invention, eliminating invalid directional light sources includes deleting directional light sources whose radiance norm is less than a radiation threshold. For example, if a certain directional light source... The norm of the radiance is less than a preset threshold. If it is considered to contribute little to the final rendering, it will be removed from the light source set.

[0071] According to one embodiment of the present invention, removing invalid directional light sources includes: deleting directional light sources with weak radiative influence, wherein, for each directional light source, the number of pixels significantly affected by the directional light source in the most recent complete rendering is counted, and directional light sources whose number is less than a threshold are considered as directional light sources with weak radiative influence. For example, the number of pixels significantly affected by the directional light source in the most recent complete rendering is counted. The number of pixels that have a significant impact (whose contribution exceeds a certain proportion of the total irradiance of that pixel). If this number is too small (less than the number threshold)... This indicates that the light source only affects a very small number of areas of the 3D model (which may be noise or redundancy), and it should be removed.

[0072] According to one embodiment of the present invention, adaptive illumination control further includes: after each merging and culling of directional light sources, updating the latest directional light source information and the optimized illumination direction. and radiance Update back to rendering parameter set .

[0073] According to one embodiment of the present invention, the total loss function used during optimization is configured as follows:

[0074]

[0075] in, Represents the total loss function. Indicates rendering loss. Represents geometric loss, Indicates material smoothness loss. This indicates a loss of physical legitimacy; , , , is a hyperparameter representing the weighting coefficients. This total loss can be used during optimization. This guides the simultaneous updating of material parameters, lighting parameters, and geometry. After the initial training reaches convergence or after a certain number of iterations, adaptive lighting control can be performed once. After each adaptive lighting control is performed, the gradient descent algorithm is used to train again until convergence or after a certain number of iterations (this can be done once or repeatedly) to optimize and adjust the geometry, material parameters, and lighting parameters to obtain the final optimized result.

[0076] Optionally, the rendering loss is the L1 distance, L2 distance, LPIPS distance, or SSIM distance between the multi-view RGB reconstructed image and the multi-view RGB captured image, or the rendering loss is a weighted sum of the combination of the L1 distance, L2 distance, LPIPS distance, and SSIM distance between the multi-view RGB reconstructed image and the multi-view RGB captured image.

[0077] According to one embodiment of the present invention, the rendering loss is used to limit the distance between the multi-view RGB reconstructed image and the multi-view RGB captured image. For example, the rendering loss is:

[0078]

[0079] in, This represents a function to calculate either the L1 distance or the L2 distance. This represents the function for calculating the SSIM distance. This represents a multi-view RGB reconstructed image. This indicates multi-view RGB captured images. , This represents the weighting coefficient.

[0080] According to one embodiment of the present invention, geometric loss is used to limit the distance between the predicted vertex positions of the facets and the labeled vertex positions of the facets, as well as the distance between the rasterized projection of the reconstructed object onto the multi-view RGB reconstructed image plane and the object mask in the multi-view RGB captured image. For example, the geometric loss is:

[0081]

[0082] in, Indicates the predicted vertex position of the face. Indicates the vertex position of the labeled face; This represents the mask loss, used to calculate the distance between the rasterized projection of the reconstructed object onto the multi-view RGB reconstructed image plane and the object mask in the multi-view RGB captured image. , This represents the weighting coefficients. Theoretically, the reconstructed geometry of an object should be as close as possible to the true value; therefore, a geometric loss in the 3D model is introduced. Jointly optimize the geometry of the object.

[0083] According to one embodiment of the present invention, a material smoothing loss is used to impose isotropic and anisotropic smoothing constraints on material parameters. For example, the material smoothing loss is:

[0084]

[0085] in, express Total Variation Loss This indicates the material parameters. (The material parameters...) Apply isotropic and anisotropic smoothing losses Avoid excessive high-frequency noise.

[0086] According to one embodiment of the present invention, loss of physical rationality This is used to constrain material parameters to fall within a reasonable range. For example, it constrains reflectivity to be within a reasonable range, metallicity, roughness, and specular reflectivity to be between [0,1], and the normal map to be close to... .

[0087] According to one embodiment of the present invention, in step S3, the geometry, material parameters and lighting parameters can be optimized in stages, and a specified item in the geometry, material parameters and lighting parameters can be optimized in each stage using the loss function of that stage.

[0088] According to an optional embodiment of the present invention, step S3 includes three optimization stages. Wherein:

[0089] Phase 1: Use rendering loss to guide the updating of reflectivity in material parameters, without updating geometry, metallicity, roughness, and lighting parameters. The loss function for Phase 1 is: .

[0090] The second stage: A weighted sum of rendering and geometric losses is used as the total loss to guide the updating of lighting parameters, roughness, and geometry, without updating reflectivity and metallicity. After this stage reaches convergence, adaptive lighting control is performed once. The loss function for the second stage is: .

[0091] The third stage: The weighted sum of rendering loss, geometric loss, material smoothing loss, and physical plausibility loss is used as the total loss to guide the updating of material parameters, lighting parameters, and geometry; the loss function for the third stage is: .

[0092] According to one example of the present invention, a three-stage strategy is employed in the optimization process. The three stages are executed sequentially, with each stage constituting one round, and then execution continues from the first stage based on the previous round. The termination condition for each stage is that the loss function converges or a preset number of iterations is reached.

[0093] The first stage only uses rendering loss terms. Furthermore, it does not update the material geometry, surface roughness, metallicity, or lighting parameters. This causes the reflectivity material details to diffuse into the triangular texture. If the lighting parameters are not preset properly, the reflection in the highlight areas will be too bright, and optimizing the roughness in this stage will lead to unstable roughness in the next stage.

[0094] The second stage uses rendering loss terms and geometric loss terms. Furthermore, the metallicity in the material parameters is not updated. This stage primarily optimizes the geometry, lighting parameters, and roughness in the material parameters. Adaptive lighting control is executed once after the loss function converges or reaches a preset number of iterations, directing the light source direction. and radiance It will also be fine-tuned after merging. Directly optimizing metallicity at this stage may lead to inconsistencies and non-convergent metallicity.

[0095] In the third stage, all loss terms are used to backpropagate the total loss, while simultaneously updating all parameters in the geometry and material maps. Illumination parameters of the remaining base light source .

[0096] Repeat the above optimization steps until the loss function converges or the preset number of iterations is reached. The final output is the optimized geometry and globally consistent high-quality material parameters. And the simplified lighting parameters .

[0097] According to another embodiment of the present invention, the implementer may also adopt other phased approaches. For example, step S3 includes four optimization phases. Wherein:

[0098] Phase 1: Using geometric loss to guide the updating of the geometric structure; the loss function for Phase 1 is: .

[0099] The second stage: Reflectivity in the material parameters is updated using rendering loss, but metallicity, roughness, and lighting parameters are not updated; the loss function for the second stage is: .

[0100] The third stage: The weighted sum of rendering and geometric losses is used as the total loss to guide the updating of lighting parameters, roughness, and geometry, without updating reflectivity and metallicity. After this stage reaches convergence, adaptive lighting control is performed once. The loss function for the third stage is: .

[0101] Phase 4: The weighted sum of rendering loss, geometric loss, material smoothing loss, and physical plausibility loss is used as the total loss to guide the updating of material parameters, lighting parameters, and geometry; the loss function for phase 4 is: .

[0102] Finally, a differentiable rendering function can be used to render based on the final optimized geometry, material parameters, lighting parameters, and new perspective as input, and output a new rendered image containing objects.

[0103] The effectiveness of the differentiable inverse rendering method proposed in this invention is verified below. For ease of evaluation, experiments are conducted on the "Xiaoyu" digital human-hand synthesis dataset, such as... Figure 3 As shown. Adaptive lighting control merges directional base light source pairs with an angle of less than 5 degrees between their illumination directions or a radiance difference of less than 0.1, and removes directional base light sources with a radiance below 0.05 or an effective number of affected pixels less than 1% of the total pixels. After approximately 8000 iterations (repeated 3 times in 3 optimization stages), the optimization converged. The average PSNR value of the re-rendered projected image of the hand model and the original image was 36.88 (only effective pixels were calculated), as shown. Figure 4 As shown. Considering that different material and lighting combinations can produce the same rendering effect, this invention first averages the generated reflectivity material map and the original reflectivity material map. After averaging, the PSNR value of the reflectivity is 30.02 (only effective pixels are calculated). The reflectivity material map is compared to... Figure 5 As shown, a high-quality (2048x2048) PBR material map (reflectivity, metallicity, roughness) was ultimately obtained, along with a streamlined light source set consisting of 12 directional lights and 1 point light (flash) (corresponding to 8 directional lights and 1 point light actually deployed in the real scene). This light source set can accurately interpret the appearance of the hand from all 62 viewpoints. The hand model and material map can be imported into 3D software and rendered with high realism under the new virtual lighting.

[0104] In summary, the present invention provides a method and system for robustly and accurately decoupling multiple types of PBR material properties of an object surface from unknown and complex environmental lighting from multi-view static images.

[0105] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order, as long as the required function can be achieved.

[0106] This invention can be a system, method, electronic device, computing device, computer program product and / or computer-readable medium.

[0107] Computer program products mainly refer to software products that implement various aspects of the present invention through computer programs, or hardware products that carry software that implements various aspects of the present invention.

[0108] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0109] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for inverting rendering, comprising performing the following operations for each object to be inverted: S1. Acquire a set of multi-view RGB images of a 3D object under fixed lighting conditions. Based on explicit 3D reconstruction technology, obtain the initial camera parameters and the initial geometric structure of the object from these images, and initialize the rendering parameter set. The rendering parameter set includes material parameters and lighting parameters. The lighting parameters initially include multiple directional light sources and one point light source. S2. Construct a differentiable rendering function, which is configured to take geometry, rendering parameter set and viewpoint as input, render multi-view linear image, and convert multi-view linear image into multi-view RGB reconstructed image. S3. The gradient descent algorithm is used to optimize the geometric structure, material parameters and lighting parameters to gradually reduce the gap between the multi-view RGB reconstructed image and the multi-view RGB captured image, and obtain the final optimization result. Among them, the optimization of lighting parameters includes adaptive lighting control, which includes merging directional light sources with similar radiation contributions, merging directional light sources with similar illumination directions and eliminating invalid directional light sources.

2. The method according to claim 1, characterized in that, The operations of merging directional light sources with similar radiation contributions include: For each pair of directional light sources, calculate the difference in lighting contribution of the two light sources in the half-length vector direction of all texels in the material map. Obtain designated light source pairs where the difference in light source contribution is less than the radiation similarity threshold. Construct two similar directional light sources in each designated light source pair into a merged directional light source. The direction of the merged directional light source is set to be consistent with the directional light source with the largest radiance in the corresponding designated light source pair. The radiance of the merged directional light source is set to be the sum of the radiances of the two directional light sources in the corresponding designated light source pair.

3. The method according to claim 2, characterized in that, Combined directional light sources with similar illumination directions include: For each pair of light sources formed by two directional light sources, calculate the cosine similarity of the illumination directions of the two light sources in the pair; Light source pairs with a cosine similarity greater than the geometric similarity threshold are selected light source pairs. The two directional light sources in each selected light source pair are used to construct a fused directional light source. The direction of the fused directional light source is set as a weighted average of the illumination directions of the two directional light sources in the corresponding selected light source pair. The radiance of the fused directional light source is set as the sum of the radiances of the two directional light sources in the corresponding selected light source pair.

4. The method according to claim 3, characterized in that, Eliminating invalid directional light sources includes: Remove directional light sources whose radiance norm is less than the radiance threshold; and / or Remove directional light sources with weak radiation influence. For each directional light source, count the number of pixels significantly affected by that directional light source in the most recent complete rendering, and directional light sources with a number less than a threshold are considered directional light sources with weak radiation influence.

5. The method according to claim 1, characterized in that, S3 employs a phased approach to optimize geometry, material parameters, and lighting parameters. In each phase, a loss function is used to optimize specific items among the geometry, material parameters, and lighting parameters.

6. The method according to any one of claims 1-5, characterized in that, Material parameters include reflectivity, metallicity, and roughness. S3 includes performing one or more rounds of the following operations: Phase 1: Use rendering loss to guide the updating of reflectivity in material parameters, without updating metallicity, roughness, and lighting parameters; Second stage: Use the weighted sum of rendering loss and geometric loss as the total loss to guide the updating of lighting parameters, roughness and geometry, without updating reflectivity and metallicity. After the convergence condition is met in this stage, perform an adaptive lighting control once. The third stage: Use the weighted sum of rendering loss, geometric loss, material smoothing loss and physical plausibility loss as the total loss to guide the updating of material parameters, lighting parameters and geometry; Among them, rendering loss is used to limit the distance between the multi-view RGB reconstructed image and the multi-view RGB captured image; geometric loss is used to limit the distance between the predicted face vertex position and the labeled face vertex position, as well as the distance between the rasterized projection of the reconstructed object on the multi-view RGB reconstructed image plane and the object mask in the multi-view RGB captured image; material smoothing loss is used to apply isotropic and anisotropic smoothing constraints to the material parameters; and physical plausibility loss is used to constrain the material parameters to fall within a reasonable numerical range.

7. The method according to any one of claims 1-5, characterized in that, Material parameters include reflectivity, metallicity, and roughness. S3 includes performing one or more rounds of the following operations: Phase 1: Using geometric loss to guide the updating of the geometric structure; Phase 2: Use rendering loss to guide the update of reflectivity in material parameters, without updating metallicity, roughness, and lighting parameters; The third stage: The weighted sum of rendering loss and geometric loss is used as the total loss to guide the updating of lighting parameters, roughness and geometry, but reflectivity and metallicity are not updated. After the convergence condition is met in this stage, adaptive lighting control is performed once. Phase 4: Use the weighted sum of rendering loss, geometric loss, material smoothing loss, and physical plausibility loss as the total loss to guide the updating of material parameters, lighting parameters, and geometry; Among them, rendering loss is used to limit the distance between the multi-view RGB reconstructed image and the multi-view RGB captured image; geometric loss is used to limit the distance between the predicted face vertex position and the labeled face vertex position, as well as the distance between the rasterized projection of the reconstructed object on the multi-view RGB reconstructed image plane and the object mask in the multi-view RGB captured image; material smoothing loss is used to apply isotropic and anisotropic smoothing constraints to the material parameters; and physical plausibility loss is used to constrain the material parameters to fall within a reasonable numerical range.

8. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1-7.

10. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1-7 by executing the executable instructions.

Images