Method and system for estimating human geometry, lighting and material from multi-view images

Abstract

The present application designs a method and system for estimating human geometry, illumination and material from multi-view images captured under unknown illumination based on the idea of joint optimization of neural radiance field rendering and inverse rendering, mainly divided into a geometry pre-training stage and a joint optimization stage, first pre-train the human geometry to obtain a good initialization result, then estimate the illumination and material and fine-tune the geometry initialization result. When estimating the material, the present application uses an anisotropic material model to model complex human material. When estimating the illumination, the present application does not use an additional training stage to model indirect illumination, but considers the exit radiance of indirect reflection points as indirect illumination, and estimates the illumination visibility to distinguish direct illumination and indirect illumination. The present application solves the problem of inaccurate estimation of complex human material, and can effectively decouple geometry, illumination and material from human multi-view images, and improves the realism of human relighting results.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of neural radiance field rendering and inverse rendering, and particularly relates to a method and system for estimating human geometry, lighting and material from multi-view images. BACKGROUND

[0002] Inverse rendering is a classic task in computer vision and computer graphics, which is very important for applications such as digitizing real world, novel view synthesis, relighting and material editing. Inverse rendering is ill-posed, that is, different geometry, material and lighting conditions can lead to similar appearance. With the rapid development of differentiable rendering and implicit neural representation, methods for inverse rendering of small object scenes have emerged. These methods usually only consider representing appearance with simple color values, without using more realistic physics-based rendering.

[0003] Recent methods represent geometry and material with neural radiance fields. These methods use coordinate-based multi-layer perceptrons and recover geometry, material and lighting by reducing the rendering loss between the rendered image and the input image. Most of these methods are suitable for small objects because their geometry and material are simple. However, these methods cannot be directly applied to inverse rendering of human bodies.

[0004] Inverse rendering of human bodies has two main challenges: ① Human material is relatively complex. For example, human skin is a layered material, and the geometry of human hair is difficult to reconstruct. Previous inverse rendering methods assume that the estimated material is isotropic. However, human hair and clothing material are anisotropic, and their physical properties are related to the viewing direction. ② Computing indirect lighting and lighting visibility is crucial for correctly estimating material. Most methods ignore indirect lighting or use additional stages and resources to train indirect lighting and lighting visibility. SUMMARY

[0005] To overcome the deficiencies of the prior art, the present application provides a method and system for estimating human geometry, lighting and material from multi-view images, which pre-trains human geometry in a geometry pre-training stage to obtain a good initialization result, and then fine-tunes the geometry initialization result in a joint optimization stage to estimate lighting and material, solving the problem of inaccurate human material estimation, reducing the cost of modeling indirect lighting and lighting visibility, and achieving more realistic inverse rendering results.

[0006] According to an aspect of the present application, a method for estimating human geometry, lighting and material from multi-view images is provided, comprising:

[0007] Obtaining a three-dimensional human model to be estimated and performing multi-view rendering;

[0008] input the rendered data into a trained human geometry, illumination, and material estimation network to output estimated human geometry, illumination, and material; wherein the training of the human geometry, illumination, and material estimation network comprises:

[0009] collecting three-dimensional human model data, and rendering multi-view human images and corresponding camera pose parameters according to the collected three-dimensional human model data;

[0010] geometric pre-training: taking the rendered data as input, using a signed distance function to implicitly represent the human surface, using a spherical tracing algorithm to obtain surface points, and calculating the appearance color values corresponding to the surface points through a neural radiance field; selecting a loss function for geometric pre-training, and gradually optimizing the signed distance function and the neural radiance field by minimizing the loss function;

[0011] joint optimization: using the pre-trained signed distance function and the spherical tracing algorithm to obtain human surface points, inputting the human surface points into a designed visibility network and a material network to output illumination visibility and material; using the illumination visibility and the pre-trained neural radiance field to calculate global illumination, and using the material to calculate a bidirectional reflectance distribution function; calculating the color values of the surface points through a rendering equation; selecting a loss function for joint optimization, and jointly optimizing the signed distance function, the neural radiance field, the visibility network, and the material network by minimizing the loss function.

[0012] As a further technical solution, the geometric pre-training further comprises:

[0013] implicitly representing the human surface as a zero value set of the signed distance function;

[0014] According to each input image and its corresponding camera pose, the pixel ray corresponding to each pixel point is obtained;

[0015] sampling on the pixel ray through the spherical tracing algorithm, calculating the SDF value of each sampling point, and regarding the sampling point as the surface point corresponding to the pixel ray when the SDF value of the sampling point is within a preset threshold approaching 0;

[0016] After obtaining the surface point coordinates, inputting the surface point coordinates into a designed neural radiance field to calculate the appearance color values.

[0017] As a further technical solution, the signed distance function and the neural radiance field are represented by a multi-layer perceptron, and in the geometric pre-training, the weight parameters of the two multi-layer perceptrons are trained by narrowing the gap between the predicted value and the true value, and are supervised by using a mask true value.

[0018] As a further technical solution, the joint optimization further comprises:

[0019] In the illumination estimation, the global illumination is decomposed into direct illumination and indirect illumination, the direct illumination is fitted by using a spherical Gaussian function, the indirect illumination is calculated by using the exit radiance of the indirect bounce points, and the visibility of the direct illumination to the surface points in the illumination direction is calculated by using the visibility indication;

[0020] In the material estimation, an anisotropic material model is used to calculate a bidirectional reflectance distribution function (BRDF), and a rendering equation is used to calculate a physically-based rendering color value.

[0021] As a further technical solution, the loss function of the joint optimization includes a geometric pre-training loss, an image reconstruction loss and a smoothing loss.

[0022] According to an aspect of the present disclosure, a system for estimating human geometry, illumination and material from multi-view images is provided, comprising:

[0023] A data acquisition module is configured to acquire a three-dimensional human model to be estimated and perform multi-view rendering;

[0024] A data estimation module is configured to input the rendered data into a trained human geometry, illumination and material estimation network, and output estimated human geometry, illumination and material; wherein the training of the human geometry, illumination and material estimation network comprises:

[0025] Collecting three-dimensional human model data, and rendering multi-view human images and corresponding camera pose parameters according to the collected three-dimensional human model data;

[0026] Geometric pre-training: taking the rendered data as input, using a signed distance function to implicitly represent the human surface, using a spherical tracking algorithm to obtain surface points, and calculating the appearance color values of the surface points by a neural radiance field; selecting a geometric pre-training loss function, and gradually optimizing the signed distance function and the neural radiance field by minimizing the loss function;

[0027] Joint optimization: using the pre-trained signed distance function and the spherical tracking algorithm to obtain human surface points, inputting the human surface points into a designed visibility network and a material network, and outputting illumination visibility and material; calculating global illumination by using the illumination visibility and the pre-trained neural radiance field, and calculating a bidirectional reflectance distribution function (BRDF) by using the material; calculating the color values of the surface points by a rendering equation; selecting a joint optimization loss function, and jointly optimizing the signed distance function, the neural radiance field, the visibility network and the material network by minimizing the loss function.

[0028] According to an aspect of the present application specification, there is provided an apparatus for estimating human geometry, illumination and material from multi-view images, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the method for estimating human geometry, illumination and material from multi-view images.

[0029] According to an aspect of the present application specification, there is provided a computer readable medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the method for estimating human geometry, illumination and material from multi-view images.

[0030] Compared with the prior art, the present application has the following beneficial effects:

[0031] The present application designs a method for estimating human geometry, illumination and material from multi-view images captured under unknown illumination based on the idea of joint optimization of neural radiance field rendering and inverse rendering, which is mainly divided into a geometry pre-training stage and a joint optimization stage. In the geometry pre-training stage, the present application first pre-trains human geometry to obtain a good initialization result, and then estimates illumination and material and fine-tunes the geometry initialization result in the joint optimization stage. When estimating material, the present application uses an anisotropic material model to model complex human material. In addition, when estimating illumination, the present application does not use an additional training stage to model indirect illumination, but considers the exit radiance of indirect reflection points as indirect illumination, and estimates illumination visibility to distinguish direct illumination and indirect illumination. The present application solves the problem of inaccurate estimation of complex human material, and can effectively decouple geometry, illumination and material from multi-view images of the human body, and improves the realism of the human re-illumination result. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0033] Fig. 1 is a method flowchart for estimating human geometry, illumination and material from multi-view images provided by the embodiments of the present application.

[0034] Fig. 2 is a framework diagram for estimating human geometry, illumination and material from multi-view images provided by the embodiments of the present application.

[0035] Fig. 3 is a diagram of different network structures provided by the embodiments of the present application. DETAILED DESCRIPTION

[0036] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the protection scope of the present application. In addition, the technical features in each embodiment or in a single embodiment provided by the present application can be combined with each other at will to form new technical solutions, and the combination is not restricted by the sequence of steps and / or structure mode, but should be based on the implementation by those of ordinary skill in the art. When the combination of technical solutions appears contradictory or cannot be implemented, it should be considered that the combination of technical solutions does not exist, and is not within the protection scope of the present application.

[0037] The embodiment of the present application provides a method for estimating human geometry, illumination and material from multi-view images. First, a three-dimensional human model to be estimated is obtained and multi-view rendering is performed. Then, the data obtained by rendering is input into a human geometry, illumination and material estimation network trained, and the estimated human geometry, illumination and material are output. The human geometry, illumination and material estimation network of the present application mainly includes a geometry pre-training stage and a joint optimization stage.

[0038] Please refer to Figs. 1 to 3 As a preferred embodiment, the method for estimating human geometry, illumination and material from multi-view images provided by the embodiment of the present application comprises the following steps:

[0039] Step 1, collect three-dimensional human models and construct a data set. Optionally, publicly available three-dimensional human models with complex materials including different ages, different skin colors, different national costumes, etc. can be purchased and downloaded from the Internet.

[0040] Step 2, import the three-dimensional human model data collected in step 1 into the rendering engine Blender and use different environment maps as illumination conditions. Write a python script to render multi-view human images and corresponding camera pose parameters.

[0041] Step 3, geometry pre-training stage. The data (multi-view images and camera pose) obtained by processing in step 2 is used as input, the signed distance function is used to implicitly represent the human surface, and the spherical tracking algorithm is used to obtain the surface points, and then the appearance color value corresponding to the surface points is calculated by the neural radiance field.

[0042] Step 4, using the image reconstruction loss as the loss function of the predicted value and the true value, gradually optimizing the signed distance function and the neural radiance field in step 3 by minimizing the loss function.

[0043] Step 5, joint optimization stage. Mainly divided into illumination calculation module and material estimation module. Use the signed distance function pre-trained in step 3 and the spherical tracing algorithm to obtain the surface points of the human body, and then input them into the designed visibility network and material network to output the visibility of the light and the material. Then use the visibility of the light and the neural radiance field pre-trained in step 3 to calculate the global illumination, and use the material to calculate the bidirectional reflectance distribution function BRDF. Finally, the color value of the surface point is calculated through the rendering equation.

[0044] Step 6, using the loss function in step 4 and the smoothing loss as the loss function of the joint optimization stage, jointly optimizing the signed distance function in step 3, the neural radiance field, and the visibility network and material network in step 5 by minimizing the loss function.

[0045] The step 2 is specifically described as follows:

[0046] For each human body model, 100 images with different viewing angles are randomly rendered on the sphere centered on the human body, of which 80 images are used as the training set and 20 images are used as the test set. The resolution of each image is 512x512. In addition, the mask of each human body image is obtained by using existing mature algorithms.

[0047] The step 3 is specifically described as follows:

[0048] The surface of the human body is implicitly represented as the zero set of the signed distance function (SDF). According to each input image and its corresponding camera pose, the pixel ray o+td corresponding to each pixel point can be obtained, where o represents the camera center, d represents the ray direction, t represents the position of the point on the pixel ray, and t=0 represents the starting point of the ray. 2048 pixel rays are sampled for each image.

[0049] Then the SDF value of each sampling point is calculated by the spherical tracing algorithm. The specific process of the spherical tracing algorithm is as follows: a t value is randomly initialized as the starting point of sampling, the SDF value s of the sampling point is calculated, if s is not within the preset threshold approaching 0, then continue sampling, take t+s as the t value of the next sampling point, repeat the operation. When the SDF value of the sampling point is within the preset threshold approaching 0, the point is approximately regarded as the surface point corresponding to the pixel ray. After obtaining the surface point coordinates, they are input into the designed neural radiance field together with the normal, viewpoint direction and feature vector to calculate the appearance color value.

[0050] Signed distance function S and neural radiance field L oare represented by multi-layer perceptrons, which are defined as follows:

[0051] v sdf f=S(x;Θ)

[0052] C r =L o (x,n,f,d;Γ)

[0053] where x denotes the surface point coordinates, n denotes the surface normal, f denotes the feature vector, and d denotes the viewpoint direction. Θ and Γ denote the weight parameters of the two multi-layer perceptrons, respectively. In addition, the gradient of S(x;Θ) is used by the embodiments of the present application to calculate the surface normal value n, that is,

[0054] The step 4 is specifically described as follows:

[0055] The weight parameters Θ and Γ of the two multi-layer perceptrons in step 3 are trained by reducing the gap between the calculated appearance color value and the true value. In addition, the embodiments of the present application also use the mask true value for supervision. The final loss function is defined as follows:

[0056] L geo =L color +λ1L mask +λ2L reg

[0057] The image reconstruction loss L color is defined as follows:

[0058]

[0059] where C k denotes the color value calculated by the neural radiance field in step 3, C mask denotes the true color value in step 2, m denotes the number of surface points, and R denotes the L1 loss.

[0060] The mask loss L p is defined as follows:

[0061]

[0062] where M reg denotes the mask value of the surface point p in step 2, S(p;Θ) denotes the SDF value in step 3, BCE denotes the cross-entropy loss, and α is a hyperparameter. The value of α is preset to 50.

[0063] The regularization loss L λ(v·ξ-1) is also used to constrain the signed distance field, which is defined as follows:

[0064]

[0065] where, denotes the gradient of the signed distance field at surface point p in step 3.

[0066] The step 5 is mainly divided into an illumination calculation module and a material estimation module, which are described as follows:

[0067] For the illumination estimation module, the global illumination is decomposed into direct illumination and indirect illumination. The direct illumination is fitted by a spherical Gaussian function, and the definition of the spherical Gaussian function is as follows:

[0068] G(v; ξ, λ, μ) = μe λ(v·ξ-1)

[0069] where, v ∈ R 3 denotes the input of the function, ξ ∈ S 2 denotes the center direction of the spherical Gaussian, λ ∈ R + denotes the attenuation rate of the spherical Gaussian, denotes the illumination amplitude value of the center position of the spherical Gaussian. A total of M = 128 spherical Gaussian functions are used to represent the direct illumination L i (ω d ) on the illumination direction ω i , which is defined as follows:

[0070]

[0071] The indirect illumination is calculated by the exit radiance of the indirect bounce point. Specifically, the current surface point x is taken as the starting point, and the indirect bounce point is obtained by the spherical tracing algorithm along the illumination direction ω i . The embodiment of the present application totally samples 32 light direction, randomly initializes a t value on the light direction as the sampling starting point, calculates the SDF value s of the sampling point, if s is not within the preset threshold close to 0, continue to sample, and take t + s as the t value of the next sampling point. When the SDF value of the sampling point is within the preset threshold close to 0, it is indicated that the surface point x can collide with other surface points along the light direction, that is, it is indicated that the surface point x will receive the indirect light emitted by other surface points along -ω i . Other surface points are regarded as indirect bounce points . The exit radiance of the indirect bounce point is taken as the indirect illumination of the surface point x. The color value of calculated by the neural radiance field L o in step 3 is taken as the exit radiance of . Therefore, the definition of the indirect illumination L ind (x, ω i ) is as follows:

[0072]

[0073] where, denotes the normal of the indirect bounce point , denotes the eigenvector of the indirect bounce point .

[0074] In addition, visibility is used to indicate direct lighting in the lighting direction ω i for a surface point x. Visibility v is estimated with a multi-layer perceptron, which is defined as follows:

[0075] v = Φ(β(x), d; θ)

[0076] where β denotes the positional encoding, x denotes the surface point, d denotes the viewing direction, and θ denotes the weights of the multi-layer perceptron. Finally, the global illumination L i (x, ω i ) is computed from the direct lighting L d (ω i ), the indirect lighting L ind (x, ω i ), and the visibility v, which is calculated as follows:

[0077] L i (x, ω i ) = vL d (ω i ) + (1-v)L ind (x, ω i )

[0078] For the material estimation module, an anisotropic material model is used to calculate the bidirectional reflectance distribution function BRDF, so as to represent the physical properties of the complex material of the human body and make the rendered color value more realistic through physically-based rendering. Physically-based rendering is a shading model that complies with physical principles. The cook-torrance model based on the microfacet principle is widely used in the industry. This model divides the bidirectional reflectance distribution function BRDF into diffuse reflection component and specular reflection component. Its definition is as follows:

[0079]

[0080] where l denotes the incident light direction, v denotes the viewpoint direction, diffuse denotes the diffuse reflection estimated by the material network, n denotes the normal direction, D denotes the normal distribution function, F denotes the Fresnel equation, and G denotes the geometric shading function. D, F, and G are calculated using the anisotropic material model. The specific calculation process is as follows:

[0081] The GTR function is used to calculate the normal distribution function D, and the GGX function is used to calculate the geometric shading function G, and the calculation formula is as follows

[0082]

[0083] where t denotes the tangent direction, b denotes the bitangent direction, h denotes the half-angle vector between the incident light direction l and the view direction v, and the final G value is G(l)G(v). In addition, a x and a y denote the roughness along the tangent direction and along the normal, respectively. a x and a y are calculated as follows:

[0084] a y = r 2 k

[0085] where k aniso denotes the anisotropy degree parameter, which is set as a trainable parameter and optimized together with the material, and r estimates the roughness. When k aniso = 0, a x = a y , the anisotropic model becomes the isotropic model, so the range of k aniso is (0, 1). In addition, the tangent direction t and the bitangent direction b are also calculated using the incident light direction ω o and the normal n, and the calculation formula is as follows:

[0086] t = ω o - (ω o · n) n, b = n x t

[0087] The calculation formula of the Fresnel equation F is as follows:

[0088] F(v, h) = F0 + (1 - F0)(1 - (v · h)) 5

[0089] where F0 denotes the specular reflectance, which is estimated by the material network, v is the view direction, and h denotes the half-angle vector between the incident light direction l and the view direction v.

[0090] The materials used for calculation include diffuse reflectance a, specular reflectance s, roughness r, subsurface scattering sss, and shininess sh. Similarly, a multi-layer perceptron is used to calculate the material, which is defined as follows:

[0091] {a, r, sss, s, sh} = M(β(x); ψ)

[0092] where β denotes the position encoding, x denotes the surface point, and ψ denotes the weight of the multi-layer perceptron.

[0093] After obtaining the global illumination and BRDF, the color value L based on physical rendering is calculated through the rendering equation surf (x,ω i ,ω o ). The definition of the rendering equation is as follows:

[0094] L surf (x,ω i ,ω o )=∫ Ω L i (x,ω i )f r (x,ω i ,ω o )(ω i ·n)dω i

[0095] Wherein, x represents a surface point, ω i represents the incident light direction, ω o represents the view point direction, L i (x,ω i ) represents the global illumination, f r (x,ω i ,ω o ) represents the bidirectional reflectance distribution function BRDF value, and n represents the normal direction.

[0096] The step 6 is specifically described as follows:

[0097] The loss function of the joint optimization stage includes the loss function L geo in step 4, the image reconstruction loss L rec and the smoothing loss L smooth . Its definition is as follows:

[0098] L total =L geo +0.1L rec +0.1L smooth

[0099] The image reconstruction loss L rec is defined as follows:

[0100]

[0101] Wherein, p represents the number of surface points, m represents the total number of sampling pixels, L p represents the color value based on physical rendering, C smooth represents the true color value, and R represents the MAE loss.

[0102] The smoothing loss L smooth is defined as follows:

[0103]

[0104] where S denotes the number of sampled pixels, x p denotes the number of surface points. The gradient of the input image is pre-computed. The roughness gradient and the specular reflectance gradient are computed during backpropagation. The loss function L total is jointly optimized with the signed distance function S in step 3, the neural radiance field L o and the visibility network Φ and the material network M in step 5.

[0105] It should be noted that the above pre-computation refers to the calculation of the gradient value of the pixel of the picture while reading the picture during data loading, wherein the reading of the picture and the calculation of the gradient value are realized by directly calling the corresponding function in the OpenCV library.

[0106] The implementation basis of each embodiment of the present application is realized by programmed processing of a device with processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present application are packaged into various modules. Based on this actual situation, on the basis of the above embodiments, an embodiment of the present application provides a system for estimating human geometry, illumination and material from multi-view images, which is used to execute one of the methods for estimating human geometry, illumination and material from multi-view images in the above method embodiments.

[0107] The system comprises a data acquisition module for acquiring a three-dimensional human model to be estimated and performing multi-view rendering; and a data estimation module for inputting the rendered data into a trained human geometry, illumination and material estimation network to output estimated human geometry, illumination and material; wherein the training of the human geometry, illumination and material estimation network comprises: collecting three-dimensional human model data, rendering multi-view human images and corresponding camera pose parameters according to the collected three-dimensional human model data; performing geometric pre-training: taking the rendered data as input, using a signed distance function to implicitly represent the human surface, using a spherical tracking algorithm to obtain surface points, and calculating the appearance color values corresponding to the surface points through a neural radiance field; selecting a loss function of the geometric pre-training, and gradually optimizing the signed distance function and the neural radiance field by minimizing the loss function; performing joint optimization: using the pre-trained signed distance function and the spherical tracking algorithm to obtain human surface points, inputting the human surface points into a designed visibility network and a material network, and outputting illumination visibility and material; using the illumination visibility and the pre-trained neural radiance field to calculate global illumination, and using the material to calculate a bidirectional reflectance distribution function; calculating the color values of the surface points through a rendering equation; selecting a loss function of the joint optimization, and jointly optimizing the signed distance function, the neural radiance field, the visibility network and the material network by minimizing the loss function.

[0108] The system for estimating human geometry, illumination and material from multi-view images provided by the embodiment of the present application faces the challenge of human inverse rendering, adopts the above-mentioned several modules, pre-trains human geometry in a geometry pre-training stage to obtain a good initialization result, and then estimates illumination and material and fine-tunes the geometry initialization result in a joint optimization stage, solves the problem of inaccurate human material estimation, reduces the cost of modeling indirect illumination and illumination visibility, and realizes a more realistic inverse rendering effect.

[0109] It should be noted that the system embodiments provided by the present application are used to implement the methods in the above method embodiments, and are also used to implement the methods in other method embodiments provided by the present application. The difference is only that the corresponding functional modules are set, and the principle is basically the same as that of the above system embodiments provided by the present application. As long as the person skilled in the art improves the modules in the above system embodiments on the basis of the above system embodiments, refers to the specific technical solutions in other method embodiments, obtains the corresponding technical means by combining technical features, and the technical solutions composed of these technical means, as long as the technical solutions have practicality, the corresponding system class embodiments are obtained, which are used to implement the methods in other method class embodiments.

[0110] Based on the same inventive concept as the foregoing embodiments, the embodiment of the present application also provides a device for estimating human geometry, illumination and material from multi-view images, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the method for estimating human geometry, illumination and material from multi-view images.

[0111] Based on the same inventive concept as the foregoing embodiments, the embodiment of the present application also provides a computer readable medium having a computer program stored thereon, wherein the program is executed by a processor to implement the steps of the method for estimating human geometry, illumination and material from multi-view images.

[0112] It should be noted that the embodiments of the present application can be realized by hardware, software, or a combination of software and hardware. The hardware part can be realized by special logic; the software part can be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or a specially designed hardware. Those skilled in the art can understand that the above-mentioned devices and methods can be realized by computer executable instructions and / or included in processor control codes, such as carrier media, such as magnetic disk, CD or DVD-ROM, programmable memory, such as read-only memory (firmware), or data carrier, such as optical or electronic signal carrier. The device of the present application and its modules can be realized by hardware circuit, such as ultra-large scale integrated circuit or gate array, semiconductor, such as logic chip, transistor, etc., or programmable hardware device, such as field programmable gate array, programmable logic device, etc., or by software executed by various types of processors, or by a combination of the above-mentioned hardware circuit and software, such as firmware.

[0113] In summary of the above embodiments, the present application designs a method and system for estimating human geometry, illumination and material from multi-view images captured under unknown illumination based on the idea of joint optimization of neural radiance field rendering and inverse rendering. The present application is mainly divided into a geometry pre-training stage and a joint optimization stage. The geometry pre-training stage first pre-trains the human geometry to obtain a good initialization result, and then estimates the illumination and material and fine-tunes the geometry initialization result in the joint optimization stage. When estimating the material, the present application uses an anisotropic material model to model the complex human material. In addition, when estimating the illumination, the present application does not use an additional training stage to model indirect illumination, but considers the exit radiance of indirect reflection points as indirect illumination, and estimates the illumination visibility to distinguish direct illumination and indirect illumination. The present application solves the problem of inaccurate estimation of complex human material, and can effectively decouple geometry, illumination and material from multi-view images of human body, and improves the realism of the human re-illumination result.

[0114] The terms "comprise" and "have" and any variations thereof in the specification and in the claims and the above-mentioned drawings are intended to cover not exclusively containing, for example, a process, method, system, product or device comprising a series of steps or units, not necessarily limited to which steps or units are clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0115] It should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present application.

Claims

1. A method of estimating human geometry, lighting and material from multi-view images, characterized in that, The method comprises the following steps: acquire a three-dimensional human body model to be estimated and perform multi-view rendering; input the rendered data into a trained human body geometry, illumination and material estimation network to output estimated human body geometry, illumination and material; wherein the training of the human body geometry, illumination and material estimation network comprises: collect three-dimensional human body model data, and render multi-view human body images and corresponding camera pose parameters based on the collected three-dimensional human body model data; geometric pre-training: input the rendered data as input, use a signed distance function to implicitly represent the human body surface, use a spherical tracking algorithm to obtain surface points, and calculate the appearance color value corresponding to the surface points through a neural radiance field; select a loss function for geometric pre-training, and gradually optimize the signed distance function and the neural radiance field by minimizing the loss function; joint optimization: use the pre-trained signed distance function and the spherical tracking algorithm to obtain human body surface points, input them into the designed visibility network and material network, and output illumination visibility and material; use the illumination visibility and the pre-trained neural radiance field to calculate the global illumination, and use the material to calculate the bidirectional reflectance distribution function; calculate the color value of the surface points through the rendering equation; select a loss function for joint optimization, and jointly optimize the signed distance function, the neural radiance field, the visibility network and the material network by minimizing the loss function; The joint optimization further comprises: in the illumination estimation, decompose the global illumination into direct illumination and indirect illumination, fit the direct illumination using a spherical Gaussian function, calculate the indirect illumination using the exit radiance of the indirect bounce points, and use the visibility to indicate the visibility of the direct illumination to the surface points in the illumination direction; in the material estimation, use an anisotropic material model to calculate the bidirectional reflectance distribution function BRDF, and use the rendering equation to calculate the color value based on physical rendering.

2. The method of claim 1, wherein, The geometric pre-training further comprises: implicitly represent the human body surface as a zero value set of the signed distance function; obtain the pixel ray corresponding to each pixel point according to each input image and the corresponding camera pose; sample on the pixel ray through the spherical tracking algorithm, calculate the SDF value of each sampling point, and regard the sampling point as the surface point corresponding to the pixel ray when the SDF value of the sampling point is within a preset threshold approaching 0; after obtaining the surface point coordinates, input them into the designed neural radiance field to calculate the appearance color value.

3. The method of claim 2, wherein, The signed distance function and the neural radiance field are represented by multi-layer perceptrons, and the weight parameters of the two multi-layer perceptrons are trained by narrowing the gap between the predicted value and the true value during geometric pre-training, and are supervised by using the mask true value.

4. The method of claim 1, wherein, The loss function of the joint optimization comprises a geometric pre-training loss, an image reconstruction loss and a smoothing loss.

5. A system for estimating human geometry, lighting and material from multi-view images, characterized in that, The method comprises the following steps: a data acquisition module for acquiring a three-dimensional human body model to be estimated and performing multi-view rendering; a data estimation module for inputting the rendered data into a trained human body geometry, illumination and material estimation network to output estimated human body geometry, illumination and material; wherein the training of the human body geometry, illumination and material estimation network comprises: Collect three-dimensional human model data, render multi-view human body images and corresponding camera pose parameters according to the collected three-dimensional human model data; Geometric pre-training: use the rendered data as input, use the signed distance function to implicitly represent the human body surface, use the spherical tracking algorithm to obtain the surface points, and calculate the appearance color value corresponding to the surface points through the neural radiation field; select the loss function of geometric pre-training, and gradually optimize the signed distance function and the neural radiation field by minimizing the loss function; Joint optimization: use the pre-trained signed distance function and the spherical tracking algorithm to obtain the human body surface points, input them into the designed visibility network and material network, and output the lighting visibility and material; use the lighting visibility and the pre-trained neural radiation field to calculate the global lighting, and use the material to calculate the bidirectional reflectance distribution function; calculate the color value of the surface points through the rendering equation; select the loss function of joint optimization, and jointly optimize the signed distance function, the neural radiation field, the visibility network and the material network by minimizing the loss function; The joint optimization further includes: in the lighting estimation, decompose the global lighting into direct lighting and indirect lighting, fit the direct lighting using the spherical Gaussian function, calculate the indirect lighting using the exit radiance of the indirect bounce point, and simultaneously use the visibility to indicate the visibility of the direct lighting to the surface points in the lighting direction; in the material estimation, use the anisotropic material model to calculate the bidirectional reflectance distribution function BRDF, and use the rendering equation to calculate the color value based on physical rendering.

6. An apparatus for estimating human geometry, lighting, and material from multi-view images, the apparatus comprising: Comprise: One or more processors; Storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the steps of the method for estimating human geometry, lighting and material from multi-view images according to any one of claims 1-4.

7. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the steps of the method for estimating human geometry, lighting and material from multi-view images according to any one of claims 1-4.

Images