High quality view synthesis with relighting

Abstract

Techniques include generating a model for novel view synthesis with lighting that is configured to decompose a light field into two components that encapsulate light transport and reflectance within a scene. The first component represents radiance due to direct illumination the second component represents radiance due to indirect illumination and is based on the radiance emitted by an illumination source and the indirect illumination's transport function. The relightable model is generated from a set of images captured with a relatively small array of cameras and illumination sources. The model may then generate a new image of the object from a new perspective with illumination from a new orientation. The illumination of the object is based on the first component and the second component of the radiance field.

Description

HIGH QUALITY VIEW SYNTHESIS WITH RELIGHTINGCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority- to U.S. Provisional Patent Application No. 63 / 586,789, filed on September 29, 2023, entitled “INTRINSIC RADIANCE DECOMPOSITION FOR HIGH-QUALITY VIEW SYNTHESIS AND RELIGHTING OF FACES”, the disclosure of which is incorporated by reference herein in its entirety.SUMMARY

[0002] The concepts discussed herein are directed to a technique for performing high-quality capture of a plurality of images of an object such as a human face for three- dimensional view synthesis and relighting of the object. The capturing of the plurality of images of the object is performed using a relatively small array of lights and cameras positioned at varied angles with respect to the object. A model configured to perform relighting is derived from the plurality of images captured with the array of lights and cameras. The model includes a light field that is divided into (e.g., decomposable into) a first component and a second component. The first component represents light (e.g.. radiance) due to direct illumination and is dependent on position in the scene and a view direction. The second component represents light (e.g., radiance) due to indirect illumination, is dependent on position in the scene, and is independent of the view direction. Furthermore, the first component is decomposed to a position-dependent diffuse component, light, view and position dependent specular component, and a light and position-dependent transport function that accounts for self-shadowing. Such a decomposition of the light field results in a volumetric model that produces renderings at high-resolution and allows for point, directional, and environmental relighting along with accurate soft and hard shadows, achieving a high degree of photorealism.

[0003] In one general aspect, a method can include capturing a plurality of images of an object, the plurality of images being captured from a plurality of perspectives and using a plurality- of directions of at least one illumination source. The method can also include defining a light field based on the plurality of images of the object, the light field having a first component and a second component, the first component representing directillumination and the second component representing indirect illumination. The method can further include generating a new image of the object from a new perspective, the new image of the object being illuminated based on the first component and the second component.

[0004] In another general aspect, a computer program product comprising a nontransitory storage medium, the computer program product including code that, when executed by processing circuitry, causes the processing circuitry’ to perform a method. The method can include capturing a plurality of images of an object, the plurality of images being captured from a plurality of perspectives and using a plurality of directions of at least one illumination source. The method can also include defining a tight field based on the plurality of images of the object, the light field having a first component and a second component, the first component representing direct illumination and the second component representing indirect illumination. The method can further include generating a new image of the object from a new perspective, the new image of the object being illuminated based on the first component and the second component.

[0005] In another general aspect, an electronic apparatus can include memory’ and processing circuitry coupled to the memory. The processing circuitry can be configured to capture a plurality of images of an object, the plurality of images being captured from a plurality of perspectives and using a plurality of directions of at least one illumination source. The processing circuitry can also be configured to define a tight field based on the plurality of images of the object, the light field having a first component and a second component, the first component representing direct illumination and the second component representing indirect illumination. The processing circuitry’ can further be configured to generate a new image of the object from a new perspective, the new image of the object being illuminated based on the first component and the second component.

[0006] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 is a diagram illustrating an example generation of a relightable model and a relit image from the relightable model in accordance with implementations described herein.

[0008] FIG. 2 is a diagram illustrating an example network architecture of a model configured to generate lighting.

[0009] FIG. 3 is a diagram that illustrates example processing circuitry of a computing device configured to generate a model configured to generate lighting.

[0010] FIG. 4 is a flow chart illustrating an example process of generating a relightable model.DETAILED DESCRIPTION

[0011] It can be difficult to capture real, three-dimensional (3D) human faces that have proper lighting. Having realistic human faces that are lit properly is important in, for example, gaming in virtual reality (VR) or augmented reality (AR) in which realistic- looking characters are a crucial part of the story. For example, a character in a game whose shadow contradicts the lighting environment in the game will appear unrealistic to the user. Such a mismatch between light and shadow can detract from the user experience. A properly lit character in which the lighting and the shadows obey expected physics, however, adds to the realism of the video game and enhances the user experience.

[0012] The lighting of a human face depends on a direction from which a camera capturing the human face views the human face. The lighting of the human face also depends on a position and direction of an illuminator that shines light on the human face. The light shining on the human face can be reflected off the face in a myriad of ways that depend on the material properties of the skin and hair as well as the complex face geometry. It is the way that the light interacts with the material and geometry of the face that makes capturing 3D human faces having proper lighting difficult.

[0013] The capture of a properly lit 3D human face, taking into account all of the interactions of light with material and geometry of the face, is accomplished through a model as disclosed in the technical solutions described herein. For example, the model can include a neural network that leams how to reproduce proper lighting based on real images of a human face from various view directions and illuminated from various angles. At least one technical problem with such models, which it is addressed by the technical solutions herein, is that they require an enormous amount (e.g., hundreds) of input images at different view directions and illumination directions in order to generate a model. Moreover, the results of such a model, even with the large number of images used to build the model, tend to be inaccurate.

[0014] At least some of the technical solutions described herein are directed to high- quality acquisition of face geometry and reflectance, within controlled environments, with the goal of reproducing photoreal renditions of human faces under arbitrary view point andlighting. At least some technical solutions described herein enable seamless blending of photoreal renditions of human faces into a desired 3D environment. These technical solutions address at least some of the technical problems with traditional methods that can require large capture rigs consisting of complex and expensive equipment such as synchronized light domes with hundreds of lights and multiple calibrated cameras with polarization filters. The technologies described herein can be used in areas such as augmented reality (AR) and virtual reality (VR). The technical solutions described herein can also be used in generating high-quality images for machine learning in perception and computational photography applications.

[0015] Many known capture setups have the technical problem of being complex, bulky, immobile and expensive. Furthermore, conventional graphics techniques using explicit mesh and texture representations require significant manual intervention by skilled artists before final assets are ready for photorealistic rendering. Although automated methods have been proposed, such automated methods achieve limited realism due to inaccurate reconstruction of the complex geometry of hair, clothing, and accessories. These disadvantages make a strong case for smaller capture arrays and simpler models that achieve high-quality photorealism and fidelity.

[0016] Reproducing photoreal renditions of human faces under arbitrary viewpoint may be accomplished by technical solutions described herein which can include capturing a set of 3D images using an array of cameras representing various viewing directions. The captured set of 3D images define a light field which is used to build a model. The model may then be used to generate a 3D image from a new perspective (novel view synthesis).

[0017] Predicting lighting under arbitrary conditions in addition to generating 3D images from a new perspective presents significant technical problems. As the description of the input 3D images, which are generated with cameras and illumination sources, becomes more complex, the model used to generate a 3D image from an arbitrary perspective with lighting becomes accordingly complex. Specifically, the number of 3D images needed to generate a model that can accurately perform novel view synthesis with lighting is much larger than that required to perform novel view synthesis without lighting. Generating the model from the light field derived from the 3D images accordingly requires significant computing resources.

[0018] Accordingly, at least one technical problem with conventional novel image synthesis is that including lighting capability into the model is very difficult. For example, in order to facilitate lighting, conventional models require additional complexities such asthe addition of a controlled light setting that models reflectance, shadows, and interreflections in order to enable lighting. Moreover, even in the face of such complexities, the resulting rendering quality, despite having a large number of input lighting conditions, is poor.

[0019] A light field that is used to generate a model for novel view synthesis depends on a view direction, i.e., a direction from which a camera views a human face. One way a model could be extended to be configured for lighting is to condition the light field of a model on an illumination direction of an illumination source in addition to the view direction. During rendering there would be an integration of the light field over the light direction. Nevertheless, it would be difficult to build a model with lighting from sparse datasets based on such a light field.

[0020] At least one technical solution to the technical problem includes introducing a model that is configured to decompose a light field into simple factors that encapsulate light transport and reflectance within a scene. The factors include a first component and a second component. The first component represents direct illumination and is based on a radiance emitted by an illumination source, the direct illumination’s transport function, and a reflectance function. In some implementations, the reflectance function is decomposed into a diffuse component and a specular component. The second component represents indirect illumination and is based on the radiance emitted by an illumination source and the indirect illumination’s transport function. The model is generated from a set of images captured with a relatively small array of cameras and illumination sources. The model may then generate a new image of the object from a new perspective and illuminated from a new orientation. The new illumination of the object is based on the first component and the second component of the radiance field.

[0021] At least one technical benefit of the technical solution is that relatively few illumination and camera viewpoints can be used to generate a simple model configured to generate a new image of the object from a new perspective and illuminated from a new orientation. In some implementations, there are as few as 15 camera orientations and 15 illumination orientations used in the generation of the model.

[0022] FIG. 1 is a diagram illustrating an example generation of a model 130 and a relit image 150 from the model 130. In FIG. 1, there are multiple images 110(a), 110(b), 110(c) of a human face captured from different perspectives. The human face, in each perspective, is illuminated by an illumination source that provides illumination at a respective direction 120(a), 120(b), 120(c). In some implementations, the differentillumination directions 120(a), 120(b), 120(c) originate from a compact array including 15 lights distributed over a frontal hemisphere.

[0023] Some implementations include a capture setup that consists of an array of 15, 26 Megapixel cameras with 50 mm lenses, framed to cover the human face from ear to ear and bust to scalp hair. In some implementations, the illumination sources are spread over the frontal hemisphere wi th average angular separation of 44 degrees horizontally and 60 degrees vertically. In some implementations, the illumination sources are one-hght-at-a- time (OLAT) sources. In some implementations, the cameras are geometrically and photometrically calibrated using standard tools. To calibrate lights, the cameras’ positions in 3D are triangulated using images of their reflections on a mirror sphere from three different locations to improve accuracy.

[0024] In contrast with previous methods that use complex camera and illuminator arrays and large training datasets to leam a relightable basis, the number of viewpoints and illumination directions used by the model is nearly an order of magnitude smaller. Thus, the relatively small array of illuminators and cameras is easier to reproduce and deploy at scale.

[0025] The multiple images 110(a), 110(b), 110(c) provide data for generating the model 130. The model 130 determines a light field 140 based on the multiple images 110(a), 110(b), 110(c). In some implementations, the light field 140 is a radiance field. The model 130 also decomposes the light field 140 into a first component 142 and a second component 144 such that the first component 142 represents direct illumination and the second component 144 represents indirect illumination. Direct illumination is illumination that is incident on a point without being scattered from another surface first. Indirect illumination is illumination that is incident on a point that results from scattering from another point first. It is assumed here that indirect illumination is diffuse in nature.

[0026] The first component 142 of the light field 140 depends on a position in a scene containing the human face and a camera viewing direction. The first component 142 represents direct illumination and is based on a radiance emitted by an illumination source, the direct illumination's transport function, and a reflectance function. In some implementations, the reflectance function is decomposed into a diffuse component and a specular component. The second component 144 of the light field 140 depends on the position in the scene. The second component 144 represents indirect illumination and is based on the radiance emitted by an illumination source and the indirect illumination’s transport function.

[0027] The direct illumination’s transport function may be a function (e g., acontinuous function) that specifies how direct illumination interacts with an object when the object is viewed from various perspectives. For example, the direct illumination’s transport function may enable a computer to describe how a ray of light interacts with the material of an object and at the perspective of a camera. As such, the direct illumination’s transport function can be used to generate an image of an object that depicts the object from a new perspective with arbitrary lighting. For instance, the new perspective may show the object using novel lighting, e.g., illumination of the object from a light source at a camera viewpoint that has not been previously captured and recorded.

[0028] The indirect illumination’s transport function may be a function (e.g., a continuous function) that specifies how indirect illumination’s illumination interacts with an object. As the indirect illumination is diffuse, the indirect illumination’s transport function is independent of viewing direction. For example, the direct illumination’s transport function may enable a computer to describe how a scattered ray of light interacts with the material of an object. It is noted that, while the direct illumination’s transport function is color-neutral, the indirect illumination’s transport function accounts for changing color due to scattering. Moreover, the indirect illumination’s transport function accounts for reflections of the scattered light.

[0029] The reflectance is associated with direct illumination and is broadly a ratio of reflected light energy to incident light energy. More specifically, reflectance is defined to be a change in radiance due to a change in irradiance at an object’s surface. The reflectance may be a general scattering distribution function. In physically-based volume rendering, this term may be defined by the volume’s albedo and phase function. The latter is the volumetric analog of a bidirectional scattering distribution function (BSDF). The specular reflectance is the portion of the reflectance due to directional, e.g., mirror-like, reflections from smooth surfaces. The diffuse reflectance is the portion of the reflectance due to diffuse reflections from rough or uneven surfaces.

[0030] The model 130 is then configured to produce a new image 150 of the human face from a new perspective and a new direction of illumination. Specifically, the first component 142 and the second component 144 of the light field 140 are configured to determine a new illumination profile of the human face.

[0031] In some implementations, the model 130 is a neural radiance field (NeRF) model. In this case, the model 130 represents a 3D scene as an emissive 3D volume. The model 130 includes a density field, and a light field 140 which are continuous functions of the 3D position x and camera viewing direction a>o. A conventional NeRF model encodesthe density field and the light field 140 compactly using a coordinate-based multilayer perceptron (MLP). Such an encoding, however, requires complex camera and illuminator arrays and large training datasets and results in poor accuracy.

[0032] Rather than directly encoding the light field, however, the model 130 encodes the direct illumination's transport function, the indirect illumination’s transport function, the diffuse component of the reflectance function (diffuse reflectance), and the specular component of the reflectance function (specular reflectance) using MLPs, as is shown in FIG. 2. The radiance emitted by an illumination source, however, is determined through measurement.

[0033] The generation of the model 130 is accomplished by deriving the light field 130 and a density field from a set of images (e.g., multiple images 110(a), 110(b), 110(c)). The densify field is a measure of how much light passes through a point in a scene including a human face being captured. The light field 140 is a measure of a brightness (e.g., a radiance) of the light at a position in a scene x and being viewed from a camera viewing direction Once the densify and light field 140 are determined, the densify and light field 140 are integrated along rays (e.g., as in volume rendering) defining position in the scene and view direction to produce the new image 150.Nevertheless, the light field 140 is not computed directly. Rather the model 130 decomposes the light field 140 into a first component 142 representing direct illumination and a second component 144 representing indirect illumination. That is, the first component 142 corresponds to a radiance term due to direct illumination LDwhich is dependent on the position in the scene x and camera viewing direction The second component 144 corresponds to a radiance term due to indirect illumination L, which is dependent on the position in the scene x.

[0034] The first component 142 is based on a radiance emitted by an illumination source, the direct illumination’s transport function, and a reflectance function. In some implementations, the first component 142 is represented as an integral of a product of the radiance emitted by an illumination source, the direct illumination’s transport function, and the reflectance function over an illumination angle. The reflectance function is further constrained to have the same value when the camera viewing direction and the illumination direction are swapped. The reflectance function is further decomposed into a specular component and a diffuse component. The specular reflectance is the portion of the reflectance due to directional, e.g.. mirror-like, reflections from smooth surfaces. The diffuse reflectance is the portion of the reflectance due to diffuse reflections from rough oruneven surfaces.

[0035] The second component 144 represents indirect illumination and is based on the radiance emitted by an illumination source and the indirect illumination’s transport function. In some implementations, the second component 144 is represented as an integral of a product of the radiance emitted by an illumination source and the indirect illumination’s transport function. Further mathematical details concerning the model 130 follow below.

[0036] In some implementations, the model 130 produces the new image 150 through volumetric rendering. Volumetric rendering is performed by integrating the light field 130 along a camera ray that correspond to the camera viewing direction. denote the position of the camera's optical center, and the origin of athat is cast into the scene. The radiance incident at an image pixel p from cameraviewing (ray) direction is then given in terms of the density field cr(x) £ B andthe light field 140Here is the accumulated transmittance term, which is also a function of the densityfield. This process is effectively equivalent to alpha-compositing colors along the ray. The integral in Eq. (1) can be efficiently estimated using ray marching.

[0037] The model 130 decomposes the light field 140 asThat is, the first component 142 corresponds to a radiance term due to direct illumination LDwhich is dependent on the position in the scene x and camera viewing direction <n0. The second component 144 corresponds to a radiance term due to indirect illumination Lj which is dependent on the position in the scene x.

[0038] Some implementations formulate the direct illumination term as an integral of a product of a reflectance function and a directly incident radiance over the unitsphere of directionsas follows:The integration may be carried out over the illumination sources indexed by eachon the unit sphere is the incident direct illumination’s transport function,accounting for transmittance (self-shadowing) and foreshortening effects. The termrepresents the radiance emitted by an illumination source along the illumination directiona>i. In some implementations, may may be known from the precalibration of theintensity and color of the light source.

[0039] The reflectance function f may be a general scattering distribution function. In physically -based volume rendering, this term may be defined by the volume’s albedo and phase function. The latter is the volumetric analog of a bidirectional scattering distribution function (BSDF). The model thus constrains this term to be reciprocal, satisfying the following property known as Helmholtz reciprocity:Helmholtz reciprocity means that the value of f remains the same if the orientations of camera and light, are swapped. Implementations leverage this physicalproperty to better constrain the model 130 during model fitting with a sparse array of camera view directions and illumination directions. Helmholtz reciprocity implies that non-symmetrical effects (e.g., foreshortening on surface-like parts) are modeled by the direct light transport function In addition, implementations further constrain f bydecomposing it into diffuse and directional (specular) parts, as follows:The diffuse component of the reflectance may be purely position-dependent andbehaves like an albedo field, modeling spatially-varying, wavelength-dependent light absorption. The specular component models directional scattering of light andalso captures effects such as specular highlights. For simplicity, implementations assume to be a scalar field (e.g., neutral color) that models dielectric reflectanceand thus focuses on representing human appearance.

[0040] Implementations model the scattering function f as a multiplayer perceptron (MLP), thus foregoing the need to explicitly decide between surface (bidirectional scattering function) and volume (phase function) rendering formalism. As a result, the model 130 generalizes to photorealistically render both surface-like parts (e.g.. skin) and volume-like areas with aggregate geometry (e.g., hair).

[0041] The contribution of the radiance term due to indirect illumination fy (x) that is due to interreflection or subsurface scattering is considered. In some implementations, the radiance term due to indirect illumination L;is diffuse and thus not a function of, and takes the following form:Here, the indirect transport function models both light transport and diffusereflectance throughout the multiple indirect bounces that are not modeled explicitly; this is in contrast to Eq. (3), where direct transport and reflectance are factorized into two fields. It is also noted tha is a scalar field with neutral color (attenuation) whileis a vector field that accounts for changing light color due to multiple scattering. Both transport functions are modulated by the predefined illumination intensity (weight) as in standard image-based relighting.

[0042] In some implementations, the direct transport function depends on anadditional per-pixel (ray) input where p denotes a pixel through which the rayhaving directio passes, that captures light visibility and foreshortening. That is, in suchimplementations, the direct transport function takes the formwherexpis the 3D point where the ray through pixel p first intersects a surface, n(xp) is the per- vertex surface normal obtained from the proxy 3D mesh, is the binaryvisibility (shadowing) of an OLAT source, and the inner product is between the illumination direction and the surface normal. This extra conditioning and guidance uses only simple light transport simulation with a precomputed 3D mesh, but no strong assumption on scene materials and reflectance. Furthermore, the extra conditioning makes this field easier to leam from the data, while also tolerating noisy inputs, leading to smoother lighting and shadowinterpolation. Ultimately, the extra conditioning allows for supplementing the incomplete supervision from real data: for missing OLAT directionsfor which there would be no data otherwise (ambiguous lighting), the simulation biases the model towards a physically plausible direct transport function

[0043] It is noted that the above derivations of assumed distant lighting.Nevertheless, the derivations may be adapted for rendering with near-point sources. As in conventional image-based relighting, it is noted that each OLAT condition describes global illumination effects and the direction serves as an index for retrieving (interpolating) thelighting condition. Whileis spatially constant for distant OLAT sources, for near sources a>i becomes spatially-varying:is a function of x and the position of the near OLAT source,Moreover, the radiance emitted by the OLAT source is attenuated according to the inverse square distance from xeto each point x in the scene’s 3D volume, as follows:At both training and test time, implementations render with near sources by simply substituting Eq. (8) into Eq. (3) and Eq. (6). At test time,may alternatively originate from the pixels of a latitude-longitude environment map, with per-pixel solid angles du>j.

[0044] During capture, implementations include starting by recording a frame of the subject with all lights on, followed by each OLAT setting, interleaving another full-on frame halfway through and at the end. As in previous OLAT-based methods, full-on frames are used to compute multi view stereo reconstruction of a 3D mesh, which can be leveraged as proxy geometry for the model 130; (2) compute optical flow between consecutive full-on frames and remove small motion of the captured subject; and (3) obtain a matting mask to segment foreground pixels for fitting the model 130.

[0045] As with a conventional NeRF, the above-defined fields in the model 130 are compactly parametrized using a sequence of MLPs, as shown in FIG. 2.

[0046] FIG. 2 is a diagram illustrating an example network architecture 200 of the model 130. The example network architecture 200 is configured to produce as output the quantities that enable computation of the radiance in Eq. (1) using a series ofMLPs

[0047] For example, the network architecture 200 includes a trunk MLP 211, a bottleneck MLP 216, and various MLPs whose final layers produce the quantities that enable computation of the radianceThe trunk MLP 211 is an MLP with eight layers of width 1024. The trunk MLP 211 takes as input the position x 210. As shown in FIG. 2, the output of the trunk MLP 211 is input into three MLPs: a density MLP 212, a diffuse MLP 214, and the bottleneck MLP 216.

[0048] The density MLP 212 is an MLP with a single layer of unit width. Its output layer is the density fieldIt is noted that the trunk MLP 211 and the density’ MLP 212 can be adopted directly from the baseline implementation of the conventional NeRF MLP architecture without changes.

[0049] The radiance MLP of the conventional NeRF that would correspond to the radiance fieldhowever, is replaced with four new MLPs representing the four new fields introduced above: the diffuse reflectancethe specular reflectance . the direct transport function , and the indirect transport function

[0050] For example, the diffuse MLP 214 is an MLP having a single layer of width 3. The output of the diffuse MLP 214 is the diffuse reflectance

[0051] The bottleneck MLP 216 is an MLP with a single layer of width 256. As shown in FIG. 2, the bottleneck MLP 216 is concatenated with three MLPs to produce the remaining three of the new fields.

[0052] The specular reflectance 224 is an output of an MLP that is aconcatenation of the bottleneck MLP 216 and a specular MLP 222. The specular MLP 222 is an MLP w ith tw o layers, one of which has width 128 and the other of which has unit width.

[0053] The direct transport function 234 is an output of an MLP that isa concatenation of the bottleneck MLP 216 and a direct transport MLP 232. The direct transport MLP 232 is an MLP with two layers, one of which has width 128 and the other of which has unit width.

[0054] The indirect transport functio 236 is an output of an MLP that is aconcatenation of the bottleneck MLP 216 and a direct transport MLP 232. The direct transport MLP 232 is an MLP with two layers, one of which has width 128 and the other of which has unit width.

[0055] To enable model 130 to be of high-quality, the light transport fields and reflectance fields encourage themodel fitting optimization to discover an intrinsic decomposition of the global illumination effects captured in real OLAT images. This decomposition is partially guided by the formulation of the model, which assigns chromaticity to particular fields that are also conditioned on different inputs.

[0056] In addition, enforcing the Helmholtz reciprocity relation in Eq. (4) constrains the problem in a physically plausible way and helps disambiguate the intrinsic decomposition by forcing the non-reciprocal light transport effects to be modeled by the other view-independent fields. During model fitting, Helmholtz reciprocity may be enforced by randomly swappingeach time is evaluated. Therefore, no explicitloss or additional complexity is added to the model.

[0057] Finally, the intrinsic radiance field decomposition is also guided by the minimization of the following compound loss:The rendering loss is the usual mean squared error between captured (real) RGB pixel colors and rendered colors Cp. after applying SRGB tone mapping T(-),The sum is carried out over the setof pixels p considering the captured real images from all camera view directions and all illumination directions. using Eq. (1)and the new radiance model for near point OLAT sources, Eq. (5)-(6). After discretization, the integration along a ray is performed hke in a conventional NeRF model,With the same density-dependent weightsbut new radiancesumming over illumination directions with as in Eq. (6). When fitting anOLAT image, the intensitie are set to zero for the other OLAT sources, which donot need to be evaluated.

[0058] In the second loss term is the weight of the distortion regularizerThe second loss term encourages the density field to be concentrated within a shortinterval along the camera ray. When fitting the density field, this regularizer is further complemented using a novel technique that leverages guidance from a 3D mesh, as detailed below.

[0059] A conventional NeRF model typically reconstructs a 3D density field by fitting to image datasets that often include over 100 viewpoints. For the model 130, however, the sparse capture setup provides a small set of views (e.g., 15) and shows several reconstruction artifacts in the estimated scene density and depth (i.e., “floaters”).

[0060] To improve density estimation with few views, implementations provide additional guidance for volumetric sampling, rendering, and fitting. Such guidance can be provided by leveraging proxy geometry obtained as a 3D mesh via multiview stereo reconstruction on the captured human portraits. As the 3D mesh has good accuracy, the 3D mesh can provide a strong prior on where the model 130 may concentrate its density, sampling, and modeling capacity to more efficiently and accurately fit the observed scene.

[0061] Some implementations involve restricting volumetric sampling andoptimization to within a smaller region of the volume, surrounding the 3D mesh, including all the observed surfaces on the human head, neck, and upper torso. This new optimization domain is referred to as the shell volume induced by the 3D mesh. This shell volume is defined by inside of an iso-surface 1 cm away from the mesh, extracted by first converting the 3D mesh into a signed distance field on a discrete grid and then performing surface extraction via marching cubes.

[0062] Compared to other density field regulanzers. the shell volume can bring the following advantages: (1) it can define a narrow interval Eq. (11) for efficientray marching at both fitting and inference times; (2) it can allow the MLPs of the model 130 to fully focus their modeling capacity near the observed surfaces, ignoring the background and avoiding the introduction of rendering artifacts in empty space (“floaters’’); (3) under OLAT illumination where the background is mostly dark, it can provide a natural mechanism for filtering the dark background pixels and hence avoid collapsing to zerodensity (that was observed with full images). It is found that the shell volume can effectively eliminate “floater” artifacts and yields high-quality novel view synthesis.

[0063] FIG. 3 is a diagram that illustrates example processing circuitry 320 of a computing device configured to generate a relightable NeRF model. The processing circuitry 320 is configured to determine the first component 142 and the second component 144 of the light field 140 in the relightable NeRF model 130.

[0064] The processing circuitry 320 includes a network interface 322, one or more processing units 324, and nontransitory memory 326. The network interface 322 includes, for example, Ethernet adaptors, Bluetooth adaptors, and the like, for converting electronic and / or optical signals received from the network to electronic form for use by the processing circuitry 320. The set of processing units 324 include one or more processing chips and / or assemblies. The memory 326 is a storage medium and includes both volatile memoiy (e.g., RAM) and non-volatile memory, such as one or more read only memories (ROMs), disk drives, solid state drives, and the like. The set of processing units 324 and the memory 326 together form part of the processing circuitry 320, which is configured to perform various methods and functions as described herein as a computer program product.

[0065] In some implementations, one or more of the components of the processing circuitry 320 can be, or can include processors (e.g., processing units 324) configured to process instructions stored in the memory 326. Examples of such instructions as depicted in FIG. 3 include an image manager 330, a model manager 340, and a new image manager 360.Further, as illustrated in FIG. 3, the memory 326 is configured to store various data, which is described with respect to the respective managers that use such data.

[0066] The image manager 330 is configured to cause the processing circuitry 320 to receive image data 332 representing a plurality of images captured from a plurality of perspectives and illuminated using a plurality of directions of at least one illumination source. In some implementations, the plurality of images are captured with the relatively small array of cameras and one-light-at-a-time (OLAT) illumination sources described above.

[0067] Accordingly, the image data 332 represents the plurality of images of an object (e.g., a human face) captured from a plurality of perspectives and illuminated using a plurality of directions of at least one illumination source. In some implementations, the image data 332 takes the form of the radiance as expressed in Eqs. (11) and (12). As show n in FIG. 3, the image data 332 includes perspective data 333, illumination data 334, visibility data 335, and source position data 336.

[0068] The perspective data 333 represent the directions from which theobject is viewed. These directions are determined, for example, from the angles at which the cameras of the relatively small array are oriented.

[0069] The illumination data 334 represent the directions from which the objectis illuminated. These directions are determined, for example, from the angles at which the relatively small array of OLAT illumination sources are oriented.

[0070] The visibility data 335 represent the light visibility The light visibility isdefined by Eq. (7).

[0071] The model manager 340 is configured to cause the processing circuitry 320 to generate a model (e g., the model 130) in the form of model data 342 based on the image data 332. For the example, the model manager 340 is configured to perform the calculations pertaining to the MLPs shown in FIG. 2 toproduce the density field the diffusereflectance the specular reflectance. the direct transport function , and the indirect transport function

[0072] The model data 342 represents the model 130. The model data 342 as shown in FIG. 3 includes MLP data 343, density data 351, and light field data 352.

[0073] The MLP data 343 represents the MLPs shown in FIG. 2 that generate the output densityand the light field quantities from the image data inputs. The MLP data 343 is generated by the model manager 340. The MLP data 343 includes trunk MLP data344, density MLP data 345, diffuse MLP data 346, bottleneck MLP data 347, specular MLP data 348. direct transport MLP data 349, and indirect transport MLP data 350.

[0074] Trunk MLP data 344 represents a trunk MLP (e.g., trunk MLP 211). The trunk MLP takes a position x as input and outputs to a density MLP (e.g., density’ MLP 212), a diffuse MLP (e.g., diffuse MLP 214), and a bottleneck MLP (e.g., bottleneck MLP 216). In some implementations, the trunk MLP has eight layers of width 1024.

[0075] Density MLP data 345 represents the density- MLP. The density' MLP takes as input the output of the trunk MLP data and outputs the densityThe density MLP has a single layer of unit width as the density is a scalar.

[0076] Diffuse MLP data 346 represents the diffuse MLP. The diffuse MLP takes as input the output of the trunk MLP data and outputs the diffuse reflectance The diffuseMLP has a single layer of width three as the diffuse reflectance is a three-dimensional vector.

[0077] Bottleneck MLP data 347 represents the bottleneck MLP. The bottleneck MLP takes as input the output of the trunk MLP data. The bottleneck MLP has a single layer of width 256. The bottleneck MLP is concatenated with a specular MLP (e.g., specular MLP 222), a direct transport MLP (e.g., direct transport MLP 232), and an indirect transport MLP (e.g., indirect transport MLP 242).

[0078] The specular MLP data 348 represents the specular MLP. The specular MLP has two layers, one of width 128, the other of unit width. The bottleneck MLP is concatenated with specular MLP. The input to the concatenated bottleneck MLP / specular MLP is a position x (through the trunk MLP input into the bottleneck MLP), a view directionand an illumination direction The output of the concatenatedbottleneck MLP / specular MLP is the specular reflectance

[0079] The direct transport MLP data 349 represents the direct transport MLP. The direct transport MLP has two layers, one of width 128, the other of unit width. The bottleneck MLP is concatenated with the direct transport MLP. The input to the concatenated bottleneck MLP / direct transport MLP is a position x (through the trunk MLP input into the bottleneck MLP), an illumination directionand a light visibility Theoutput of the concatenated bottleneck MLP / direct transport MLP is the direct transport function

[0080] The indirect transport MLP data 350 represents the indirect transport MLP. The indirect transport MLP has two layers, one of width 16, the other of width three. Thebotleneck MLP is concatenated with the indirect transport MLP. The input to the concatenated botleneck MLP / indirect transport MLP is a position x (through the trunk MLP input into the botleneck MLP) and an illumination directionThe output of the concatenated botleneck MLP / indirect transport MLP is the indirect transport function

[0081] The density data 351 represents the output of the density MLP, i.e., the density output by the density MLP.

[0082] The light field data 352 represents a light field (e.g., light field 140) As shown in FIG. 3, the light field data 352 includes first component data 353,second component data 357. and radiance data 359.

[0083] The first component data 353 represents a radiance due to direct illumination (e.g., first component 142) The first component data 353 includes diffusereflectance data 354, specular reflectance data 355, and direct transport data 356. The diffuse reflectance data 354 represents the diffuse reflectance output by the diffuseMLP. The specular reflectance data 355 represents the specular reflectanceoutput by the concatenated bottleneck MLP / specular MLP. The direct transport data 356 represents the direct transport function output by the concatenated botleneckMLP / direct transport MLP.

[0084] The second component data 357 represents a radiance due to indirect illumination (e.g., second component 144) L The second component data 357 includesindirect transport data 358. The indirect transport data 358 represents the indirect transport function output by the concatenated botleneck MLP / indirect transport MLP.

[0085] The radiance data 359 represents the radianceemited by an OLAT source along the illumination directionThe radiance data 359 is known from the precalibration of the intensity and color of the light source.

[0086] The new image manager 360 is configured to generate new image data 362 representing a new image of the object that is determined based on the density data 351 and the light field data 352. The new image data 362 includes perspective data 364 representing a new direction from which the object is viewed. The illumination data represents a new direction from which the object is illuminated. The new image of the object has a perspective and illumination profile that may be different from the perspectives and illumination profiles of the plurality of images represented by the image data.

[0087] The components (e.g., modules, processing units 324) of processing circuitry320 can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms) that can include one or more types of hardware, software, firmware, operating systems, runtime libraries, and / or so forth. In some implementations, the components of the processing circuitry 320 can be configured to operate within a cluster of devices (e.g., a sen' er farm). In such an implementation, the functionality' and processing of the components of the processing circuitry 320 can be distributed to several devices of the cluster of devices.

[0088] The components of the processing circuitry 320 can be, or can include, any ty pe of hardware and / or software configured to process attributes. In some implementations, one or more portions of the components shown in the components of the processing circuitry- 320 in FIG. 3 can be, or can include, a hardware-based module (e.g., a digital signal processor (DSP), a field programmable gate array (FPGA), a memory), a firmware module, and / or a software-based module (e.g., a module of computer code, a set of computer- readable instructions that can be executed at a computer). For example, in some implementations, one or more portions of the components of the processing circuitry 320 can be. or can include, a software module configured for execution by at least one processor (not shown). In some implementations, the functionality of the components can be included in different modules and / or different components than those shown in FIG. 3, including combining functionality illustrated as two components into a single component.

[0089] Although not shown, in some implementations, the components of the processing circuitry 320 (or portions thereof) can be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, and / or so forth. In some implementations, the components of the processing circuitry 320 (or portions thereof) can be configured to operate within a network. Thus, the components of the processing circuitry 320 (or portions thereof) can be configured to function within various types of network environments that can include one or more devices and / or one or more server devices. For example, the network can be, or can include, a local area network (LAN), a wide area network (WAN), and / or so forth. The network can be. or can include, a wireless network and / or wireless network implemented using, for example, gateway devices, bridges, switches, and / or so forth. The network can include one or more segments and / or can have portions based on various protocols such as Internet Protocol (IP) and / or a proprietary protocol. The network can include at least a portion of the Internet.

[0090] In some implementations, one or more of the components of the searchsystem can be, or can include, processors configured to process instructions stored in a memory. For example, image manager 330 (and / or a portion thereof), model manager 340 (and / or a portion thereof), and new image manager 360 (and / or a portion thereof) are examples of such instructions.

[0091] In some implementations, the memory 326 can be any type of memory such as a random-access memory, a disk drive memory, flash memory, and / or so forth. In some implementations, the memory 326 can be implemented as more than one memory component (e g., more than one RAM component or disk drive memory) associated with the components of the processing circuitry 320. In some implementations, the memory 326 can be a database memory. In some implementations, the memory 326 can be, or can include, a non-local memory. For example, the memory 326 can be, or can include, a memory shared by multiple devices (not shown). In some implementations, the memory 326 can be associated with a server device (not shown) within a network and configured to serve the components of the processing circuitry 320. As illustrated in FIG. 3, the memory 326 is configured to store various data, including image data 332 and model data 342.

[0092] FIG. 4 is a flow chart illustrating a method 400 of generating a model for performing novel view synthesis with lighting. The method 400 may be performed with processing circuitry , e.g., processing circuitry 320 as shown in FIG. 3.

[0093] At 402, an image manager (e.g., image manager 330) captures a plurality of images of an object (e.g., images 110(a), 110(b), 110(c)), the plurality of images being captured from a plurality of perspectives and using a plurality of directions of at least one illumination source (e.g., directions of illumination sources 120(a), 120(b), 120(c)).

[0094] At 404, a model manager (e.g., model manager 340) defines a light field (e.g., light field 140) based on the plurality of images of the object, the light field having a first component (e.g., first component 142) and a second component (e.g., second component 144), the first component representing direct illumination (e.g., L ) and the secondcomponent representing indirect illumination (e.g., Ltyx)).

[0095] At 406, a new image manager (e.g., new image manager 360) generates a new image of the object (e.g., new image 150) from a new perspective, the new image of the object being illuminated based on the first component and the second component.

[0096] Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments. Example embodiments, however, may be embodied in many alternate forms and should not be construed as limitedto only the embodiments set forth herein.

[0097] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes," and / or "including." when used in this specification, specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0098] It will be understood that when an element is referred to as being "coupled," "connected," or "responsive" to, or "on," another element, it can be directly coupled, connected, or responsive to, or on, the other element, or intervening elements may also be present. In contrast, when an element is referred to as being "directly coupled," "directly connected," or "directly responsive" to, or "directly on," another element, there are no intervening elements present. As used herein the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0099] Spatially relative terms, such as "beneath," "below," "lower," "above," "upper," and the like, may be used herein for ease of description to describe one element or feature in relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the term "below" can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 70 degrees or at other orientations) and the spatially relative descriptors used herein may be interpreted accordingly.

[0100] Example embodiments of the concepts are described herein with reference to cross-sectional illustrations that are schematic illustrations of idealized embodiments (and intermediate structures) of example embodiments. As such, variations from the shapes of the illustrations as a result, for example, of manufacturing techniques and / or tolerances, are to be expected. Thus, example embodiments of the described concepts should not be construed as limited to the particular shapes of regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing. Accordingly, the regions illustratedin the figures are schematic in nature and their shapes are not intended to illustrate the actual shape of a region of a device and are not intended to limit the scope of example embodiments.

[0101] It will be understood that although the terms "first" "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a "first" element could be termed a "second" element without departing from the teachings of the present embodiments.

[0102] Unless otherwise defined, the terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which these concepts belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and / or the present specification and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0103] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover such modifications and changes as fall within the scope of the implementations. It should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and / or sub-combinations of the functions, components, and / or features of the different implementations described.

Claims

WHAT IS CLAIMED IS:

1. A method, comprising: capturing a plurality of images of an object, the plurality of images being captured from a plurality of perspectives and using a plurality of directions of at least one illumination source; defining a light field based on the plurality of images of the object, the light field having a first component and a second component, the first component representing direct illumination and the second component representing indirect illumination; and generating a new image of the object from a new perspective, the new image of the object being illuminated based on the first component and the second component.

2. The method as in claim 1, wherein defining the light field includes: decomposing the light field into the first component and the second component according to a model.

3. The method as in claim 2, wherein the first component of the light field depends on a position in a scene in which the light field is defined and a direction at which a camera is viewing the scene, and the second component of the light field depends on the position in the scene.

4. The method as in claim 3, wherein defining the light field includes: defining the first component of the light field based on a product of a first function and a second function, the first function representing a reflectance of the direct illumination and the second function being a product of a direct transport function of the direct illumination and a radiance of the at least one illumination source.

5. The method as in claim 4, wherein the first function depends on the position in the scene, the direction at which the camera is viewing the scene, and a direction of the plurality of directions of the at least one illumination source, andwherein the second function depends on the position in the scene and the direction of the plurality of directions of the at least one illumination source.

6. The method as in claim 5, wherein the direct transport function further depends on a product of a visibility of the object at a point on a surface of the object illuminated from the direction of the plurality of directions of the at least one illumination source and an inner product of the direction of the plurality of directions of the at least one illumination source and a normal to the point on the surface of the obj ect.

7. The method as in claim 5, wherein the reflectance of the direct illumination remains the same if the direction at which the camera is viewing the scene and the direction of the at least one illumination source are sw apped.

8. The method as in claim 5, wherein the reflectance of the direct illumination is a sum of a first component and a second component, the first component representing a diffuse reflected illumination, the second component representing specular reflected illumination.

9. The method as in claim 8, wherein the first component depends on the position in the scene, and the second component depends on the position in the scene, the direction at which the camera is viewing the scene, and a direction of the plurality of directions of the at least one illumination source.

10. The method as in claim 4, wherein the at least one illumination source is located at a position with respect to the scene, and the radiance of the at least one illumination source further depends on the position of the at least one illumination source w ith respect to the scene.

11. The method as in claim 3, wherein defining the light field includes: defining the second component of the light field based on a product of an indirect transport function of the indirect illumination and a radiance of the at least one illumination source.

12. The method as in claim 11, wherein the at least one illumination source is located at a position with respect to the scene, and the radiance of the at least one illumination source further depends on the position with respect to the scene.

13. The method as in claim 11, wherein the second component depends on position in the scene and the direction of the plurality of directions of the at least one illumination source.

14. The method as in claim 2, wherein the model includes a compound loss based on a rendering loss and a distortion loss, the rendering loss being a mean squared error between a captured pixel colors after applying a tone mapping and rendered pixel colors after applying the tone mapping.

15. A computer program product comprising a non transitory storage medium, the computer program product including code that, when executed by processing circuitry, causes the processing circuitry to perform a method, the method comprising: capturing a plurality of images of an object, the plurality' of images being captured from a plurality of perspectives and using a plurality of directions of at least one illumination source; defining a light field based on the plurality of images of the object, the light field having a first component and a second component, the first component representing direct illumination and the second component representing indirect illumination; and generating a new image of the object from a new perspective, the new image of the object being illuminated based on the first component and the second component.

16. The computer program product as in claim 16. wherein defining the light field includes: decomposing the light field into the first component and the second component according to a model.

17. The computer program product as in claim 16. wherein the first component of the light field depends on a position in a scene in which the light field is defined and a direction at which a camera is viewing the scene, and the second component of the light field depends on the position in the scene.

18. An electronic apparatus, the electronic apparatus comprising: memory; and processing circuitry coupled to the memory, the processing circuitry being configured to: capture a plurality of images of an object, the plurality of images being captured from a plurality of perspectives and using a plurality of directions of at least one illumination source; define a light field based on the plurality of images of the object, the light field having a first component and a second component, the first component representing direct illumination and the second component representing indirect illumination; and generate a new image of the object from a new perspective, the new image of the object being illuminated based on the first component and the second component.

19. The electronic apparatus as in claim 18, wherein the processing circuitry is further configured to: decompose the light field into the first component and the second component according to a model.

20. The electronic apparatus as in claim 18, wherein the first component of the light field depends on a position in a scene in which the light field is defined and a direction at which a camera is viewing the scene, and the second component of the light field depends on the position in the scene.

Images