Dark Flash Normal Camera

Abstract

A technique for estimating surface normals and reflectance from a poorly illuminated image includes using an image illuminated with near-infrared (NIR) radiation in addition to an RGB image of an object from a set of objects to determine an albedo map and a surface normal map for performing relighting of the image, the image being captured with NIR radiation from essentially the same viewpoint as the RGB image was captured. In some implementations, a prediction engine takes as input a single RGB image and a single NIR image and estimates surface normals and reflectance from the object.

Description

[Technical field]

[0001] REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application claiming priority to U.S. Provisional Patent Application No. 63 / 198,736 entitled "A DARK FLASH NORMAL CAMERA," filed November 9, 2020, the contents of which are incorporated herein by reference in their entirety. This application also is a nonprovisional application claiming priority to U.S. Provisional Patent Application No. 63 / 198,836 entitled "A DARK FLASH NORMAL CAMERA," filed November 16, 2020, the contents of which are incorporated herein by reference in their entirety.

[0002] Technical Field The present description relates to performing relighting of images captured in poor lighting conditions, for example in mobile photography and videography applications. [Background technology]

[0003] background Some mobile photography and videography is often performed under poorly controlled lighting, which results in low-quality images and degrades the performance of downstream image processing and computer vision algorithms. Controlling the visible lighting in the environment or compensating for it with a flash is often too difficult or too disruptive to be practical. Summary of the Invention [Means for solving the problem]

[0004] overview The implementations described herein relate to estimating high-quality normal and albedo maps of scenes showing people (faces and torsos) in poor lighting conditions, i.e. so-called dark flash images, by supplementing available visible spectrum illumination with a single near-infrared light source and camera. The implementations herein take as input a monochromatic image captured under arbitrary visible illumination and a single dark flash image captured under controlled NIR illumination at the same viewpoint, and compute normal and albedo maps of the scene. Because capturing a ground truth normal map of a face is difficult, the implementations herein include a novel training technique that combines information from multiple noisy sources, specifically stereo and photometric shading cues. The performance of the implementations is evaluated over a range of subjects and lighting conditions.

[0005] In one general aspect, a method can include receiving image training data representing a plurality of color (RGB) images and a plurality of near-infrared (NIR) images, each of the plurality of RGB images captured with a visible spectrum illumination source and each of the plurality of NIR images captured with a NIR illumination source, the plurality of RGB images and the plurality of NIR images each including a subset of the plurality of RGB images and a subset of the plurality of NIR images, each subset of the plurality of RGB images and each subset of the plurality of NIR images including images of a respective object of a set of objects in poses through a plurality of lighting conditions. The method can also include generating a prediction engine based on the image training data, the prediction engine configured to generate an estimated surface normal map of the user and an estimated reflectance map of the user from the single RGB image of the user and the single NIR image of the user, the single RGB image of the user and the single NIR image of the user being captured within a time period of less than a threshold time period and from viewpoints that differ by less than a threshold viewpoint.

[0006] In another general aspect, a computer program product includes a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit of a computing device, causes the processing circuit to perform a method. The method can include receiving image training data representing a plurality of color (RGB) images and a plurality of near-infrared (NIR) images, each of the plurality of RGB images being captured with a visible spectrum illumination source and each of the plurality of NIR images being captured with a NIR illumination source, the plurality of RGB images and the plurality of NIR images respectively including a subset of the plurality of RGB images and a subset of the plurality of NIR images, each subset of the plurality of RGB images and each subset of the plurality of NIR images including images of a respective object of a set of objects in poses through a plurality of lighting conditions. The method can also include generating a prediction engine based on the image training data, the prediction engine configured to generate an estimated surface normal map of the user and an estimated reflectance map of the user from the single RGB image of the user and the single NIR image of the user, the single RGB image of the user and the single NIR image of the user being captured within a time period of less than a threshold time period and from viewpoints that differ by less than a threshold viewpoint.

[0007] In another general aspect, an electronic device includes a memory and a control circuit coupled to the memory. The control circuit can be configured to receive image training data representing a plurality of color (RGB) images and a plurality of near-infrared (NIR) images, each of the plurality of RGB images being captured with a visible spectrum illumination source and each of the plurality of NIR images being captured with a NIR illumination source, the plurality of RGB images and the plurality of NIR images each including a subset of the plurality of RGB images and a subset of the plurality of NIR images, each subset of the plurality of RGB images and each subset of the plurality of NIR images including images of a respective object of a set of objects in poses through a plurality of lighting conditions. The control circuit can also be configured to generate a prediction engine based on the image training data, the prediction engine configured to generate an estimated surface normal map of the user and an estimated reflectance map of the user from the single RGB image of the user and the single NIR image of the user, the single RGB image of the user and the single NIR image of the user being captured from viewpoints that differ by less than a threshold viewpoint within a time period less than a threshold time period.

[0008] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an exemplary electronic environment in which the improved techniques described herein may be implemented. [Diagram 2] FIG. 1 illustrates an example prediction engine configured to estimate surface normals and reflectance in an image. [Diagram 3] FIG. 1 illustrates an example system for generating RGB and NIR images for training a prediction engine. [Figure 4A] 11 is a plot illustrating an exemplary effect on angular error of using a NIR image in addition to an RGB image. [Figure 4B]11 is a plot illustrating an exemplary effect on angular error of using a NIR image in addition to an RGB image. [Figure 4C] 11 is a plot illustrating an exemplary effect on angular error of using a NIR image in addition to an RGB image. [Diagram 5] FIG. 1 is a flowchart illustrating an example method for estimating surface normals and reflectance in an image, in accordance with a disclosed implementation. [Figure 6] FIG. 1 illustrates an example of a computing device and a mobile computing device that can be used to implement the described techniques. [Figure 7] FIG. 1 illustrates an example of a distributed computing device that can be used to implement the described techniques. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Detailed Description Some conventional approaches for relighting poorly illuminated images include performing a "shape from shading" operation on the color image, which recovers shape from gradual changes in shading in the image. Other conventional approaches for relighting poorly illuminated images include eigenimage decomposition techniques, which factorize the image as a product of a reflectance image and a shading image.

[0011] A technical problem with the above-mentioned conventional techniques for relighting poorly lit images is that such techniques are not suitable for estimating surface normal and albedo maps of surfaces in an image that are used for image relighting. For example, using any of these techniques to determine image relighting may result in an effectively intractable problem of estimating surface normal and albedo maps. These techniques may be made more tractable with the availability of ground truth reflectance maps, but such ground truth maps may not be available in many mobile photography situations.

[0012] According to implementations described herein, a technical solution to the above technical problem includes using an image illuminated with near-infrared (NIR) radiation in addition to an RGB image of an object of a set of objects to determine an albedo map and a surface normal map to perform relighting of the image, the image being captured with NIR radiation from essentially the same viewpoint as the RGB image was captured. In some implementations, a prediction engine takes as input a single RGB image and a single NIR image and estimates surface normals and reflectance from the object. In some implementations, the reflectance includes an albedo component and a specular component.

[0013] In some implementations, the improved technique includes receiving image training data representing a plurality of color (RGB) images and a plurality of near-infrared (NIR) images, each of the plurality of RGB images being captured with a white light illumination source and each of the plurality of NIR images being captured with a NIR illumination source, the plurality of RGB images and the plurality of NIR images each including a subset of the plurality of RGB images and a subset of the plurality of NIR images, each subset of the plurality of RGB images and each subset of the plurality of NIR images including images of a respective object of a set of objects in poses through a plurality of lighting conditions, the improved technique further including generating a prediction engine based on the image training data, the prediction engine configured to generate an estimated surface normal map of the user and an estimated reflectance map of the user from the single RGB image of the user and the single NIR image of the user, the single RGB image of the user and the single NIR image of the user being captured simultaneously from essentially the same viewpoint.

[0014] In some implementations, the improved technique includes, prior to generating the prediction engine, performing a semantic segmentation operation on the RGB images of a corresponding subset of the plurality of RGB images for each of a set of objects to generate a label image for that object, the label image having a specified number of classes into which each of a plurality of pixels of the RGB image is categorized, the label image for that object being included in the image training data.

[0015] In some implementations, a single RGB image of the user and a single NIR image of the user are captured essentially simultaneously, and in some implementations, in this scenario, the RGB lighting used in the image capture does not have to be generated by an RGB lighting source, but can be generated using any lighting.

[0016] In some implementations, each subset of the plurality of RGB images and each subset of the plurality of NIR images are captured using an image capture configuration, each of which includes a plurality of RGB sources, a plurality of NIR illumination sources, and an RGB and NIR illumination detector, and the plurality of collocated RGB and NIR illumination sources are arranged in a geometric pattern surrounding the RGB and NIR illumination detector. In some implementations, the plurality of RGB sources and the plurality of NIR illumination sources are arranged at corners of a rectangle surrounding the RGB and NIR illumination detector. In some implementations, each of the plurality of illumination conditions includes one of the plurality of RGB illumination sources, one of the plurality of NIR illumination sources that generates illumination, all others of the plurality of RGB illumination sources, and all others of the plurality of NIR illumination sources that do not generate illumination. In some implementations, the RGB and NIR illumination detector includes a first NIR camera, a second NIR camera, and an RGB camera, and the first NIR camera and the RGB camera are misaligned by an amount less than a specified misalignment threshold. In some implementations, the image capture arrangement further includes an NIR dot projector configured to project a dot speckle pattern onto the object, the dot speckle pattern being interleaved in time with illumination emitted by an NIR illumination source among the multiple NIR illumination sources.

[0017] In some implementations, the prediction engine includes a first branch and a second branch, the first branch configured to generate a surface normal map and the second branch configured to output a predicted reflectance map. In some implementations, the prediction engine includes a neural network having a unet encoder-decoder architecture with skip-level connections, the unet encoder-decoder architecture includes an encoder and a decoder, the encoder includes a set of blocks, each of the set of blocks includes a set of convolutional layers and a set of ReLU activation layers, and the decoder is configured to output the surface normal map in the first branch and the predicted reflectance map in the second branch. In some implementations, generating the prediction engine includes generating a photometric loss based on rendering from the estimated surface normal map and the estimated reflectance map under a lighting condition of the plurality of lighting conditions. In some implementations, the estimated reflectance map includes a diffuse component and a specular component, and generating the photometric loss includes generating a diffuse component of the estimated reflectance map using a Lambertian reflectance model and generating a specular component of the estimated reflectance map using a Blinn-Phong Bidirectional Reflectance Distribution Function (BRDF). In some implementations, generating the photometric loss includes generating a binary shadow map based on the stereo depth map and a position of a light source used in generating an RGB image of the plurality of RGB images, generating an observed intensity map based on the estimated reflectance map, and generating a Hadamard product of the binary shadow map and the difference between the observed intensity map and the RGB image as the photometric loss. In some implementations, the improved technique further includes obtaining a stereo depth map, performing a smoothing operation on the stereo depth map to generate a smoothed stereo depth map, and generating a stereo loss based on a gradient of the estimated surface normal map and the smoothed stereo depth map.In some implementations, generating the stereo loss includes generating an L1 norm of a difference between the estimated surface normal map and a gradient of the smoothed stereo depth map as the L1 vector loss, generating a dot product of the estimated surface normal map and the gradient of the smoothed stereo depth map as the angle loss, and generating a difference between the L1 vector loss and the angle loss as the stereo loss.

[0018] A technical advantage of the disclosed implementations is that application of the improved techniques, unlike conventional approaches, results in a tractable problem for determining albedo and surface normal maps for use in image relighting. Reducing the problem of generating such maps to a tractable problem provides a robust solution to the image relighting problem described above when images are taken under poor lighting conditions.

[0019] Moreover, this robust solution is provided when no ground truth is available due to a novel prediction engine that uses NIR illumination in addition to conventional visible spectrum illumination. Note that many mobile devices have the capability to take photos using both visible spectrum illumination and NIR illumination, which are taken from the same orientation. In addition, such visible spectrum illuminated images and NIR illuminated images may be acquired essentially simultaneously.

[0020] 1 is a diagram showing an exemplary electronic environment 100 in which the above mentioned technical solutions may be implemented. A computer 120 is configured to train and run a prediction engine configured to estimate surface normals and reflectances from image data.

[0021] The computer 120 includes a network interface 122, one or more processing units 124, and a memory 126. The network interface 122 includes, for example, an Ethernet adapter, a token ring adapter, etc., for converting electronic and / or optical signals received from a network into an electronic format for use by the computer 120. The set of processing units 124 includes one or more processing chips and / or assemblies. The memory 126 includes both volatile memory (e.g., RAM) and non-volatile memory, such as one or more ROMs, disk drives, solid state drives, etc. Together, the set of processing units 124 and the memory 126 form control circuitry that is configured and arranged to perform various methods and functions as described herein.

[0022] In some implementations, one or more of the components of computer 120 may be or include a processor (e.g., processing unit 124) configured to process instructions stored in memory 126. Examples of such instructions as shown in Figure 1 include an image acquisition manager 130, a semantic segmentation manager 140, and a prediction engine manager 150. Additionally, as shown in Figure 1, memory 126 is configured to store various data, which will be described with respect to each manager that uses such data.

[0023] Image acquisition manager 130 is configured to receive image training data 131. In some implementations, image acquisition manager 130 receives image training data 131 via network interface 122, i.e., over a network (such as network 190) from display device 170. In some implementations, image acquisition manager 130 receives image training data 131 from local storage (e.g., disk drive, flash drive, SSD, etc.).

[0024] In some implementations, the image acquisition manager 130 is further configured to crop and resize face images from the image training data 131 to generate portraits at a standard size. By cropping and resizing the images to a standard size so that each subject's face is in approximately the same position, training of predicted images becomes more robust.

[0025] The image training data 131 is a set of face images in different lighting conditions, including types of light (e.g., visible spectrum / RGB and NIR) and positions of light relative to the subject. Exemplary acquisition of images under such conditions is discussed in more detail with respect to FIG. 3. In some implementations, the RGB color channels may be replaced by other color channels, e.g., YUV, Y'UV, YCbCr, Y'IQ.

[0026] 1, image training data 131 includes multiple RGB images 132(1), ..., 132(M), where M is the number of RGB images in image training data 131. Each RGB image, e.g., image 132(1), includes pixel data 133(1) that indicates the intensity and / or color per pixel.

[0027] 1, image training data 131 includes multiple NIR images 134(1)...134(N), where M is the number of NIR images in image training data 131. Each NIR image, such as image 134(1), includes pixel data 135(1) that indicates the intensity per pixel. In some implementations, N=M.

[0028] In some implementations, image training data 131 includes NIR ring light image 136. In some implementations, NIR ring light image 136 is captured using a NIR ring light source that surrounds a NIR detector (e.g., a camera). This captured image is in the same pose as used in images 131(1..M). In some implementations, NIR ring light image 136 is captured using a NIR light source that does not surround the NIR detector, but is rather proximate to the NIR detector, i.e., off to the side. In such implementations, the NIR ring light may have a shape other than a ring, i.e., a small disk, a square, etc.

[0029] In some implementations, the image training data 131 includes an illuminant depth map 138 that represents an image of a person as a set of distances from a light source to the person's surface. Such a depth image 138 may be captured essentially simultaneously and in the same pose as the RBG images 131 (1..M) and / or NIR images 133 (I..N). In some implementations, the illuminant depth map 138 is a stereo depth map acquired using a stereo dot projector when the RGB images are acquired, as discussed in further detail in FIG. 3.

[0030] The semantic segmentation manager 140 is configured to perform a semantic segmentation operation on at least one of the RGB images 132 (1 ..M) to generate semantic segmentation data 142. In some implementations, the semantic segmentation operation involves using a neural network with a convolutional hidden layer and an output segmentation layer. In some implementations, the semantic segmentation operation involves using an encoder / decoder structure where the spatial resolution of the input is downsampled to develop a low-resolution feature mapping that is trained to be highly efficient at distinguishing between classes, and the feature representation may then be upsampled to a full-resolution segmentation map.

[0031] The semantic segmentation data 142 represents a semantic segmentation of the image into a specified number of classes, i.e., a segmentation map 144. Such a segmentation map 144 may be used as an auxiliary input to the prediction engine to help the engine determine image shape and reflectance. In some implementations, the segmentation is a 6-class segmentation, i.e., there are 6 classes of pixels in each segmentation map.

[0032] The prediction engine training manager 150 represents a prediction engine configured to generate prediction engine data 153, which represents data generated by the prediction engine manager 150 to generate surface normal and reflectance maps for images illuminated with visible spectrum and NIR illumination. In some implementations, the prediction engine manager 150 is configured to perform training operations on the image training data 131 and implement a loss function tuned to minimize an amount of loss associated with generating surface normal and reflectance maps from the visible spectrum / RGB and NIR images. As shown in FIG. 1, the prediction engine manager includes an encoder 151 and a decoder 152.

[0033] In some implementations, the prediction engine includes a neural network having a unet encoder-decoder architecture with skip-level connections, the unet encoder-decoder architecture including an encoder 151 and a decoder 152. In such implementations, the encoder 151 includes a set of blocks, each of which includes a set of convolutional layers and a set of ReLU activation layers.

[0034] The encoder 151 is configured to take the image training data 131 as input and generate parametric values ​​for the fully connected layers that are input to the decoder 152. The decoder 152 is configured to take the parametric values ​​generated by the encoder 151 as input and generate prediction engine data 153. For example, the encoder 151 takes the RGB images 132 (1..M), the NIR images 134 (1..N), the thematic segmentation data 142, and the illuminant depth map 138, and generates intermediate quantities that are used by the decoder 152 to generate estimated surface normals (e.g., estimated normal data 156). The cost function in this case reflects, for example, the difference between the estimated normal data 156 and surface normals of the same object and pose obtained in another way, for example, using stereo images of the same object. Further details regarding stereo imaging are described with respect to FIG. 3.

[0035] The prediction engine, represented by prediction engine manager 150, is configured to generate not only estimated surface normals of an object from the RGB and NIR images, but also an estimated reflectance map of the object from those images. To perform this estimation of both the surface normals and the reflectance map, decoder 152 is configured to output the surface normal map in a geometry branch and the reflectance map in a reflectance branch.

[0036] Prediction engine data 153 represents data generated by prediction engine manager 150 to generate surface normal and reflectance maps for images illuminated with visible spectrum and NIR illumination. As shown in FIG. 1, prediction engine data 153 includes geometry branch data 154, reflectance branch data 155, and rendering data 161.

[0037] Geometry branch data 154 includes data related to estimating surface normals of objects in the RGB and NIR images. Decoder 152 is configured to output estimated normal data 156 representing surface normals as a vector map, i.e., a three-dimensional vector at each pixel classified as an object, onto pixels associated with the object's surface.

[0038] However, in training the prediction engine, the estimated normal data 156 is only one component of the prediction engine training. To perform the training, a loss function is required. In this case, the loss function is defined using both the estimated normal data 156 and the reference data. As shown in FIG. 1, the reference data includes stereo normal data 157, and in some implementations, the stereo normal data 157 includes or is the light source depth map 138. The prediction engine manager 150 is configured to generate stereo loss data 158 based on the estimated normal data 156 and the stereo normal data 157.

[0039] Reflectance branch data 155 includes data related to estimating reflectance component data from surfaces in images of people. For example, reflectance may include an albedo component and a specular reflectance component. Albedo is defined as diffuse reflectance, i.e., the ratio of diffuse radiant flux emitted by a surface to the radiant flux density received by the surface; this is in contrast to the specular radiant flux reflected by the surface. Thus, as shown in FIG. 1, reflectance branch data 155 includes estimated albedo data 159 and estimated specular reflectance data 160.

[0040] The prediction engine uses the reflectance branch data 155 to estimate another component of the loss for training the prediction engine, namely the photometric loss. The prediction engine estimates the photometric loss by applying a rendering model to the reflectance component maps represented by the estimated albedo data 159 and the estimated specular reflectance data 160 to generate diffuse and specular image layers of the image, which, when added together, result in an estimated image of the object. The photometric loss is defined as the difference between the estimated image and one of the input images 132 (1..M) or 134 (1..N). By optimizing the photometric and stereo losses, the prediction engine acts as an image formation model that connects its output to images of the scene taken under known point illumination.

[0041] Thus, the rendering data 161 includes illumination model data 162 that represents a radiometric reflectance model that connects the output of the decoder 152 to an image taken under known lighting conditions. For example, a reflectance function f is introduced that gives the ratio of reflected to incident light for a particular unit length light vector l, viewing vector v, surface normal n, four-channel (RGB NIR) albedo α, scalar specular intensity ρ, and specular exponent m as follows:

[0042]

number

[0043] The observed intensity at a pixel due to a point light is given by:

[0044]

number

[0045] where L is the light intensity. The reflected intensity is not observed from enough unique light directions at each pixel to estimate all of the parameters in equation (1). In some implementations, to address this issue, the specular exponent is specified to be m=30, based on previous measurements of human skin and our own observations, and we estimate only n, α, and ρ. The geometric quantities l and v, as well as the light intensity L, are determined by a calibration procedure.

[0046]

number

[0047] Rather than relying on ground truth normal or reflectance data to supervise the training of the prediction engine, the stereo loss data 158 and photometric loss data 165 described above are combined.

[0048] The stereo loss, represented by stereo loss data 158, combines the L1 vector loss and the angle loss as follows:

[0049]

number

[0050] The photometric loss represented by the photometric loss data 165 is calculated between each of the rendered RGB images 132(1 ..M) and / or NIR images 134(1 ..N) according to equation (2) and the output of the prediction engine for the corresponding lighting condition as follows:

[0051]

number

[0052] A prior that promotes piecewise constant variation may be applied to the albedo map;

[0053]

number

[0054] This prior distribution may be applied to pixels that are classified as clothes, body, or arms in the semantic segmentation map 144.

[0055] The total loss function is a weighted sum of the above loss terms,

[0056]

number

[0057] The components of the user device 120 (e.g., modules, processing unit 124) may be configured to operate based on one or more platforms (e.g., one or more similar or different platforms), which may include one or more types of hardware, software, firmware, operating systems, runtime libraries, etc. In some implementations, the components of the computer 120 may be configured to operate within a cluster of devices (e.g., a server farm). In such implementations, the functionality and processing of the components of the computer 120 may be distributed across several devices of the cluster of devices.

[0058] The components of computer 120 may be or include any type of hardware and / or software configured to process attributes. In some implementations, one or more portions of the components shown in the components of computer 120 of FIG. 1 may be or include hardware-based modules (e.g., digital signal processor (DSP), field programmable gate array (FPGA), memory), firmware modules, and / or software-based modules (e.g., modules of computer code, sets of computer-readable instructions that can be executed on a computer). For example, in some implementations, one or more portions of the components of computer 120 may be or include software modules configured for execution by at least one processor (not shown). In some implementations, the functionality of the components may be included in different modules and / or different components than those shown in FIG. 1, including combining functionality shown as two components into a single component.

[0059] Although not shown, in some implementations, components of computer 120 (or portions thereof) may be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, etc. In some implementations, components of computer 120 (or portions thereof) may be configured to operate within a network. Thus, components of computer 120 (or portions thereof) may be configured to function within various types of network environments that may include one or more devices and / or one or more server devices. For example, the network may be or include a local area network (LAN), a wide area network (WAN), etc. The network may be or include, for example, a wireless network and / or a wireless network implemented using gateway devices, bridges, switches, etc. The network may include one or more segments and / or have portions based on various protocols, such as Internet Protocol (IP) and / or proprietary protocols. The network may include at least a portion of the Internet.

[0060] In some implementations, one or more of the components of computer 120 may be or include a processor configured to process instructions stored in memory. For example, image acquisition manager 130 (and / or portions thereof) and predictive image training manager 140 (and / or portions thereof) may be a combination of a processor and memory configured to execute process-related instructions to achieve one or more functions.

[0061] In some implementations, the memory 126 may be any type of memory, such as random access memory, disk drive memory, flash memory, etc. In some implementations, the memory 126 may be implemented as multiple memory components (e.g., multiple RAM components or disk drive memory) associated with components of the VR server computer 120. In some implementations, the memory 126 may be a database memory. In some implementations, the memory 126 may be or include a non-local memory. For example, the memory 126 may be or include a memory shared by multiple devices (not shown). In some implementations, the memory 126 may be associated with a server device (not shown) in a network and configured to serve components of the computer 120. As shown in FIG. 1, the memory 126 is configured to store various data, including image training data 131, semantic segmentation data 136, and prediction engine data 153.

[0062] Figure 2 illustrates an example prediction engine 200 configured to estimate surface normals and reflectance in an image. As shown in Figure 2, the prediction engine 200 includes an encoder 220, a decoder 222, a geometry branch 230, a reflectance branch 232, and a renderer 260. The prediction engine 200 illustrated in Figure 2 is a standard UNet with skip connections.

[0063] The encoder 220 accepts as input a single RBG image 210(a) and a single NIR image 210(b), as well as a label image 210(c) generated from the semantic segmentation map 144 as described with respect to FIG. 1. In some implementations, the RGB image 210(a) and the NIR image 210(b) are taken essentially simultaneously. For example, a mobile phone camera may have both a visible spectrum illumination source (i.e., a camera flash) and a NIR source (a different flash on the mobile phone) activated by the user at the same time. Thus, the images / maps 210(a,b,c) represent the user in the same pose.

[0064] The encoder 220 and the decoder 222 each consist of five blocks, each with three convolution layers. The bottleneck has 256 channels. As shown in Figure 2, the output of the decoder 222 is fed into two branches: a geometry branch 230 and a reflectivity branch 232.

[0065]

number

[0066] Both the geometry branch 230 and the reflectance branch 232 have three convolutional layers with 32 channels and one final output layer.

[0067] During training, the prediction engine 200 uses predicted normals 240 and observed normals 242 to generate the stereo loss. The observed normals 242 are input with the images 210(a, b, c) and may be obtained from a stereo depth map obtained with the training data.

[0068] During training, the prediction engine 200 performs a training process using the diffuse layer 270, the specular layer 272, and the shadow map 280 (i.e., S j), and a one-light-at-a-time (OLAT) image 284 that is input to the prediction engine 200 as part of training to generate photometric losses. The prediction engine 200 generates layers 270 and 272 based on the light position 250 used in generating the OLAT 284 and the diffuse and specular light rendering models (e.g., Equation (1)) in the renderer 260. The prediction engine 200 applies the light position 250 and the renderer 260 to the predicted normal 240, the diffuse albedo map 242, and the specular intensity map 244 to generate the diffuse layer 270 and the specular layer 272. The sum of the diffuse layer 270 and the specular layer 272 generates the rendered image 282, which, along with the OLAT 284 and the shadow map 280, generates the photometric losses.

[0069] 3 illustrates an example system 300 for generating RGB and NIR images for training a prediction engine. The system 300 includes multiple NIR illumination sources 310 (1, 2, 3, 4), multiple RGB sources 320 (1, 2, 3, 4), and an image capture arrangement 330.

[0070] 3, multiple NIR illumination sources 310(1,2,3,4) and multiple RGB sources 320(1,2,3,4) are arranged at the corners of a rectangle with an image capture arrangement 330 at its center. In some arrangements, the number of NIR illumination sources is different from the number of RGB sources. In some implementations, the NIR illumination sources and the RGB sources are arranged in different geometric patterns, for example, at the vertices of a polygon, in a circle or ellipse, along a line, etc.

[0071] The image capture configuration 330 includes a pair of NIR detectors (e.g., cameras) 332(1,2), a NIR ring light 336 surrounding the NIR detector 332(1), and an RGB detector 334, as shown in FIG. 3. In some implementations, the NIR ring light 336 may be located at another position near the NIR detector 332(1). In some implementations, the NIR ring light 336 may be replaced with an NIR illumination source of a different shape, for example, a disk shape. In some implementations, the image capture configuration 330 also includes a pair of NIR stereo dot projectors configured to project a series of dots onto the object to generate a stereo normal.

[0072] Ideally, the RGB and NIR images are taken at the same time and from the same viewpoint so that the training images of the subject all have the same pose. Nevertheless, since the light sources may not be precisely timed or co-located together, there may be slight differences in the time the images are taken and their viewpoints. The time difference is less than a time threshold in some implementations. In some implementations, the time threshold corresponds to a single frame in the video (e.g., 1 / 60 seconds for 60 fps, 1 / 24 seconds for 24 fps).

[0073] As an example, the RGB detector 334 may be a 7.0MP RGB camera operating at 66.67fps along with a stereo pair of 2.8MP NIR cameras operating at 150fps. The RGB camera and one of the NIR cameras are collocated using a plate beam splitter and a light trap. The RGB and NIR cameras in this example have linear photometric response and all of the images may be downsampled by 2x in each dimension; a center crop may be taken to cover the face at a resolution of 960x768. Visible spectrum (RGB) illumination may be provided by four wide-angle LED spotlights positioned at the corners of a rectangle approximately 1.5m x 0.8m (width x height) surrounding the camera located approximately 1.1m from the subject. NIR illumination may be provided by five adjacent NIR spotlights, one for each of the visible lights, and a flash LED light located near the reference NIR camera to generate a "dark flash" input. These NIR light sources are interleaved in time with a projector that emits a NIR dot speckle pattern to aid in stereo matching. A microcontroller may coordinate the triggering of lights and cameras to ensure that only one visible and one NIR light source is active at any time. All light sources may be calibrated for position and intensity and geometrically treated as point sources. The light intensity term L in equation (2) takes into account these calibrated colors. Note that the NIR and visible light sources are not collocated, and therefore slightly different values ​​of L are used in equation (2) for those two conditions.

[0074] 4A, 4B, and 4C are curve plots 400, 430, and 460, respectively, which show the average angular error (i.e.,

[0075]

number

[0076] 4 shows an exemplary effect of using an NIR image in addition to an RGB image on the overexposure level (400), color temperature difference (430), and noise level (460) of the RGB image. The curves for overexposure level (400), color temperature difference (430), and noise level (460) all show increasingly larger errors using RGB alone, while RGB and NIR show remarkable stability as exposure level, color temperature difference, and noise level increase.

[0077] 5 is a flow chart illustrating an example method 500 for generating relighting via surface normal and reflectance estimation according to the improved techniques described above. The method 500 may be performed by a software configuration described in connection with FIG. 1 , resident in the memory 126 of the computer 120 and executed by the set of processing units 124.

[0078] At 502, image acquisition manager 130 receives image training data (e.g., image training data 131) representing a plurality of color (RGB) images (e.g., images 132(1..M)) and a plurality of near-infrared (NIR) images (e.g., images 134(1..N)), where each of the plurality of RGB images was captured using a visible spectrum illumination source (e.g., RGB source 334) and each of the plurality of NIR images was captured using a NIR illumination source (e.g., NIR illumination source 332(1,2)), where the plurality of RGB images and the plurality of NIR images each include a subset of the plurality of RGB images and a subset of the plurality of NIR images, and where each subset of the plurality of RBG images and each subset of the plurality of NIR images includes images of a respective subject of a set of subjects in a pose through a plurality of lighting conditions.

[0079] At 504, the prediction engine manager 150 generates a prediction engine (e.g., prediction engine data 153) based on the image training data, the prediction engine configured to generate an estimated surface normal map (e.g., estimated normal data 156) and an estimated reflectance map (e.g., estimated albedo data 159 and / or specular reflectance data 160) for the user from a single RGB image of the user and a single NIR image of the user, where the single RGB image of the user and the single NIR image of the user are captured from viewpoints that differ by less than the threshold viewpoint within a time period less than the threshold time period.

[0080] In some implementations, the threshold time period is less than 100 milliseconds, less than 10 milliseconds, less than 1 millisecond, or less than a time period, hi some implementations, the threshold viewpoint is less than 10 degrees, less than 5 degrees, less than 2 degrees, less than 1 degree, less than 0.5 degrees, or less rotation around any axis of the user.

[0081] In some implementations, the prediction engine described above may be applied to stereo refinement applications. Stereo methods excel at measuring coarse geometric shapes, but often struggle to recover fine-scale surface details. This can be overcome by refining the stereo depth according to accurate high-resolution normals, typically estimated using photometric techniques. Normals produced by our method for refining depth measurements produced by NIR spatiotemporal stereo algorithms may be compared to using a standard bilateral filter for smoothing stereo depth. Normals produced by the improved technique described herein result in much higher quality reconstructions, most notably around the mouth, nose, and eyes, as well as better recovery of fine lines and wrinkles in the skin.

[0082] In some implementations, the prediction engine described above may be applied to lighting adjustments to improve lighting in portraits, for example by adding virtual fill lights to brighten shadowed areas of the face. Normal and reflectance maps estimated by our method may be used to render the contribution of virtual point lights located within the field of view of the shadowed regions, and these maps may be combined with the original RGB image. The model provided by the prediction engine allows for reliable effects, even resulting in realistic specular highlights along nasolabial folds and the tip of the nose.

[0083] 6 shows an example of a general purpose computing device 600 and a general purpose mobile computing device 650 that may be used with the techniques described herein. The computing device 600 is an example of a configuration of the computer 120 of FIGS.

[0084] As shown in Figure 6, computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components illustrated herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the invention described and / or claimed herein.

[0085] Computing device 600 includes a processor 602, a memory 604, a storage device 606, a high-speed interface 608 that connects to memory 604 and a high-speed expansion port 610, and a low-speed interface 612 that connects to a low-speed bus 614 and storage device 606. Each of components 602, 604, 606, 608, 610, and 612 are interconnected using various buses and may be implemented on a common motherboard or in other manners as desired. Processor 602 can process instructions for execution within computing device 600, including instructions stored in memory 604 or storage device 606, to display graphical information for a GUI on an external input / output device, such as a display 616 coupled to high-speed interface 608. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as desired. Multiple computing devices 600 may also be connected, with each device providing a portion of the required operations (e.g., as a server bank, as a group of blade servers, or as a multiprocessor system).

[0086] The memory 604 stores information within the computing device 600. In one implementation, the memory 604 is one or more volatile memory units. In another implementation, the memory 604 is one or more non-volatile memory units. The memory 604 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0087] The storage device 606 can provide mass storage for the computing device 600. In one implementation, the storage device 606 may be or include a computer-readable medium, such as a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. The computer program product can be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 604, the storage device 606, or a memory on the processor 602.

[0088] The high-speed controller 608 manages bandwidth-intensive operations for the computing device 500, and the low-speed controller 612 manages less bandwidth-intensive operations. Such an allocation of functions is merely exemplary. In one implementation, the high-speed controller 608 is coupled to the memory 604 (e.g., via a graphics processor or accelerator), to a display 616, and to a high-speed expansion port 610 that may accept various expansion cards (not shown). In this implementation, the low-speed controller 612 is coupled to the storage device 506 and to a low-speed expansion port 614. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), and may be coupled, e.g., via a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router.

[0089] The computing device 600 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 620, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 624. In addition, it may be implemented in a personal computer, such as a laptop computer 622. Alternatively, components from the computing device 600 may be combined with other components in a mobile device (not shown), such as device 650. Each such device may include one or more of the computing devices 600, 650, and the entire system may be composed of multiple computing devices 600, 650 in communication with each other.

[0090] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0091] These computer programs (also known as programs, software, software applications or codes) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device used to provide machine instructions and / or data to a programmable processor (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)), including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0092] To provide for user interaction, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may be used to provide for user interaction, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic, speech, or tactile input.

[0093] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), a middleware component (e.g., an application server), a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0094] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0095] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the specification.

[0096] It is also understood that when an element is referred to as being on, connected to, electrically connected to, coupled to, or electrically coupled to other elements, it may be directly on, directly connected to, or directly coupled to other elements, or there may be one or more intervening elements. In contrast, when an element is referred to as being directly on, directly connected to, or directly coupled to other elements, there are no intervening elements. Although the terms directly on, directly connected, or directly coupled may not be used throughout the detailed description, elements that are shown as being directly on, directly connected, or directly coupled may be so referred to. The claims of this application may be amended to describe the exemplary relationships described in the specification or shown in the drawings.

[0097] While certain features of the described implementations are shown as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It should therefore be understood that the claims are intended to cover all such modifications and changes that are within the scope of the implementations. It should be understood that these are presented only as examples and not as limitations, and that various changes in form and details may be made. Any part of the apparatus and / or method described herein may be combined in any combination except mutually exclusive combinations. The implementations described herein may include various combinations and / or subcombinations of the functions, components and / or features of the different implementations described.

[0098] Additionally, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Additionally, other steps may be provided or removed from the described flows, and other components may be added to or removed from the described systems. Accordingly, other implementations are within the scope of the claims.

Claims

1. 1. A method comprising: receiving image training data representing a plurality of color images and a plurality of near infrared (NIR) images, each of the plurality of color images being captured with a visible spectrum illumination source and each of the plurality of NIR images being captured with a NIR illumination source, the method further comprising: and generating a prediction engine based on the image training data, the prediction engine being configured to (i) generate an estimated surface normal map of the user and an estimated reflectance map of the user from a color image of the user and an NIR image of the user, and (ii) generate a photometric loss based on rendering from the estimated surface normal map and the estimated reflectance map under one of a plurality of lighting conditions, wherein the color image of the user and the NIR image of the user are captured from viewpoints that differ by less than a threshold viewpoint within a time period that is less than a threshold time period.

2. the plurality of color images and the plurality of NIR images each comprise a subset of the plurality of color images and a subset of the plurality of NIR images, each subset of the plurality of color images and each subset of the plurality of NIR images comprising an image of a respective object of a set of objects in a pose through the plurality of lighting conditions; The method further comprises:

2. The method of claim 1 , prior to generating the prediction engine, performing a semantic segmentation operation on a color image of a corresponding subset of the plurality of color images for each of the set of objects to generate a label image for that object, the label image having a specified number of classes into which each of a plurality of pixels of the color image is categorized, the label image for that object being included in the image training data.

3. 3. The method of claim 2, wherein each subset of the plurality of color images and each subset of the plurality of NIR images are captured using an image capture arrangement, the image capture arrangement including a plurality of color illumination sources, a plurality of NIR illumination sources, and a color detector and a NIR illumination detector, respectively, the plurality of color illumination sources and the plurality of NIR illumination sources arranged in a geometric pattern surrounding the color detector and the NIR illumination detector.

4. The method of claim 3 , wherein the plurality of color illumination sources and the plurality of NIR illumination sources are positioned at corners of a rectangle surrounding the color detector and the NIR illumination detector.

5. 4. The method of claim 3, wherein each of the plurality of lighting conditions includes a position of one of the plurality of color illumination sources, a position of one of the plurality of NIR illumination sources that produces illumination, all other positions of the plurality of color illumination sources, and all other positions of the plurality of NIR illumination sources that do not produce illumination.

6. 4. The method of claim 3, wherein the color detector and the NIR illumination detector include a first NIR camera, a second NIR camera, and a color camera, and the first NIR camera and the color camera are misaligned by an amount less than a specified misalignment threshold.

7. 4. The method of claim 3, wherein the image capture arrangement further comprises an NIR dot projector configured to project a dot speckle pattern onto the object, the dot speckle pattern being temporally interleaved with illumination emitted by an NIR illumination source of the plurality of NIR illumination sources.

8. The method of any preceding claim, wherein the color image of the user and the NIR image of the user are captured essentially simultaneously.

9. 9. The method of claim 1, wherein the prediction engine comprises a first branch and a second branch, the first branch configured to generate a surface normal map and the second branch configured to output a predicted reflectance map.

10. 10. The method of claim 9, wherein the prediction engine comprises a neural network having a unet encoder-decoder architecture with skip-level connections, the unet encoder-decoder architecture comprising an encoder and a decoder, the encoder comprising a set of blocks, each of the set of blocks comprising a set of convolutional layers and a set of ReLU activation layers, and the decoder configured to output a surface normal map in the first branch and a predicted reflectance map in the second branch.

11. generating the prediction engine The method of claim 9 , comprising using stereo loss and the photometric loss to supervise training operations for the prediction engine.

12. Generating the prediction engine further comprises: The method of claim 11 , comprising generating the photometric loss based on rendering from the estimated surface normal map and the estimated reflectance map under a lighting condition of the plurality of lighting conditions.

13. the estimated reflectance map includes a diffuse component and a specular component; generating said photometric loss generating a diffuse component of the estimated reflectance map using a Lambertian reflectance model; and generating a specular component of the estimated reflectance map using a Blinn-Phong bidirectional reflectance distribution function (BRDF).

14. generating said photometric loss generating a binary shadow map based on the stereo depth map and the positions of light sources used in generating a color image of the plurality of color images; generating an observed intensity map based on the estimated reflectance map; The method of claim 12 , comprising generating a Hadamard product of the binary shadow map and a difference between the observed intensity map and the color image as the photometric loss.

15. Obtaining a stereo depth map; performing a smoothing operation on the stereo depth map to generate a smoothed stereo depth map; The method of claim 12 , further comprising: generating a stereo loss based on gradients of the estimated surface normal map and the smoothed stereo depth map.

16. generating said stereo loss generating an L1 norm of the difference between the estimated surface normal map and the gradient of the smoothed stereo depth map as an L1 vector loss; generating an inner product of the estimated surface normal map and the gradient of the smoothed stereo depth map as an angle loss; and generating as the stereo loss a difference between the L1 vector loss and the angle loss.

17. 17. The method of claim 1, further comprising using the prediction engine to generate the estimated surface normal map of the user and the estimated reflectance map of the user from the color image of the user and the NIR image of the user.

18. The method according to any of the preceding claims, wherein the colour image of the user is a single colour image and the NIR image of the user is a single NIR image.

19. A program comprising code which, when executed by a computer, causes the computer to carry out the method according to any one of claims 1 to 18.

20. An apparatus comprising: Memory, and a control circuit coupled to the memory, the control circuit comprising: and configured to receive image training data representing a plurality of color images and a plurality of near infrared (NIR) images, each of the plurality of color images being captured with a visible spectrum illumination source and each of the plurality of NIR images being captured with a NIR illumination source, the control circuitry further comprising: and generating a prediction engine based on the image training data, the prediction engine being configured to (i) generate an estimated surface normal map of the user and an estimated reflectance map of the user from a color image of the user and an NIR image of the user, and (ii) generate a photometric loss based on rendering from the estimated surface normal map and the estimated reflectance map under one of a plurality of lighting conditions, wherein the color image of the user and the NIR image of the user are captured from viewpoints that differ by less than a threshold viewpoint within a time period of less than a threshold time period.

Images