DEEP LEARNING SYSTEMS, DEVICES AND METHODS FOR PREDICTING HIGH DYNAMIC RANGE ENVIRONMENTAL PANORAMAS

Abstract

DEEP LEARNING SYSTEMS, DEVICES, AND METHODS FOR PREDICTING HIGH DYNAMIC RANGE ENVIRONMENT PANORAMAS Aspects of lighting estimation and related models are provided, including aspects for training such models. A lighting estimation model pretrained using synthetic data is proposed to alleviate the costs and difficulty of obtaining paired datasets of real portrait images and HDR environment maps. To improve model performance, the model trains using a discriminator configured to predict one or more average color values ​​of a defined percentage of highest intensity pixels of a predicted environment map and to determine a color dropout associated with the predicted environment map and the one or more average color values.The trained model can be used for a wide range of downstream tasks, including being used to generate hair renderings with realistic lighting effects for virtual try-on experiences. Figure for abstract: none.

Description

Title of the invention: DEEP LEARNING SYSTEMS, DEVICES AND METHODS FOR PREDICTING PANORAMAS HIGH DYNAMIC RANGE ENVIRONMENT FIELD OF THE INVENTION

[0001] The present disclosure relates to computer image processing and artificial intelligence, in particular to systems and methods for estimating lighting and more particularly for predicting high dynamic range (HDR) environment panoramas from single low dynamic range (LDR) limited field of view portrait images. CONTEXT

[0002] The ambient light color and light direction in a real-world scene have a significant impact on the content of an image that captures a view of that scene. Scene images may be processed and new images may be output to provide augmented reality experiences, including virtual try-on-top (VTO) experiences where one or more effects are applied to an object in the images. For example, the processing may apply hair effects, makeup effects, and the like to a person's head in the image. VTO and other augmented reality experiences are often presented via a consumer-facing computing device such as a smartphone, tablet, laptop, or desktop computer where an input image of a scene includes a single low dynamic range (LDR) limited field of view (FOV) portrait image, such as a "selfie" of a user.

[0003] A deep learning lighting estimation model may estimate the lighting scene of an input image, for example, to provide an environment map. In one example, the environment map may provide information about realistic rendering results in an augmented reality experience. Compiling portrait images with paired lighting data (e.g., an environment map) to train a light estimation model is expensive and time-consuming.

[0004] It is desired to make improvements to deep learning models for lighting estimation and their training. Abstract

[0005] In at least some aspects, it is desirable to have a deep learning computer system and method that predicts HDR environment panorama(s) from LDR limited field of view portrait image(s) for estimate lighting conditions and further generate realistic image data for use in augmented reality experiences.

[0006] In at least some aspects, the ability to estimate lighting conditions and predict HDR environment panorama(s) using a deep learning computer system and method using an estimation model trained on a synthetic image dataset would help overcome the costly and time-consuming process associated with collecting paired data of portrait images and HDR environment maps to train lighting estimation models. In at least some aspects, such a model trained on a synthetic image dataset would help address the problem of model biases toward certain demographic groups, which may result from the typical need to train such systems on a small real-world dataset due to time or cost constraints.

[0007] According to one aspect of the present invention, a computer system and method using deep learning lighting estimation model(s), including aspects for training such model(s), are provided. Further, according to one embodiment, a synthetic dataset used for training lighting estimation model(s) is provided to alleviate the costs and difficulty of obtaining paired datasets of real portrait image and HDR environment map.In accordance with one embodiment, a generated synthetic dataset is provided in which a large synthetic dataset of subjects and environments is sampled to ensure that the dataset covers a diversity of lighting conditions and subject demographics, including subjects of diverse gender, ethnicity, and age group, to ensure that the lighting estimation models are not biased toward certain demographic groups.

[0008] In order to further improve model performance, according to one embodiment, an evaluation pipeline is provided for evaluating the model's ability to infer lighting conditions, in the form of an environment map, from real portrait images. According to one embodiment, a video evaluation pipeline is provided for qualitatively measuring the prediction consistency of the model's environment map.

[0009] The following statements provide various aspects and features disclosed in the embodiments herein. These and other aspects will be apparent to those skilled in the art, such as computer program product aspects. It is also understood that computing device aspects may have corresponding method aspects and vice versa.

[0010] Statement 1: A computing device comprising a processor coupled to a storage device storing instructions executable by the processor to cause the computing device to: predict a predicted environment map for lighting conditions in an input image using a generator of a deep learning model, the predicted environment map encoding one or more light sources and an estimate of the light source color in the input image, the deep learning model having been defined by training using a discriminator that predicts one or more average color values ​​of a defined percentage of highest intensity pixels of an environment map,the discriminator being defined using a color loss determined from: one or more average color values ​​of a defined percentage of highest intensity pixels of a ground truth environment map; and a prediction by the discriminator for a training-time predicted environment map generated by the generator; and generating an output image comprising one or more objects of the input image to which one or more respective effects are applied, wherein a property of each of the respective effects is adapted to the predicted environment map.

[0011] Statement 2: The computing device of Statement 1, wherein the training-time predicted environment map encodes one or more predicted light sources and a light source color estimate of one of a plurality of synthetically generated images provided to the generator, the plurality of synthetically generated images having lighting conditions including light source direction and light source color.

[0012] Statement 3: The computing device of Statement 2, wherein at least one of: the plurality of synthetically generated images comprises a portrait image dataset containing combinations of subjects and environments sampled to cover a variety of demographics, light source directions, and light source colors; the plurality of synthetically generated images comprises synthetic faces sampled based on various genders, ethnicities, and ages; the plurality of synthetically generated images are generated to include instances of neutral facial expressions and extreme facial expressions and at various head positions and orientations; or the plurality of synthetically generated images comprises images with a field of view set at a viewing angle between 50 and 70 degrees.

[0013] Statement 4: The computing device of Statement 1, wherein the deep learning model is defined by training comprising generating one or more bounding boxes to align a center of a head in the input image and perform one or both of a rotation or transposition of an environment of the input image to align a center of the environment with the center of the head.

[0014] Statement 5: The computing device of Statement 1, wherein the defined percentage of the highest intensity pixels of the ground truth environment map is in the range of 5 percent to 10 percent.

[0015] Statement 6: The computing device of Statement 5, wherein the pixel intensity of the ground truth environment map is defined based on a perceptual brightness (L*) channel of the CIELAB color space.

[0016] Statement 7: The computing device of Statement 1, wherein the generator comprises one or more convolution output layers and one or more upsampling blocks, each of the upsampling blocks comprising a convolution layer, a ReLu activation layer, and a bilinear upsampling layer.

[0017] Statement 8: The computing device of Statement 1, wherein the input image comprises a limited field of view low dynamic range portrait image and the output image comprises a high dynamic range image.

[0018] Statement 9: The computing device of Statement 1, wherein the effect applied to the one or more objects from the input image is a virtual try-on effect.

[0019] Statement 10: Computing device of Statement 1, wherein the generator is defined by training using a reconstruction loss, the reconstruction loss being defined from an L2 loss applied to the training-time predicted environment map and the ground truth environment map.

[0020] Statement 11: The computing device of Statement 10, wherein the generator is defined by training using a generator loss, the generator loss being determined from a weighted sum of the reconstruction loss and the color loss, and wherein the weighted sum is set to 10 to 1.

[0021] Statement 12: The computing device of Statement 1, wherein the generator is defined by training using a coherence loss, the coherence loss being determined from: a first predicted environment map generated by the generator for the lighting conditions of a first consecutive video frame; and the sum of a second predicted environment map generated by the generator for the lighting conditions of a second consecutive video frame and a third predicted environment map generated by the generator for the lighting conditions of a third consecutive video frame.

[0022] Statement 13: The computing device of Statement 1, further comprising a camera device configured to perform one or both of capturing or recording the input image.

[0023] Statement 14: A method of configuring a deep learning model that predicts lighting conditions in an input image and generates a predicted environment map for the lighting conditions in the input image, the method comprising: defining the deep learning model by training using a discriminator that predicts one or more average color values ​​of a defined percentage of highest intensity pixels of an environment map, the discriminator being defined using a color loss determined from: one or more average color values ​​of a defined percentage of highest intensity pixels of a ground truth environment map; and a prediction by the discriminator for a predicted environment map in training time generated by a generator of the deep learning model for a lighting condition of a training image;and generating the predicted environment map using the generator, the generator having been defined by training using the color loss. ;

[0024] Statement 15: The method of Statement 14, wherein the training image comprises one of a plurality of synthetically generated images provided to the generator, the plurality of synthetically generated images having lighting conditions comprising a light source direction and a light source color.

[0025] Statement 16: The method of Statement 15, wherein at least one of: the plurality of synthetically generated images comprises a portrait image dataset containing combinations of subjects and environments sampled to cover a variety of demographics, light source directions, and light source colors; the plurality of subjects of synthetically generated images of subjects comprises synthetic faces sampled based on various genders, ethnicities, and ages; the plurality of synthetically generated images are generated to include instances of neutral facial expressions and extreme facial expressions and at various head positions and orientations; or the plurality of synthetically generated images comprises images with a field of view set at a viewing angle between 50 and 70 degrees.

[0026] Statement 17: The method of Statement 14, wherein the generator is defined by training using a reconstruction loss, the reconstruction loss being defined from an L2 loss applied to the training-time predicted environment map and the ground truth environment map.

[0027] Statement 18: The method of Statement 17, wherein the generator is defined by training using a generator loss, the generator loss being determined from a weighted sum of the reconstruction loss and the color loss, and wherein the weighted sum is set to 10 to 1.

[0028] Statement 19: The method of Statement 14, wherein the generator is defined by training using a coherence loss, the coherence loss being determined from a first predicted environment map generated by the generator for the lighting conditions of a first consecutive video frame; and the sum of a second predicted environment map generated by the generator for the lighting conditions of a second consecutive video frame and a third predicted environment map generated by the generator for the lighting conditions of a third consecutive video frame.

[0029] Statement 20: A computing device comprising: a VTO (virtual try-on) rendering engine configured to produce an output image for display, wherein the output image comprises an object from an input image to which a VTO effect is applied, the VTO rendering engine comprising a deep learning model configured by training to provide a predicted environment map for lighting conditions in the input image and wherein the effect is applied to the object as adapted by the predicted environment map, the deep learning model being defined by training by: providing a generator;providing a discriminator that predicts one or more average color values ​​of a defined percentage of highest intensity pixels of an environment map, the discriminator being defined using a color loss determined from: one or more average color values ​​of a defined percentage of highest intensity pixels of a ground truth environment map; and a prediction by the discriminator for a training-time predicted environment map generated by the generator; and wherein the training-time predicted environment map encodes one or more predicted light sources and a light source color estimate of one or more of a plurality of synthetically generated images provided to the generator, the plurality of synthetically generated images having lighting conditions including a light source direction and a light source color;and wherein the generator is defined by training using a reconstruction loss, the reconstruction loss being defined from an L2 loss applied to the training-time predicted environment map and the ground truth environment map; and one or both of a VTO product recommendation interface or a product purchase interface, wherein the VTO effect is associated with a product to simulate a fitting. ; Brief description of the drawings

[0030] These and other features of the invention will become more apparent from the following description in which reference is made to the accompanying drawings in which:

[0031] [Fig-1] [Fig. 1] is a block diagram showing an example of a deep learning lighting estimation model, according to one embodiment.

[0032] [Fig.2] [Fig.2] is a block diagram showing the estimation model lighting of [Fig.l] as adapted with a hair rendering, in accordance with one embodiment.

[0033] [Fig.3] [Fig.3] shows a set of evaluation images according to a embodiment.

[0034] [Fig.4] [Fig.4] is a block diagram of a computer system, in accordance with one embodiment.

[0035] [Fig.5] [Fig.5] is a block diagram showing the estimation model lighting of [Fig.l] as adapted with additional training refinement using consecutive video frames, according to one embodiment. DETAILED DESCRIPTION

[0036] The systems, devices, and methods herein are directed to improving existing deep learning lighting estimation applications by providing a lighting estimation model trained on a synthetic image dataset. Generally, in at least some embodiments, systems and methods are provided that use an estimation model trained on synthetic datasets to estimate lighting conditions in an input photo.

[0037] In order to achieve realistic rendering results in augmented reality experiences, research has been conducted to estimate the lighting scene of images through deep learning models. Gardner et al. [9] first presented an end-to-end deep neural network that directly regresses key lighting locations and intensities from a limited field-of-view photo, without strong assumptions about scene geometry, material properties, or lighting. Previous models used to estimate the lighting scene of images can be classified into two main categories: regression models that estimate low-dimensional panoramic lighting parameters and generative models that generate non-parametric illumination maps or light probes.For example, there are models that regress on more lighting parameters such as light source color [9], angular size of light in steradians. [9] or the 5th-order spherical harmonic representation of lighting

[10] . With the increasing popularity of generative models, there are also frame-by-frame generative models that directly generate an HDR environment map, such as styleLight [7]. Regression models have less flexibility and loss details in describing lighting information and require additional post-processing steps to convert lighting parameters into an HDR environment map for image-based relighting, and as such, it is desirable to have a generative model for lighting estimation.

[0038] Previous work has explored ways to infer lighting information in real-time augmented reality applications. LeGender et al. [2] first proposed models that predict low-resolution HDR light probes based on unconstrained LDR images with limited field of view, and later extended the work to focus on predicting light probes based on portrait LDR images with limited field of view [5]. These models are small models that can infer in real-time on mobile applications. However, since light probes have low resolution and only capture lighting information behind the camera, this limits the realism when using light probes to render images. To address this issue, Somanath et al.[6] presented a model that infers high-resolution HDR panorama maps based on unconstrained LDR images with limited field of view.

[0039] Developing generative models for lighting estimation presents many challenges. For example, collecting paired portrait image and HDR environment map data is an expensive task, as it requires facial lighting in the portrait images to be consistent with the corresponding environment map. This limits the ability to collect a large portrait and light map dataset with diverse environments and subjects. Thus, models trained on small real-world datasets only cover a small variety of subjects and environments, and may result in model biases toward certain demographic groups.

[0040] Previously, the HDR Laval Face+Lighting dataset [1] was created by inviting 9 subjects into 25 environments with different lighting conditions, and sequentially photographing the environments at different orientations and exposures, and the subjects. LeGender et al. [2] generated such a dataset by first recording the reflectance field and alpha matte of 70 diverse subjects using a light scene, and separately collecting HDR environment maps, and then relighting the subjects using image-based relighting to create portrait image and environment map pairs. Similarly, Sztrajman et al. [3] also generated the dataset by combining scanned faces relightable images from the ICT 3d Relightable Facial Expression database [4] with environment maps from the HDR Laval Face+Lighting dataset [1]. This may increase the variety of environment maps in the dataset, but the dataset still only covers a small variety of subjects. MODEL ARCHITECTURE

[0041] [Fig.l] shows a block diagram of a computing device in which one or more technologies or methodologies may be implemented, such as, for example, estimating lighting conditions for synthetic content in an augmented reality environment, a mixed reality environment, a virtual try-on environment, and the like. In one embodiment, the computing device includes processor computing circuitry, and one or more deep learning components configured to predict high dynamic range (HDR) environment panoramas from single low dynamic range (LDR) limited field of view portrait images.

[0042] In one embodiment, the computing device is configured to provide an estimation stream 100 in accordance with an embodiment in which an estimation model 102 is configured to receive an input image 104 as input, and provides as output an environment map prediction 112. In a training mode, the input image 104 represents a training image associated with a matched ground truth map 126. In an inference time mode, the input image 104 represents an inference time image such as a user image for use in an augmented reality experience.

[0043] It should be understood that [Fig.l] represents the estimation stream 100 as data components and executable components stored within one or more storage devices 103 of a computing device 105, in accordance with one embodiment. The executable components are executable by a processor (not shown) of the computing device. In one embodiment, the input image 104 comprises a limited field-of-view LDR portrait image. In another embodiment, the input image 104 comprises a synthetically generated image. In one embodiment, the estimation model 102 comprises a generator based on an encoder-decoder architecture. In one embodiment, the image feature extraction 106 uses MobileNet v2 to extract image features from the input image 104.In one embodiment, the upsampling blocks 108 comprise 5 upsampling blocks, wherein each upsampling block comprises a convolution layer, a ReLu activation layer and a bilinear upsampling layer. According to the embodiment of [Fig.l], the estimation model 102 further comprises the convolution output layer 110. The estimation model . 102 outputs a predicted environment map, generated by a generator, as environment map prediction 112. In one embodiment, environment map prediction 112 is predicted in logarithmic scale to convey long-tail RGB color distribution data of HDR environment maps.

[0044] In one embodiment, the discriminator 116 comprises 6 convolution layers and each convolution layer is followed by a Leaky ReLu activation layer. In some embodiments, the discriminator 116 further comprises a fully connected output layer (not shown) configured to predict average RGB values. In one embodiment, the estimation stream 100 represents a training stream at a training time. The components 102 are retained during the use of the inference time.

[0045] The input image 104 may be received from another computing device (not shown) over a communications network (not shown) or at least partially input by a user to a computing device for the estimation model (e.g., a computing device 300 shown in [Fig. 3]). Loss functions

[0046] Previous work [7, 8] typically applies a per-pixel loss on the first 5-10% of highest intensity pixels of an environment map. In accordance with the experimental results and the techniques and teachings herein, estimation model training is improved when such a loss is instead computed by a discriminator. As such, in one embodiment, the environment map prediction 112 generated by a generator is output by the estimation model 102 and passed to the discriminator 116 (an executable component).The discriminator 116 is trained to predict the average RGB value of defined percentages of highest intensity pixels of a ground truth environment map (e.g., matched ground truth map 126) based on an L1 loss, wherein defined percentages of the highest intensity pixels of the ground truth environment map are calculated based on the perceptual brightness (L*) channel of the CIELAB color space. In one embodiment, the defined percentages are the first 10% of highest intensity pixels and the first 10-50% of highest intensity pixels. In one embodiment, the color loss based on a discriminator 118 is determined according to the formula: .

[0047] Discriminator loss = L1(D toplQ(map gt ), map gt. mask topWgt )+ L1(D toplQ_5Q(map gt ), map gt. mask topw>-wgt )

[0048] where D is the discriminator 116, map gt is the matched ground truth map 126, and maskgt is a binary mask based on the defined percentage of highest intensity pixels of the matched ground truth map. In one embodiment, the defined percentage of mask toPiogt is the first 10% of highest intensity pixels. In one embodiment, the defined percentage of mask io_sogt is the first 10-50% of highest intensity pixels.

[0049] In one embodiment, the estimation model 102 is trained on the sum of a reconstruction loss 120 and a light source color loss 114, wherein the relative weights of the reconstruction loss to the light source color loss are set to 10 to 1. In another embodiment, the discriminator-based color loss 118 is further used to update the training of the estimation model 102.

[0050] In one embodiment, the light source color loss 114 is calculated by first transmitting a predicted environment map (e.g., the environment map prediction 112) to the discriminator 116 to calculate the average RGB of a defined percentage of highest intensity pixels, where the intensity is defined based on the perceptual brightness (L*) channel of the CIELAB color space. An L1 loss is applied between the output of the discriminator 116 and the average RGB of the defined percentage of highest intensity pixels of the ground truth environment map (e.g., the matched ground truth map 126) according to the formula:

[0051] Loss of light wall = L1(D lopW(E(x)), map gt* mask lopWgl )+ L1(D lopW_50(E(x)), map gt * mask toPio-50gt )

[0052] where D is the discriminator 116, E is the estimation model 102, x is the input image 104, cartegt is the matched ground truth map 126, and masquegt is a binary mask based on the defined percentage of highest intensity pixels of the matched ground truth map 126. In one embodiment, the defined percentage of mask topWgt is the first 10% of highest intensity pixels. In one embodiment, the defined percentage of mask topio-5ogt is the first 10-50% of highest intensity pixels. In one embodiment, the color loss module 124 includes an executable component configured to execute a light source color loss function and provide a light source color loss 114 to the estimation model 102.

[0053] In one embodiment, the reconstruction loss 120 comprises an L2 loss applied to minimize a per-pixel difference between the environment map prediction 112 and a ground truth environment map (e.g., the matched ground truth map 126) according to the formula:

[0054] Loss Reconstitution = E2(E(x), map gt )

[0055] wherein E is the estimation model 102, x is the input image 104, and the mapgt comprises a ground truth environment map (e.g., matched ground truth map 126).

[0056] In one embodiment, a global training loss function is applied by the estimation model 102 comprising the sum of the discriminator-based color loss 118 and the weighted sum of the reconstruction color loss 120 and a light source color loss 114, wherein the relative weights of the reconstruction color loss to the light source color loss are set to 10 to 1. In one embodiment, the reconstruction loss module 122 comprises an executable component configured to execute a reconstruction loss function and provide a reconstruction loss 120 to the estimation model 102. Synthetic training data

[0057] In one embodiment, the estimation model 102 is trained with images (e.g., the input image 104) sampled from a synthetic training dataset created using the DataGen platform. In one embodiment, the synthetic training dataset includes 50,000 portrait images with combinations of over 5,000 subjects and 250 synthetic environment maps. In accordance with one embodiment, the synthetic subjects and environments are broadly sampled to ensure that the synthetic training dataset covers a diversity of demographics and lighting conditions. In one embodiment, the environment maps include both indoor and outdoor scenes, and are at different times of day, e.g., morning and evening.

[0058] To provide a synthetic training dataset representing directional lights from different angles, in accordance with one embodiment, the environment maps in the dataset are rotated horizontally at different angles when rendering portrait images. To ensure that an estimation model (e.g., 102) is not biased toward certain demographic groups, the synthetic subjects are sampled from the synthetic dataset based on various groups of genders, ethnicities, and ages. In accordance with another embodiment, synthetic subjects are generated for the synthetic training dataset with neutral and extreme expressions, and at different head positions and orientations, such as frontal and profile faces. In accordance with one embodiment, the field of Camera view in synthetic portrait images is set between 50 and 70 degrees to resemble the front camera of mobiles and webcams.

[0059] In one embodiment, bounding boxes are used to center aligned heads in portrait images to ensure that the estimation model (e.g., 102) focuses on learning the light map of the faces during training. In one embodiment, the environment maps are rotated and / or shifted so that the centers of the environment maps are aligned with the location of the subjects' faces in the portrait images. In another embodiment, random horizontal flipping of images and increasing color flicker of the images are used during training of an estimation model (e.g., 102).

[0060] In one embodiment, the environment map prediction 112 output by the estimation model 102 may be used to locate key direction light sources and estimate the ambient light color from the input image 104. In one embodiment, the estimation model 102 may infer environment panoramas from the input image 104 in real-time and may be deployed in mobile or web applications. Evaluation with synthetic data

[0061] In one embodiment, the performance of an estimation model (e.g., 102) is quantitatively evaluated using synthetic data. An estimation model (e.g., 102) is examined based on the root mean square error (RMSE), scaled root mean square error (sMRSE), RGB angular error, and Fréchet inception distance (FID) between the environment map prediction (e.g., 112) and a ground truth environment map (e.g., the matched ground truth map 126).

[0062] In one embodiment, the angular error is calculated directly on predicted equirectangular environment maps (e.g., 112), while other metrics are calculated based on a cubic map converted from the predicted equirectangular maps. Estimation model with hair rendering

[0063] [Fig. 2] shows a block diagram of an exemplary computing device configured to provide an estimation stream 100 in accordance with an embodiment of [Fig. 1]. In one embodiment, the computing device 105 is further configured to generate a hair rendering 222 suitable for environment map prediction 112. In one embodiment, the hair rendering 222 mimics real hair in portrait images (e.g., the input image 104). For example, the rendering operations overlay hair pixels onto a portrait image (e.g., an input image) to simulate a hairstyle in a generated output image. In one embodiment, the hairstyle is defined according to a 3D hair mesh defining a plurality of hair strands, etc. for the hairstyle. In one embodiment, the lighting source direction(s) and lighting source color(s) of the environment map are used to adapt a property of the pixels in the output image. The color and / or brightness of the hair pixels or other pixels may be adjusted, for example, using shading and lighting techniques to illuminate, for example, the hair to match the environmental lighting of the input image. Shadows may be determined and rendered.In one embodiment, the environment map is converted into a set of discrete light sources such as by detecting brightest points on the map and finding the average color for each point to determine the light color and intensity. Each light point is converted into a directional light source. In one embodiment, hair shading is performed using a Marschner hair shading model with, for example, directional light sources. In one embodiment, indirect illumination is approximated using a double scattering technique. In one embodiment, lookup tables help optimize real-time performance. Evaluation with real data

[0064] Since the provided estimation models are trained on synthetic data, it is preferable to assess how well the estimation models fit real data. Evaluation with real data is typically done either by directly comparing predicted environment maps (e.g., 112) and real environment maps, or by rendering re-illuminated objects based on predicted environment maps (e.g., 112) and real environment maps. Such an assessment is difficult due to the lack of ground truth lighting and albedo maps of faces or hair in portrait images.

[0065] To address this challenge, an estimation model (e.g., 102) is evaluated by re-illuminating hair renderings (e.g., 222) based on predicted environment maps (e.g., 112) that mimic actual hair in portrait images, and comparing per-frame changes in the predicted environment maps (e.g., 112) for actual videos.

[0066] According to one embodiment, to examine the model's ability to predict ambient light color and locate strong directional lights, light maps are rendered on hair renderings that mimic hair. real in real portrait images. 10 hair experts were asked to annotate hair color on 1000 portraits. For each portrait image, the color with the highest agreement among the hair expert annotations was used as the ground truth color. Based on the annotated hair color, the average RGB of the corresponding hair strand images was calculated and hair renderings were generated based on the average RGB. In one embodiment, the real portrait images (e.g., 104) are input into an estimation model (e.g., 102) to predict light maps (e.g., environment map prediction 112) and hair meshes (e.g., a hair simulation from a 3D mesh definition) are rendered (e.g., 222) based on the predicted light maps.Images of the rendered hair meshes are compared to the actual portrait images input into the estimation model to assess how close the overall hair colors are and whether the specular light locations are consistent.

[0067] [Fig. 3] shows an evaluation image set 300 in accordance with one embodiment. The image set 300 includes portrait images 302A, 302B (e.g., input images), rendered hair mesh images 304A, 304B based on the predicted environment map (not shown), and annotated hair color and annotated hair color images 306A and 306B. The images 306A and 306B are compiled from the average RGB annotated by hair experts. For simplicity, [Fig. 3] is shown as grayscale representations of the images based on the original RGB colors. The subject of image 302A originally appears in front of a primarily green vegetation background with dark-colored (near-black) hair. The subject of image 302B appears in front of a primarily white and gray background with hair that is generally dark reddish-blonde in color.

[0068] In one embodiment, a video-based evaluation method is also employed to examine the ability of an estimation model (e.g., 102) to locate key direction lights and the consistency in generating an environment map prediction (e.g., 112). Videos with stable and moving camera positions are used to assess whether the model is capable of making reasonable and consistent predictions of strong light sources. In one embodiment, lighting estimation models are run on each frame of a video input in an estimation model to estimate a frame-wise light map.Subsequently, qualitative evaluations are conducted on the stability of a predicted light map in videos where the background of an input video is stable, and on the smoothness of movements of key light sources in videos when the camera recording the input video moves.

[0069] Quantitative analysis indicated that an estimation model according to one embodiment was capable of locating strong light sources from an input image and identifying whether the ambient light of an input image is cool or warm.

[0070] Computing Device Using Estimation Model in VTO Pipeline

[0071] [Fig. 4] is a block diagram of a computing system 400, in accordance with one embodiment. In the embodiment, an estimation model in accordance with one embodiment herein is integrated with a virtual try-on (VTO) application to provide hair simulation, such as simulating hair from a 3D hair mesh providing a definition of a hairstyle to be simulated. Rendering may be performed as described with reference to [Fig. 2]. The device 400 is non-limiting and is simplified for brevity.

[0072] The computing device 400 includes one or more processors 402, one or more input devices 404, one or more communication units 408, one or more output devices 410, a display screen 448 (e.g., providing one or more graphical user interfaces), a camera 446, and a memory 406. The computing device 400 also includes one or more storage devices 414 storing one or more executable computer modules including an application and lighting estimation data 416 including: input image 418 (e.g., 104), environment map prediction 422 (e.g., 112), estimation model 420 (e.g., estimation model 102), hair rendering 424 (e.g., hair rendering 222), estimation model data store 428, VTO pipeline 440, user interface 448, product purchase interface 442, and image with hair rendering 444.The computing device 400 may include additional computing modules or data stores in various embodiments. Additional computing modules and devices that may be included in various embodiments are not shown in [Fig. 4] to avoid undue complexity of the description, such as communicating with one or more other computing devices, as the case may be, using communication unit(s) 408, to obtain the input image 418 including via a communication network (not shown).

[0073] The storage device(s) 414 store(s) computer-readable data and / or instructions for execution by a processing unit (e.g., processor(s) 402), such that once executed, the instructions cause the computing device to perform operations such as one or more methods. The one or more storage devices 414 may take various forms and / or configurations, e.g., short-term memory or long-term memory. The storage device(s) 414 may be configured for short-term storage of information as volatile memory, which does not retain the stored contents when power is removed. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), etc. The storage device(s) 414, in some examples, also includes one or more computer-readable storage media, e.g., to store larger amounts of information than volatile memory and / or to store this information long-term, retaining the information when power is removed.Examples of non-volatile memory include magnetic hard drives, solid-state hard drives, optical disks, floppy disks, flash memory, or forms of electrically programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM). The computing device 400 may store data / information (e.g., the input image 418, the environment map prediction 422, or the hair-rendered image 444) in the storage device(s) 414.

[0074] One or more processors 402 may implement functionality and / or execute instructions within the computing device 400. For example, the processor(s) 402 may be configured to receive instructions and / or data from the storage device(s) 414 to execute the functionality of the estimation models (e.g., 102) shown in Figures 1 and 2, among other modules (e.g., operating system 430, browser 432, email and / or messaging application 434, social media application 436, etc.). The processor(s) 402 includes one or more central processing units (e.g., CPUs), and / or graphics processing units (e.g., GPUs) having one or more processors / microprocessors, controllers / microcontrollers, etc. Other types of processors may be used.GPUs can be particularly useful for accelerating graphics processing tasks and / or deep learning processing tasks (e.g., training and / or inference).

[0075] The input device(s) 404 and the output device(s) 410 may include one or more buttons, switches, pointing devices, a keyboard, a microphone, one or more sensors, a speaker, a buzzer, one or more lights, etc. One or more of these may be coupled via a wired connection (e.g., Ethernet, USB A, USB C, Thunderbolt (TM from Intel Corporation) or other communication channel. In one embodiment, the camera 446 is a camera capable of capturing and / or recording a field-of-view LDR portrait image limited vision. In one embodiment, camera 446 may be used to take selfies.

[0076] One or more communication units 408 may communicate with external computing devices via one or more networks by transmitting and / or receiving network signals over the one or more networks. The communication units 408 may include various antennas and / or network interface cards, etc. for wireless and / or wired communications.

[0077] The display screen 448 presents images such as components of a graphical user interface, camera images, etc. In one embodiment, the display screen is a touchscreen device, a type of I / O device, configured to receive gesture inputs (e.g., swipes, taps, etc.) that interact with the region(s) of the screen and in association with the user interface components (e.g., controls) presented by an application executed by the processor(s) 402.

[0078] The computing device 400 may include additional computing modules or data stores in various embodiments. Additional modules, data stores, and devices that may be included in various embodiments may not be shown in [Fig. 4] to avoid undue complexity of the description. Other examples of the computing device 400 may be a tablet computer, a personal digital assistant (PDA), a laptop computer, a desktop computer, a portable media player, a digital reader, a watch, a client device, a user device, or another type of computing device.

[0079] The storage device(s) 414 stores components of a light estimation application and related data (e.g., 416). Representative components are shown. The light estimation application and data 416 include the user interface component 438 (e.g., screens, instructions, icons, controls, etc.). The user interface provides output to a user and receives input such as input for the application workflow. An estimation model 420 is provided and includes an estimation model (e.g., 102) as previously described for predicting image lighting conditions (e.g., environment map prediction 422). A VTO pipeline 440 is provided for simulating hair rendering with environment-aware lighting in association with (e.g., on or in) an input image.

[0080] In one embodiment, a user may provide an input image (e.g., 418) similar to the input image 104, for lighting estimation. In one embodiment, the input image (e.g., 418) is captured and / or recorded by a camera (e.g., camera 446). The environment map prediction 422 is generated by the estimation model 420 using the input image 418 in accordance with an embodiment provided herein. The environment map prediction 422 may be used to generate a hair rendering 424 simulating hair adapted to the lighting conditions present in the input image 418. The hair rendering 424 may be displayed on the display screen 448. The user may invoke the light estimation application 416 (via input to a command) to have the application simulate hair in response to the lighting conditions present in an input image 418 using the image 418 and the environment map prediction 422 as input to produce a hair-rendered output image 444.

[0081] The hair-rendered output image 444 may be presented via the user interface component 438 on the display screen 448. The products or services or both may be purchased through the interface 442. Such an interface 442 may direct the user (e.g., the computing device) to a web-based e-commerce service (e.g., a website (not shown)) to make a purchase, reservation, or the like.

[0082] Other components stored in the storage device(s) 414 include an operating system 430, a browser 432 (e.g., for browsing web pages), an email and / or messaging application 434 (e.g., SMS or other type), and a social media application 436. The hair-rendered output image 444 may be shared (e.g., communicated) via the applications 434 and / or 436, for example.

[0083] [Fig. 5] shows a block diagram of an exemplary computing device configured to provide an estimation stream 100 in accordance with an embodiment of [Fig. 1]. In one embodiment, the computing device 105 is further configured to train the estimation model 102 by refinement associated with the video frames. Consecutive video frames 528, comprising two or more consecutive frames of an input video, are provided to the estimation model 102. In one exemplary embodiment, three consecutive frames of an input video are provided to the estimation model 102. The estimation model 102 is configured to provide a video frame prediction map 530, comprising a light environment map prediction for each of two or more consecutive video frames 528, to the coherence loss module 532.

[0084] In one embodiment, the coherence loss 534 is calculated by applying a per-pixel L1 loss between the video frame prediction map 530 of one of the consecutive video frames 428, and a sum of the predicted video frame map 530 for two additional consecutive video frames 528 according to the formula:

[0085] Loss Consistency = Ll((E(v l+] )), 0.5 * (E(vt) + E(v t+2 )))

[0086] in which E is the estimation model 102, vt indicates 528 consecutive video frames at time t, vt+1 indicates 528 consecutive video frames at time t+1, and vt+2 indicates 528 consecutive video frames at time t+2.

[0087] In one embodiment, the computing device 105 is configured to refine the training of the estimation model 102 using a loss function comprising the sum of the reconstruction loss 120, the light source color loss 114, the discriminator-based color loss 118, and the coherence loss 534.

[0088] A practical implementation may include some or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, program products, and other ways, combining the features described herein. A number of embodiments have been described. Nevertheless, it is understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided, or steps may be eliminated, from the described process, and other components may be added to or removed from the described systems. Accordingly, other embodiments fall within the scope of the claims that follow.

[0089] Throughout the description and claims of this specification, the words "include," "contain," and variations thereof mean "including but not limited to" and are not intended to (and do not) exclude other components, integers, or steps. Throughout this specification, the singular includes the plural, unless the context otherwise requires. In particular, when the indefinite article is used, this specification is to be understood as contemplating plurality as well as singularity, unless the context otherwise requires.

[0090] Features, integers, characteristics or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood as being applicable to any other aspect, embodiment or example, unless inconsistent therewith. Any features disclosed herein (including the claims, the abstract and the accompanying drawings), and / or any steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of these features and / or steps are mutually exclusive. The invention is not limited to the details of the foregoing examples or embodiments. The invention extends to any novel feature, or any novel combination, of the features disclosed herein (including any claim, the abstract and any accompanying drawing) or to any new feature, or any new combination, of the steps of any disclosed method or process. REFERENCES

[0091] The following documents are cited:

[0092] D. Calian, J.-F. Lalonde, PFU Gotardo, T. Simon, I. Matthews and K. Mitchell, “From Faces to Outdoor Light Probes”, Computer Graphics Forum, 37.2 / 2018

[0093] LeGendre, C., Ma, WC, Pandey, R., Fanello, S., Rhemann, C., Dourgarian, J., Busch, J. and Debevec, P., 2020. Leaming illumination from diverse portraits. In SIGGRAPH Asia 2020 Technical Communications (pp. 1-4).

[0094] Sztrajman, A., Neophytou, A., Weyrich, T. and Sommerlade, E., 2020, November. High-Dynamic-Range Lighting Estimation From Face Portraits. In 2020 International Conference on 3D Vision (3DV) (pp. 355—>363). IEEE.

[0095] Stratou, G., Ghosh, A., Debevec, P. et Morency, L.P., 2011, mars. Effect of illumination on automatic expression récognition: a novel 3D relightable facial database. Dans 2011 IEEE International Conférence on Automatic Face & Gesture Récognition (FG) (pp. 611-618). IEEE.

[0096] LeGendre, C., Ma, W.C., Fyffe, G., Flynn, J., Charbonnel, L., Busch, J. et Debevec, P., 2019. Deeplight: Leaming illumination for unconstrained mobile mixed reality. Dans Proceedings of the IEEE / CVF Conférence on Computer Vision and Pattern Récognition (pp. 5918-5928).

[0097] Somanath, G. et Kurz, D., 2021. HDR environment map estimation for real-time augmented reality. Dans Proceedings of the IEEE / CVF Conférence on Computer Vision and Pattern Récognition (pp. 11298—>11306).

[0098] Wang, G., Yang, Y., Loy, C.C., and Liu, Z., 2022, October. Stylelight: Hdr panorama generation for lighting estimation and editing. In European Conference on Computer Vision (pp. 477-492). Cham: Springer Nature Switzerland.

[0099] Gardner, MA, Hold-Geoffroy, Y., Sunkavalli, K., Gagné, C. and Lalonde, JF, 2019. Deep parametric indoor lighting estimation. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 7175-7183).

[0100] Gardner, MA, Sunkavalli, K., Yumer, E., Shen, X., Gambaretto, E., Gagné, C. and Lalonde, JF, 2017. Leaming to predict indoor illumination from a single image. arXiv preprint arXiv: 1704.00090.

[0101] Garon, M., Sunkavalli, K., Hadap, S., Carr, N. and Lalonde, JF, 2019. Fast spatially-varying indoor lighting estimation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 6908-6917).

Claims

Claims

1. A computing device comprising a processor coupled to a storage device storing instructions executable by the processor to cause the computing device to: (a) predict a predicted environment map for lighting conditions in an input image using a generator of a deep learning model, the predicted environment map encoding one or more light sources and a light source color estimate in the input image, the deep learning model having been defined by training using a discriminator that predicts one or more average color values ​​of a defined percentage of highest intensity pixels of an environment map,the discriminator being defined using a color loss determined from: (i) one or more average color values ​​of a defined percentage of highest intensity pixels of a ground truth environment map; and (ii) a prediction by the discriminator of a training-time predicted environment map generated by the generator; and (b) generating an output image comprising one or more objects of the input image to which one or more respective effects are applied, wherein a property of each of the respective effects is adapted to the predicted environment map.,

2. The computing device of claim 1, wherein the training-time predicted environment map encodes one or more predicted light sources and an estimate of the light source color of one of a plurality of synthetically generated images provided to the generator, the plurality of synthetically generated images having lighting conditions including a light source direction and a light source color.

3. The computing device of claim 2, wherein at least one of: (a) the plurality of synthetically generated images comprises a portrait image dataset containing combinations of subjects and environments sampled to cover a diversity of demographics, light source directions, and light source colors; (b) the plurality of synthetically generated image subjects comprises synthetic faces sampled based on various genders, ethnicities, and ages; (c) the plurality of synthetically generated images are generated to include instances of neutral facial expressions and extreme facial expressions and at different head positions and orientations; or (d) the plurality of synthetically generated images comprises images with a field of view established at a viewing angle between 50 and 70 degrees.

4. The computing device of claim 1, wherein the deep learning model is defined by training comprising generating one or more bounding boxes to align a center of a head in the input image and performing one or both of a rotation or transposition of an environment of the input image to align a center of the environment with the center of the head.

5. The computing device of claim 1, wherein the set percentage of the highest intensity pixels of the ground truth environment map is in the range of 5 percent to 10 percent; and wherein the pixel intensity of the ground truth environment map is set based on a perceptual brightness channel (L*) of the CIELAB color space.

6. The computing device of claim 1, wherein the generator comprises one or more convolution output layers and one or more upsampling blocks, each of the upsampling blocks comprising a convolution layer, a ReLu activation layer, and a bilinear upsampling layer.

7. The computing device of claim 1, wherein the input image comprises a limited field of view low dynamic range portrait image and the output image comprises a high dynamic range image.

8. The computing device of claim 1, wherein the effect applied to the one or more objects of the input image is a virtual try-on effect.

9. The computing device of claim 1, wherein the generator is defined by training using a reconstruction loss, the reconstruction loss being defined from an L2 loss applied to the training-time predicted environment map and the ground truth environment map.

10. The computing device of claim 1, wherein the generator is defined by training using a coherence loss, the coherence loss being determined from: (a) a first predicted environment map generated by the generator for the lighting conditions of a first consecutive video frame; and (b) the sum of a second predicted environment map generated by the generator for the lighting conditions of a second consecutive video frame and a third predicted environment map generated by the generator for the lighting conditions of a third consecutive video frame.

Images