Photo Relighting Using Deep Neural Networks and Confidence Learning

Abstract

Apparatus and methods related to applying an illumination model to an image of an object are provided. A neural network may be trained to apply the illumination model to an input image. The training of the neural network may utilize confidence learning based on light predictions and prediction confidence values ​​associated with illumination of the input image. A computing device may receive an input image of the object and data regarding a particular illumination model to be applied to the input image. The computing device may determine an output image of the object by applying the particular illumination model to the input image of the object using the trained neural network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 62 / 735,506, filed on September 24, 2018, entitled “Photo Relighting using Deep Neural Networks and Confidence Learning,” and U.S. Patent Application No. 62 / 749,081, filed on October 22, 2018, entitled “Photo Relighting using Deep Neural Networks and Confidence Learning,” the entire contents of which are incorporated herein by reference in their entirety for all purposes. Background Art

[0003] Many modern computing devices (including mobile phones, personal computers, and tablet computers) include image capture devices, such as still and / or video cameras. The image capture device can capture images, such as images including people, animals, landscapes, and / or objects.

[0004] Some image capture devices and / or computing devices can correct or modify captured images. For example, some image capture devices can provide "red eye" correction, which removes artifacts such as the red appearance of people and animals' eyes that may appear in images captured using bright lights such as flash lighting. After the captured image is corrected, the corrected image can be saved, displayed, transmitted, printed on paper, and / or otherwise utilized.

[0005] In one aspect, a computer-implemented method is provided. A neural network is trained to apply an illumination model to an input image. The training of the neural network utilizes confidence learning based on light predictions and prediction confidence values ​​associated with illumination of the input image. A computing device receives an input image of an object and data about a specific illumination model to be applied to the input image. The computing device determines an output image of the object by applying the specific illumination model to the input image of the object using the trained neural network.

[0006] In another aspect, a computing device is provided. The computing device includes one or more processors and a data storage device. The data storage device has computer executable instructions stored thereon, which, when executed by the one or more processors, cause the computing device to perform functions. The functions include: training a neural network to apply an illumination model to the input image using confidence learning based on light predictions and prediction confidence values ​​associated with illumination of the input image; receiving an input image of an object and data about a specific illumination model to be applied to the input image; and determining an output image of the object by applying the specific illumination model to the input image of the object using the trained neural network.

[0007] In another aspect, an article of manufacture is provided. The article of manufacture includes one or more computer-readable media having computer-readable instructions stored thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include: training a neural network to apply an illumination model to the input image using confidence learning based on light predictions and prediction confidence values ​​associated with illumination of the input image; receiving an input image of an object and data about a specific illumination model to be applied to the input image; and determining an output image of the object by applying the specific illumination model to the input image of the object using the trained neural network.

[0008] In another aspect, a computing device is provided. The computing device includes: means for training a neural network to apply an illumination model to the input image using confidence learning based on light predictions and prediction confidence values ​​associated with illumination of the input image; means for receiving an input image of an object and data about a specific illumination model to be applied to the input image; and means for determining an output image of the object by applying the specific illumination model to the input image of the object using the trained neural network.

[0009] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become more apparent by reference to the drawings and the following detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 An image with imperfect lighting is shown according to an example embodiment.

[0011] Figure 2 and Figure 3 Example images for training a convolutional neural network to vary image illumination are shown in accordance with example embodiments.

[0012] Figure 4 is a diagram depicting the training of a convolutional neural network for varying image illumination according to an example embodiment.

[0013] Figure 5 According to an example embodiment Figure 4 Block diagram of a convolutional neural network.

[0014] Figure 6 is a diagram showing a method according to an example embodiment Figure 4 Block diagram of confidence learning used by a convolutional neural network during training.

[0015] Figure 7-Figure 18 According to an example embodiment, Figure 4 Example images of faces generated by a convolutional neural network.

[0016] Fig.19 is a diagram illustrating the training and inference phases of a machine learning model according to an example embodiment.

[0017] Fig. 20 A distributed computing architecture according to an example embodiment is depicted.

[0018] Fig.21 is a block diagram of a computing device according to an example embodiment.

[0019] Fig. 22 Depicted is a network of computing clusters arranged as a cloud-based server system according to an example embodiment.

[0020] Fig.23 is a flowchart of a method according to an example embodiment. DETAILED DESCRIPTION

[0021] The present application relates to using machine learning techniques (such as, but not limited to, neural network techniques) to change the lighting of an image of an object (such as an object depicting a human face). When a mobile computing device user takes an image of an object (such as a person), the resulting image may not always have ideal lighting. For example, the image may be too bright or too dark, the light may come from an undesirable direction, or the lighting may include different colors that give the image an undesirable tint. Furthermore, even if the image does have the desired lighting at one time, the user may later want to change the lighting. Thus, a technical problem associated with image processing arises that involves adjusting the lighting of an image that has already been obtained.

[0022] In order to allow users to control the illumination of images, especially images of faces and similar objects, the technology described here applies a model based on a convolutional neural network to correct the illumination of the image. The technology described here includes: receiving an input image and data about a specific illumination model to be applied to the input image; using a convolutional neural network to predict an output image applying data about a specific illumination model to be applied to the input image; and generating an output based on the output image. The input and output images can be high-resolution images, such as images of millions of pixels in size captured by a camera of a mobile computing device. Convolutional neural networks can work well with input images captured under various natural and artificial lighting conditions. In some examples, the trained convolutional neural network model can work on various computing devices, including but not limited to mobile computing devices (e.g., smart phones, tablets, cellular phones, laptops), fixed computing devices (e.g., desktop computers), and server computing devices. Therefore, the convolutional neural network can apply a specific illumination model to the input image, thereby adjusting the illumination of the input image, and solving the technical problem of adjusting the illumination of the image that has been obtained.

[0023] The input image may be illuminated, and the illumination of the input image may be represented by a primitive illumination model. An illumination model, such as the specific illumination model or primitive illumination model described above, may be represented by a grid of lighting cells, wherein each lighting cell contains data related to the illumination of at least a portion of the corresponding image. The data related to the illumination of at least a portion of the corresponding image may represent one or more colors, intensities, albedos, directions, and / or surface normals of the illumination of the portion of the corresponding image. For example, an input image of 512×512 pixels may have a corresponding primitive illumination model represented by a 16×32 grid of lighting cells. Other examples of input images and grids of lighting cells are also possible, e.g., images of different sizes, larger illumination models, etc. In addition, more, less, and / or different data may be stored in the lighting cells of the lighting cell grid.

[0024] A neural network (such as a convolutional neural network) can be trained using a training dataset of images to apply the lighting model to images of objects (such as human faces). In some examples, the neural network can be arranged as an encoder / decoder neural network.

[0025] Although the examples described here involve determining an illumination model and applying it to images of objects with human faces, a neural network can be trained to determine an illumination model and apply it to images of other objects, such as objects that reflect light like human faces. Due to direct reflected light, human faces typically reflect light diffusely with some specular highlights. For example, specular highlights may be caused by direct light reflection from the surface of the eyes, glasses, jewelry, etc. In many images of human faces, the area ratio of such specular highlights is relatively small in proportion to the surface area of ​​the face that diffusely reflects light. Therefore, a neural network can be trained to apply an illumination model to images of other objects that diffusely reflect light, which may have some relatively small specular highlights (e.g., tomatoes or walls painted with matte paint). The images in the training data set can display one or more specific objects using lighting provided under multiple different conditions, such as lighting provided from different directions, lighting provided at different intensities (e.g., brighter and darker lighting), lighting provided by light sources of different colors, lighting provided by different numbers of light sources, etc.

[0026] Confidence learning can be used to train a neural network, where predictions can be weighted by one (or more) confidence values. For example, a neural network can generate light predictions for each of a plurality of "patches" or portions of an image. Then, as part of confidence learning, the light predictions for the patches can be mathematically combined (e.g., multiplied) with the confidence values ​​of the predictions, which are also generated by the neural network. In some examples, the confidence value can be an output value explicitly predicted and / or otherwise provided by the neural network. In some examples, the confidence value can be an implicit value based on one or more weights of the neural network used to provide the prediction (e.g., a specific weight value, a maximum weight value, an average weight value, a minimum weight value, or some other weight value determined by combining some or all weights in the neural network). Confidence learning allows the neural network to weight the lighting predictions based on the confidence values, resulting in better light predictions by weighting more confident predictions over less confident predictions, thereby improving the quality of the output lighting predictions, and therefore improving the quality of the predicted output image.

[0027] Once trained, the neural network can receive an input image and information about a desired illumination model. The trained neural network can process the input image to determine a prediction of an original illumination model for the illumination provided to capture the input image. The trained neural network can also process the image to apply a particular illumination model to the original image and predict an output image where the particular illumination model has been applied to the input image. The trained neural network can then provide an output including a predicted output image and / or a predicted original illumination model. In some examples, the neural network can be trained to predict only the original illumination model or only the output image.

[0028] In one example, (a copy of) the trained neural network may reside on a mobile computing device. The mobile computing device may include a camera that may capture an input image of an object, such as a portrait of a person's face. A user of the mobile computing device may view the input image and determine that the input image should be relighted. The user may then provide the input image and information about how the input image should be relighted to the trained neural network resident on the mobile computing device. In response, the trained neural network may generate a predicted output image that displays the input image relighted as indicated by the user, and then output the output image (e.g., providing the output image for display by the mobile computing device). In other examples, the trained neural network does not reside on the mobile computing device; instead, the mobile computing device provides the input image and information about how the input image should be relighted to a remotely located trained neural network (e.g., via the Internet or another data network). The remotely located convolutional neural network may process the input image and information about how the input image should be relighted as described above, and provide an output image to the mobile computing device that displays the input image relighted as indicated by the user. In other examples, non-mobile computing devices may also use the trained neural network to relight images, including images not captured by a camera of the computing device.

[0029] In some examples, the trained neural network can work in conjunction with other neural networks (or other software) and / or be trained to identify whether an input image of an object is poorly lit. Then, when it is determined that the input image is poorly lit, the trained neural network described herein can apply a corrective lighting model to the poorly lit input image, thereby correcting the poor lighting of the input image. The corrective lighting model can be selected based on user input and / or can be predetermined. For example, a user-input lighting model or a predetermined lighting model can provide "flat light" or illuminate the object with a lighting model that provides minimal contrast in the image (e.g., a model that minimizes the difference between brightly lit "highlights" and dimly lit "shadows" in the image).

[0030] In some examples, a trained neural network may take as input an input image and one or more illumination models and provide one or more resulting output images. The trained neural network may then determine the one or more resulting output images by applying each of the plurality of illumination models to the input image. For example, the one or more illumination models may include a plurality of illumination models that represent one (or more) light sources that change position, illumination color, and / or other characteristics in each of the plurality of illumination models. More specifically, the plurality of illumination models may represent one or more light sources, wherein at least one light source changes position (e.g., changes by a predetermined amount) between the provided models. In such a scheme, the resulting output image represents the input image as shown when the changing light source appears to rotate or otherwise move around one (or more) objects depicted in the input image. Similarly, the changing light source may change the color (e.g., a predetermined distance in a color space) between the provided models, so that the resulting output image represents the input image displayed with light of various colors. The plurality of output images may be provided as still images and / or video images. Other effects may be produced by causing the trained neural network to apply a plurality of illumination models to an image (or, relatedly, causing the trained neural network to apply a single illumination model to a plurality of input images).

[0031] Thus, the techniques described herein can improve images by applying more desirable and / or selectable lighting models to the images, thereby improving their actual and / or perceived quality. Improving the actual and / or perceived quality of images, including portraits of people, can provide emotional benefits to people who think their photos look better. The techniques are flexible, so a wide variety of lighting models can be applied to images of faces and other objects, particularly other objects with similar lighting characteristics. Furthermore, by changing the lighting model, different aspects of the image can be highlighted, which can lead to a better understanding of the objects depicted in the image.

[0032] Image relighting techniques using neural networks

[0033] Figure 1 An image 100 with imperfect lighting according to an example embodiment is shown. Image 100 includes image 110, image 120, and image 130. Image 110 is an image including a face with inconsistent lighting, where the left side of the face is brighter than the right side of the face. Image 120 is an image including a face with a relatively large number of shadows (including several shadows obscuring the face). Image 130 is an image including a face taken with dark lighting, which has a slightly green hue that can be considered "moody". Other examples of images with imperfect lighting and other types of imperfect lighting are also possible.

[0034] Figure 2 and Figure 3 Example images for training a convolutional neural network to vary image illumination are shown in accordance with example embodiments. Figure 2 The upper part of shows an image 210 which shows a scene with several light sources. Figure 2 The lower part of shows images 220, 230, 240, 250, which are images of faces captured under different lighting conditions. Each of images 220, 230, 240, 250 is an image of the same person and is taken from the same perspective and distance, for example, an image captured by a camera at the same distance and orientation as the person.

[0035] The images of the person may be captured while illuminated by each individual light source depicted in image 210. For example, image 220 may be captured while illuminated by a first light source depicted in image 210, image 230 may be captured while illuminated by a second light source depicted in image 210, image 240 may be captured while illuminated by a third light source depicted in image 210, etc. Each of images 220, 230, and 240 may be multiplied or otherwise combined with the color of each respective light source. Figure 2 It is shown that the pixels of image 220 can be multiplied by data representing the color of a first light source, the pixels of image 230 can be multiplied by data representing the color of a second light source, and the pixels of image 240 can be multiplied by data representing the color of a third light source. Then, after each image is combined with the color of the corresponding light source, the resulting color images can be added or otherwise combined to obtain image 250, which represents an image of a person taken using all of the several light sources of the scene shown in image 210, as shown in FIG. Figure 2 Thus, image 250 appears to have been taken using the lighting provided in the scene shown in image 210. In some examples, some or all of images 220, 230, 240, 250 may be provided as part of a one-light-at-a-time (OLAT) dataset.

[0036] Figure 3Indoor images 310, 320, 330 and outdoor images 330, 350, 360 representing indoor lighting conditions are shown, which can be used to train the convolutional neural network described herein. The images including images 310, 320, 330, 340, 350, 360 can be part of a training data set of images, where the training data set can include these images representing indoor and lighting conditions, as well as one or more OLAT data sets, other object images such as faces, and possibly other images. The resulting training data set of images can then be used to train the convolutional neural network described herein.

[0037] Figure 4 is a diagram depicting the training of a convolutional neural network 430 for changing image illumination according to an example embodiment. Convolutional neural network 430 may be a fully convolutional neural network as described herein. During training, convolutional neural network 430 may receive as input one or more input training images and an input target illumination model. For example, Figure 4 A convolutional neural network 430 is shown trained on an input raw image 410 and an input target lighting model (TLM) 420. During training, the convolutional neural network 430 is directed to generate predictions of a target image 440 and predictions of a raw lighting model 450. The target image 440 may be a predicted (or generated) image produced by applying the input target lighting model 420 to the input raw image 410. Thus, the target image 440 may be a prediction of how the raw image 410 will appear illuminated by the input target lighting model 420, rather than a prediction of the actual lighting conditions that illuminated the raw image 410 when the raw image 410 was originally captured. In this way, the target image 440 predicts how the raw image 410 will appear re-illuminated by the target lighting model 420.

[0038] The original illumination model 450 is an illumination model that predicts actual illumination conditions for illuminating the original image 410. The illumination model may include a grid or other arrangement of illumination model data related to illumination of part or all of one or more images. The illumination model data may include, but is not limited to, data representing one or more colors, intensities, albedos, directions, and / or surface normals of illumination of part or all of one or more images. Thus, each of the illumination models 420, 450 may include illumination model data related to illumination of at least a portion of the corresponding image. The target illumination model 420 may relate to illumination of the target image 440, and the original illumination model 450 may relate to illumination of the original image 410.

[0039] Figure 4Also shown are example sizes of the images and illumination models - each of the original image and the target image is an image of size 512 x 512 pixels, and each of the target illumination model 420 and the original illumination model 450 is a grid of 16 x 32 cells of illumination model data. Other sizes of the original image, original illumination model, target image, and target illumination model are possible.

[0040] Figure 5 4 is a block diagram of a convolutional neural network 430 according to an example embodiment. The convolutional neural network 430 may receive as input an original image 410 and a target illumination model 420, as shown in the upper left and lower right corners, respectively. The convolutional neural network 430 may process the original image 410 and the target illumination model 420. The original image 410 may be processed to determine a prediction of an original illumination model (OLM) 450 and provide an input to determine a prediction of a target image 440. The target illumination model 420 may be used to determine a prediction of the target image 440.

[0041] The convolutional neural network 430 may include a node layer for processing the original image 410 and the target illumination model 420. Example layers may include, but are not limited to, an input layer, a convolutional layer, an activation layer, a pooling layer, a fully connected layer, and an output layer. The input layer may store input data, such as pixel data of the original image 410 and inputs from other layers of the convolutional neural network 430. The convolutional layer may calculate the output of neurons connected to local areas in the input. In some cases, the convolutional layer may act as a transposed convolution or deconvolution layer to apply a filter to a relatively small input, thereby expanding / upsampling the relatively small input into a larger output. The activation layer may determine whether the output of the previous layer is "activated" or actually provided (e.g., provided to a later layer). The pooling layer may downsample the input. For example, the convolutional neural network 430 may include one or more pooling layers that downsample the input by a predetermined factor (e.g., factor 2) in the horizontal and / or vertical dimensions. The fully connected layer may determine a score associated with the prediction. Such a score may include, but is not limited to, a score associated with the predicted original illumination model, and / or a score associated with the predicted target image. The output layer can provide the output of the convolutional neural network 430 to software and / or hardware interfacing with the convolutional neural network 430; for example, hardware and / or software for displaying, printing, communicating, and / or otherwise providing the target image 440. Other layers, such as batch normalization layers, can also be in the convolutional neural network 430. The layers 510, 512, 514, 516, 520, 522, 524, 530, 532, 534, 540, 542, 544, 546 can include one or more input layers, output layers, convolutional layers, activation layers, pooling layers, fully connected layers, and / or other layers described herein.

[0042] exist Figure 5 and Figure 6 4, the layers of the convolutional neural network 430 involved in processing the original image 410 and the target illumination model 420 to determine the prediction of the target image 440 are shown using grey blocks. In addition, information of the convolutional neural network 430 used to determine the prediction of the original illumination model and in processing the target illumination model 420 is shown using yellow blocks. More specifically, the convolutional neural network 430 may include original layers 510, 512, 514, 516, which are arranged in the order of layers L1, L2, L3, L4, respectively, each layer successively convolving its input and providing its output to the successive layers until the original layer L4 516 is reached.

[0043] The output layer L4 516 can be associated with a first original illumination model (OLM) information layer 520, which can provide outputs to a second original illumination model information layer 522, which can in turn provide outputs to a third original illumination model information layer 524, which can include an output layer to provide a predicted original illumination model 450. For example, the original illumination model information layers 520, 522, 524 can include one or more fully connected layers for predicting the original illumination model 450. In some examples, the original illumination model information layer 520 can receive illumination-related features of the original image 410 determined by the original layers 510, 512, 514, 516. For example, the original layer 516 can output or otherwise provide the illumination-related features to the original illumination model information layer 520.

[0044] The first target lighting model (TLM) information layer 530 may act as an input layer to receive the target lighting model 420 as input. The target lighting model information layer 530 may provide an output to a second target lighting model information layer 532, which in turn may provide an output to a third target lighting model information layer 534. The target lighting model information layer 534 may include an output layer to provide lighting features related to the target lighting model 420. For example, the target lighting model information layers 530, 532, 534 may include a fully connected layer for predicting the original lighting model 450. In some examples, the original lighting model information layer 520 may receive lighting-related features of the original image 410 determined by the original layers 510, 512, 514, 516. For example, the original layer 516 may output or otherwise provide lighting-related features to the original lighting model information layer 520.

[0045] exist Figure 5 and Figure 6, the depicted layers may include one or more actual layers. For example, original layer L1 510 may have one or more input layers, one or more activation layers, and / or one or more additional layers. As another example, original layer L2 512, original layer L3 514, and / or original layer L4 516 may include one or more convolutional layers, one or more activation layers (e.g., having a one-to-one relationship with one or more convolutional layers), one or more pooling layers, and / or one or more additional layers.

[0046] In some examples, some or all of the pooling layers in the convolutional neural network 430 may downsample the input by a common factor in both the horizontal and vertical dimensions without downsampling the depth dimension associated with the input. The depth dimension may store data for pixel color (red, green, blue) and / or data representing a fraction. For example, assume that the original image 410 is of size 512×512 and has a depth of D, each of the original layers 510, 512, 514, 516 includes a pooling layer, and each of the original layers 510, 512, 514, 516 downsamples the original image 410 by a factor of 2 in both the horizontal and vertical dimensions. In this case, the output size of the original layer 510 will be 256×256×D, the output size of the original layer 512 will be 128×128×D, the output size of the original layer 514 will be 64×64×D, and the output size of the original layer 516 will be 32×32×D. One or more (pooling) layers of the convolutional neural network 430 may also use a common factor other than two for downsampling.

[0047] The original layer L1 510 may receive and process the original image 410 and provide an output to the original layer L2 512. The original layer L2 512 may process the output of the original layer L1 and provide an output to the original layer L3 514. The original layer L3 514 may process the output of the original layer L2 and provide an output to the original layer L4 516. The original layer L4 516 may process the output of the original layer L3. At least a portion of the output of the original layer L4 516 may be provided as an input to the original illumination model information layer 520.

[0048] The convolutional neural network 430 may use the original illumination model information layers 520, 522, and 524 to predict the original illumination model 450. The original illumination model 450 may be output by the illumination model information layer 524 of the convolutional neural network 430. The convolutional neural network 430 may use confidence learning to train the original illumination model information layers 520, 522, and 524 to predict the original illumination model 450. Here, at least Figure 6 Confidence learning is discussed in more detail in the context of .

[0049] To predict the target image 440, the convolutional neural network 430 may process the target illumination model 420 using the target illumination model information layers 530, 532, and 534. The output of the target illumination model information layer 534 may be provided as input to the target layer L1 540 along with data copied from the original layer L4 516 (e.g., using a skip connection between the original layer L4 516 and the target layer L1 540) to begin predicting the target image 440. The target layer L2 542 may receive and process input from the target layer L1 540 and the original layer L3 514 (e.g., using a skip connection between the original layer L3 514 and the target layer L2 542) to provide an output to the target layer L3 544. The target layer L3 544 may receive and process inputs from the target layer L2 542 and the original layer L2 512 (e.g., using a skip connection between the original layer L2 512 and the target layer L3 544) to provide an output to the target layer L4 546. The target layer L4 546 may receive and process inputs from the target layer L3 544 and the original layer L1 510 to provide a prediction of the target image 440, which may then be output from the target layer L4 546. The data provided by the skip connections between the original layers 516, 514, 512, 510 and the respective target layers 540, 542, 544, 546 may be used by each respective target layer to provide additional detail for generating the target layer's contribution to the prediction of the target image 440. In some examples, each of the target layers 540, 542, 544, 546 for predicting the target image 440 can include one or more convolutional layers (possibly performing transposed convolutions / deconvolutions), one or more activation layers, and possibly one or more input and / or output layers. In some examples, some or all of the layers 510, 512, 514, 516, 520, 522, 524, 530, 532, 534, 540, 542, 544, 546 can act as a convolutional encoder / decoder network.

[0050] A loss measure may be used during training of the convolutional neural network 430. For example, during training of the convolutional neural network 430 for predicting the target image 440, an L2 loss measure between the target image prediction and the training image may be minimized. As another example, during training of the convolutional neural network 430 for predicting the original illumination model 450, a log L1 loss measure between the original illumination model prediction and the training illumination model data may be minimized. Other loss measures may also be used, or may alternatively be used.

[0051] In some examples, the convolutional neural network 430 can include perceptual loss processing. For example, the convolutional neural network 430 can use a generative adversarial net (GAN) loss function to determine whether part or all of an image would be predicted to be actually illuminated by a particular illumination model, thereby satisfying one or more perceptual-related conditions for the illumination of that part of the image. In some examples, a cycle loss can be used to feed the predicted target image and / or the original illumination model back into the convolutional neural network 430 to generate and / or further refine the predicted target image and / or the original illumination model. In some examples, the convolutional neural network 430 can utilize deep supervision techniques to provide constraints on intermediate layers. In some examples, the convolutional neural network 430 can have a loss function that is more than the perceptual loss function. Figure 5 More, fewer and / or different layers as shown.

[0052] Figure 6 6 is a block diagram illustrating confidence learning 630 used during training of a convolutional neural network 430 according to an example embodiment. As described above, the original lighting model information layers 520, 522, and 524 may be used to predict the original lighting model 450. Confidence learning may be used during training of one or more of the original lighting model information layers 520, 522, and 524.

[0053] To determine the original lighting model 450, the original lighting model information layers 520, 522, and 524 may determine predictions about lighting characteristics, such as predictions about light direction. Figure 6 The upper portion of FIG. 4 shows the location "patch 1" of the original image 410, and a possible predicted normal direction 620 of a ray falling on the face at patch 1 is shown using a red arrow. The possible predicted normal direction 620 faces outward from the depicted face at patch 1 because the ray falling on patch 1 may be emitted from outside the depicted face and thus has a normal incident on the depicted face from the face outward. In contrast, Figure 6 The non-predicted normal direction 622 depicted using a blue arrow in the image would be associated with light coming from the interior of the face depicted at patch 1, which is unlikely. After training of the convolutional neural network 430, light coming from the non-predicted normal direction 622 should not be predicted. However, during training of the original illumination model information layers 520, 522, and / or 524, an illumination model indicating light coming from the non-predicted normal direction 622 may be predicted, wherein the confidence that the light may be coming from the non-predicted normal direction 622 decreases as training proceeds.

[0054] Confidence learning 630 can be used to apply such confidence information during training of the convolutional neural network 430. For example, when training the original illumination model information layers 520, 522, and / or 524, the convolutional neural network 430 (e.g., original layer L4 516 and / or the original illumination model information layers 520, 522, 524) can determine the original illumination information 610. The original illumination information 610 can include illumination model predictions at a patch or portion of an image; for example, light predictions 640 regarding the normal direction and / or other properties of light falling on patch 1 of a face depicted in the original image 410. In addition, the convolutional neural network 430 (e.g., original layer L4 516 and / or the original illumination model information layers 520, 522, 524) can determine a confidence value 650 associated with the prediction 640 of the illumination model at patch 1.

[0055] The convolutional neural network 430 may be used to explicitly predict the confidence value 650 or implicitly provide the confidence value 650 based on some or all of the weights of the convolutional neural network 430. In some examples, the original layer L4 516 and / or one or more of the original illumination model information layers 520, 522, 524 may be used to explicitly predict the confidence value 650 or implicitly provide the confidence value 650 based on some or all of the weights of the original layer L4 516 and / or one or more of the original illumination model information layers 520, 522, 524. More specifically, one or more weights of the original layer L4 516 and / or one or more of the original illumination model information layers 520, 522, 524 may be used as the confidence value 650. Confidence learning 630 may then involve multiplying and / or otherwise mathematically combining the light prediction 640 with the predicted confidence value 650 to determine an updated light prediction 660. Compared to using light predictions 640 during training, using updated light predictions 660 generated by confidence learning 630 can result in emphasizing relatively-confident predictions over relatively-non-confident predictions during training of convolutional neural network 430, thereby providing additional use and feedback of confidence in lighting predictions. Other examples and / or uses of confidence learning are also possible.

[0056] Figure 7-Figure 18 An example image of a human face generated by a convolutional neural network 430 according to an example embodiment is shown. In particular, Figure 7-Figure 9 An example image of a human face associated with an illumination model prediction made by a convolutional neural network 430 is shown, Figure 10-Figure 18 An example image of a human face is shown in relation to the target model prediction made by the convolutional neural network 430. In general, Figure 7-Figure 18It is shown that the convolutional neural network 430 can generate accurate predictions of the lighting model and re-lit face images in a wide range of lighting environments.

[0057] Figure 7 An image 700 is shown, which was captured while illuminated by light modeled by the “Groundtruth Original Light” indicated in the lower left corner of the image 700. The convolutional neural network 430 predicts the illumination model shown by the “Predicted Original Light” in the lower right corner of the image 700. Figure 7 The real original light and the predicted original light depicted in the figure are based on Figure 7 Environment map shown on the left. The environment map indicates that the upper portion of both the real original light and the predicted original light is associated with light from the rear of the environment depicted in image 700, and indicates that the lower portion of both the real original light and the predicted original light is associated with light from the front of the environment depicted in image 700. The environment map also indicates that the left portion of both the real original light and the predicted original light is associated with light on the left side of the face depicted in image 700, which corresponds to the right side of image 700, and also indicates that the right portion of both the real original light and the predicted original light is associated with light on the right side of the face depicted in image 700, which corresponds to the left side of image 700. Figure 7 Depicted and used in Figure 7 The same environment diagram is also depicted in Figure 8-Figure 18 The lighting model shown in Figure 8-Figure 18 The lighting model shown in .

[0058] Figure 7 The real original light and predicted original light for image 700 are shown to both have a bright portion in the upper left, indicating that most of the light of the environment of image 700 is predicted to, and actually is, coming from behind the face depicted in image 700 and falling to the left of the face, which is shown on the right side of image 700. Image 700 confirms the prediction made by the predicted original light that image 700 is brighter illuminated on the right side than on the left side.

[0059] Figure 8 The lower left portion of image 800 is shown showing the real original lighting of the image, while the lower right portion of image 800 shows the predicted original lighting of the image, where the predicted original lighting of image 800 is generated by convolutional neural network 430. Both the real original lighting and the predicted original lighting of image 800 have bright portions in the upper right portion, indicating that most of the light of the environment of image 800 is predicted to and actually comes from behind the face depicted in image 800 and falls on the right side of the face, which is shown on the left side of image 800. Image 800 confirms the prediction made by the predicted original lighting that image 800 is brighter illuminated on the left side than on the right side.

[0060] Fig. 9 The lower left portion of image 900 shows the true original light of the image, while the lower right portion of image 900 shows the predicted original light of the image, where the predicted original light of image 900 is generated by convolutional neural network 430. Both the true original light and the predicted original light of image 900 have a relatively large bright portion in the upper right portion and a relatively small bright portion in the upper left portion, indicating that most of the light of the environment of image 900 is predicted to and actually comes from two light sources: a larger light source behind the face depicted in image 900, from which light falls on the right side of the face shown on the left side of image 900; and a smaller light source behind the face depicted in image 900, from which light falls on the left side of the face shown on the right side of image 900. Image 900 confirms the prediction made by the predicted original light that image 900 is brighter illuminated on the left side than on the right side, as is image 800, but is more evenly illuminated across the face than image 800.

[0061] Figure 10-Figure 18 Each of 4 shows a set of three images: an original image; a true target image with a corresponding target illumination model; and a predicted target image generated by a trained version of the convolutional neural network 430 by applying the target illumination model to the original image. For example, Fig.10 An original image 1010, a real target image 1020, and a predicted target image 1030 are shown, wherein the original image 1010 is shown with an environment map of the original illumination model indicating that the image 1010 is backlit with relatively uniform light. Both the real target image 1020 and the predicted target image 1030 are shown with an environment map of the target illumination model showing three light sources, one of which is further forward in the environment than the light used for the original image 1010. Both the real target image 1020 and the predicted target image 1030 show similar illumination reflected by the target illumination model shown in each respective image.

[0062] Fig.11 An original image 1110, a true target image 1120, and a predicted target image 1130 are shown, wherein original image 1110 is shown with an environment map of the original lighting model indicating that image 1110 is backlit with a relatively dim light from a relatively small light source near a face depicted in image 1110. Both true target image 1120 and predicted target image 1130 are shown with an environment map of the target lighting model showing relatively bright backlighting compared to original image 1110. Both true target image 1120 and predicted target image 1130 show similar lighting reflecting the target lighting model shown in each respective image.

[0063] Fig.12An original image 1210, a real target image 1220, and a predicted target image 1230 are shown, wherein the original image 1210 is shown with an environment map of the original lighting model, the environment map indicating that the image 1210 is backlit with a stronger light on the left side of the depicted face (shown on the right side of the image 1210) than on the right side of the depicted face (shown on the left side of the image 1210). Both the real target image 1220 and the predicted target image 1230 are shown with an environment map of the target lighting model, the environment map showing one relatively large light source dominating the lighting environment. Both the real target image 1220 and the predicted target image 1230 show similar lighting reflecting the target lighting model shown in each respective image. However, the predicted target image 1230 is slightly darker than the target image 1220, possibly reflecting the relatively dark lighting of the input original image 1210.

[0064] Fig.13 An original image 1310, a real target image 1320, and a predicted target image 1330 are shown, where original image 1310 is shown with an environment map of the original lighting model indicating that image 1310 is backlit with relatively uniform light. Both real target image 1320 and predicted target image 1330 are shown with an environment map of the target lighting model showing two light sources, with the larger light source to the right of the face depicted in image 1320. Both real target image 1320 and predicted target image 1330 show similar lighting reflecting the target lighting model shown in each respective image.

[0065] Fig.14 An original image 1410, a real target image 1420, and a predicted target image 1430 are shown, wherein original image 1410 is shown with an environment map of the original lighting model indicating that image 1410 is backlit with a relatively large white light source. Both real target image 1420 and predicted target image 1430 are shown with an environment map showing a target lighting model of three light sources, a relatively large white light source to the left of a face depicted in image 1420, a relatively small white light source to the right of the face depicted in image 1420, and a relatively large yellow light source located in the center of the lighting environment of image 1420. Both real target image 1420 and predicted target image 1430 show similar lighting reflecting the target lighting model shown in each respective image.

[0066] Fig.15An original image 1510, a real target image 1520, and a predicted target image 1530 are shown, wherein original image 1510 is shown with an environment map of the original lighting model indicating that image 1510 is backlit with relatively uniform light proximate to the left and right sides of a face depicted in image 1510. Both real target image 1520 and predicted target image 1530 are shown with an environment map of the target lighting model showing one light source to the left of the face depicted in image 1520. Both real target image 1520 and predicted target image 1530 show similar lighting reflecting the target lighting model shown in each respective image. However, the lighting on the face depicted in image 1530 is darker than the lighting on the face depicted in image 1520.

[0067] Fig.16 An original image 1610, a true target image 1620, and a predicted target image 1630 are shown, where original image 1610 is shown with an environment map of an original lighting model indicating that image 1610 is primarily illuminated by a light source to the left of a face depicted in image 1610. Both true target image 1620 and predicted target image 1630 are shown with an environment map of a target lighting model dominated by a relatively bright light source to the right of the face depicted in image 1620. Both true target image 1620 and predicted target image 1630 show similar lighting reflecting the target lighting model shown in each respective image.

[0068] Fig.17 An original image 1710, a true target image 1720, and a predicted target image 1730 are shown, wherein original image 1710 is shown with an environment map of an original lighting model indicating that image 1710 is primarily illuminated by a light source to the right of a face depicted in image 1710. Both true target image 1720 and predicted target image 1730 are shown with an environment map showing a target lighting model of three light sources, two of which are relatively large and are used to backlight the face depicted in image 1720 from the left and right, and a third, relatively smaller light source to the left of the face depicted in image 1720. Both true target image 1720 and predicted target image 1730 show similar lighting reflecting the target lighting model shown in each respective image.

[0069] Fig.18An original image 1810, a real target image 1820, and a predicted target image 1830 are shown, wherein original image 1810 is shown with an environment map of the original lighting model indicating that image 1810 is primarily illuminated by one white light source relatively close to a face depicted in image 1810. Both real target image 1820 and predicted target image 1830 are shown with an environment map of a target lighting model showing three yellow light sources. Two of the light sources are relatively large and primarily backlight the face depicted in image 1820 from the left. Another light source is relatively small and is located to the right of and close to the face depicted in image 1820. Both real target image 1820 and predicted target image 1830 show similar yellow-hued lighting, which reflects the target lighting model shown in each respective image.

[0070] Train machine learning models to generate inferences / predictions

[0071] Fig.19 A diagram 1900 is shown illustrating a training phase 1902 and an inference phase 1904 of a trained machine learning model 1932 according to an example embodiment. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to identify patterns in the training data and provide output inferences and / or predictions about (the patterns in) the training data. The resulting trained machine learning algorithm may be referred to as a trained machine learning model. For example, Fig.19 A training phase 1902 is shown, wherein one or more machine learning algorithms 1920 are being trained on training data 1910 to become a trained machine learning model 1932. Then, during an inference phase 1904, the trained machine learning model 1932 may receive input data 1930 and one or more inference / prediction requests 1940 (perhaps as part of the input data 1930), and in response provide as output one or more inferences and / or predictions 1950.

[0072] Thus, the trained machine learning model 1932 may include one or more models of one or more machine learning algorithms 1920. The machine learning algorithm 1920 may include, but is not limited to, an artificial neural network (e.g., a convolutional neural network using the confidence learning technique described herein, a recurrent neural network), a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system. The machine learning algorithm 1920 may be supervised or unsupervised, and may implement any suitable combination of online and offline learning.

[0073] In some examples, the machine learning algorithm 1920 and / or the trained machine learning model 1932 can be accelerated using an on-device coprocessor (e.g., a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), and / or an application-specific integrated circuit (ASIC)). Such an on-device coprocessor can be used to accelerate the machine learning algorithm 1920 and / or the trained machine learning model 1932. In some examples, the trained machine learning model 1932 can be trained, reside, and executed to provide reasoning on a particular computing device and / or can make reasoning for a particular computing device.

[0074] During the training phase 1902, the machine learning algorithm 1920 may be trained by providing at least training data 1910 as training input using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques. Unsupervised learning involves providing a portion (or all) of the training data 1910 to the machine learning algorithm 1920, and the machine learning algorithm 1920 determines one or more output inferences based on the portion (or all) of the provided training data 1910. Supervised learning involves providing a portion of the training data 1910 to the machine learning algorithm 1920, the machine learning algorithm 1920 determines one or more output inferences based on the portion of the provided training data 1910, and accepting or correcting the output inferences based on correct results associated with the training data 1910. In some examples, supervised learning of the machine learning algorithm 1920 may be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or the set of labels may be used to correct the inferences of the machine learning algorithm 1920.

[0075] Semi-supervised learning involves having a correct result for a portion of the training data 1910, but not all. During semi-supervised learning, supervised learning is used for a portion of the training data 1910 with the correct result, and unsupervised learning is used for a portion of the training data 1910 that does not have the correct result. Reinforcement learning involves the machine learning algorithm 1920 receiving a reward signal about a previous reasoning, where the reward signal can be a numerical value. During reinforcement learning, the machine learning algorithm 1920 can output an inference and receive a reward signal as a response, wherein the machine learning algorithm 1920 is configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum of the numerical values ​​provided by the reward signal over time. In some examples, other machine learning techniques can be used to train the machine learning algorithm 1920 and / or the trained machine learning model 1932, including, but not limited to, incremental learning and curriculum learning.

[0076] In some examples, the machine learning algorithm 1920 and / or the trained machine learning model 1932 may use transfer learning techniques. For example, the transfer learning technique may involve the trained machine learning model 1932 being pre-trained on a set of data and being additionally trained using the training data 1910. More specifically, the machine learning algorithm 1920 may be pre-trained on data from one or more computing devices, and the resulting trained machine learning model is provided to the computing device CD1, wherein CD1 is intended to execute the trained machine learning model during the inference phase 1904. Then, during the training phase 1902, the pre-trained machine learning model may be additionally trained using the training data 1910, wherein the training data 1910 may be derived from the kernel and non-kernel data of the computing device CD1. This further training of the machine learning algorithm 1920 and / or the pre-trained machine learning model using the training data 1910 of the data of CD1 may be performed using supervised or unsupervised learning. Once the machine learning algorithm 1920 and / or the pre-trained machine learning model has been trained on at least the training data 1910, the training phase 1902 may be completed. The machine learning model obtained as a result of the training can be used as at least one of the trained machine learning models 1932.

[0077] In particular, once the training phase 1902 has been completed, the trained machine learning model 1932 can be provided to the computing device, if not already on the computing device. The inference phase 1904 can begin after the trained machine learning model 1932 is provided to the computing device CD1.

[0078] During the inference phase 1904, the trained machine learning model 1932 may receive input data 1930 and generate and output one or more corresponding inferences and / or predictions 1950 about the input data 1930. In this way, the input data 1930 may be used as an input to the trained machine learning model 1932 for providing corresponding inferences and / or predictions 1950 to kernel components and non-kernel components. For example, the trained machine learning model 1932 may generate inferences and / or predictions 1950 in response to one or more inference / prediction requests 1940. In some examples, the trained machine learning model 1932 may be executed by a portion of other software. For example, the trained machine learning model 1932 may be executed by an inference or prediction daemon so that it is readily available for providing inferences and / or predictions upon request. The input data 1930 may include data from the computing device CD1 executing the trained machine learning model 1932 and / or input data from one or more computing devices other than CD1.

[0079] Input data 1930 may include a collection of images provided by one or more sources. The collection of images may include images of an object (e.g., a face, where the images of the face are taken under different lighting conditions), images of multiple objects, images residing on the computing device CD1, and / or other images. Other types of input data are also possible.

[0080] Inferences and / or predictions 1950 may include output images, output lighting models, numerical values, and / or other output data produced by a trained machine learning model 1932 operating on input data 1930 (and training data 1910). In some examples, the trained machine learning model 1932 may use the output inferences and / or predictions 1950 as input feedback 1960. The trained machine learning model 1932 may also rely on past inferences as input for generating new inferences.

[0081] Convolutional neural network 430 may be an example of machine learning algorithm 1920. After training, the trained version of convolutional neural network 430 may be an example of trained machine learning model 1932. In this method, an example of inference / prediction request 1940 may be a request to apply a particular lighting model to an input image of an object, and a corresponding example of inference and / or prediction 1950 may be a prediction of an output image with the particular lighting model applied to the input image.

[0082] In some examples, a computing device CD_SOLO may include a trained version of the convolutional neural network 430, possibly after training the convolutional neural network 430. The computing device CD_SOLO may then receive a request to apply a specific illumination model to a corresponding input image, and use the trained version of the convolutional neural network 430 to generate an output image that predicts the application of the specific illumination model to the input image. In some of these examples, the request for an output image received by CD_SOLO that predicts the application of a specific illumination model to the input image may include a request for an original illumination model, or be replaced by a request for an original illumination model, each of which may model the illumination that illuminates the corresponding input image. CD_SOLO may then use the trained version of the convolutional neural network 430 to generate the output image and / or the original illumination model according to the request.

[0083] In some examples, two or more computing devices CD_CLI and CD_SRV may be used to provide an output image; for example, a first computing device CD_CLI may generate and send a request to a second computing device CD_SRV to apply a specific illumination model to a corresponding input image. CD_SRV may then use a trained version of the convolutional neural network 430, possibly after training the convolutional neural network 430, to generate an output image that predicts the application of the specific illumination model to the input image, and respond to the request for the output image from CD_CLI. CD_CLI may then, upon receiving a response to the request, provide the requested output image (e.g., using a user interface and / or display, a printed copy, an electronic communication, etc.). In some examples, the request for an output image that predicts the application of a specific illumination model to the input image may include or be replaced by a request for an original illumination model, each of which may model the illumination that illuminates the corresponding input image. CD_SRV may then use the trained version of the convolutional neural network 430 to generate the output image and / or the original illumination model in accordance with the request. Other examples for generating an output image that predicts the application of a particular lighting model to an input image and / or for generating an original lighting model using a trained version of convolutional neural network 430 are also possible.

[0084] Sample Data Network

[0085] Fig. 20A distributed computing architecture 2000 is depicted according to an example embodiment. The distributed computing architecture 2000 includes server devices 2008, 2010, which are configured to communicate with programmable devices 2004a, 2004b, 2004c, 2004d, 2004e via a network 2006. The network 2006 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, an intranet, the public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 2006 may also correspond to a combination of one or more LANs, WANs, intranets, and / or the public Internet.

[0086] although Fig. 20 Only five programmable devices are shown, but the distributed application architecture can serve dozens, hundreds or thousands of programmable devices. In addition, programmable devices 2004a, 2004b, 2004c, 2004d, 2004e (or any other programmable device) can be any kind of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mounted device (HMD), a network terminal, a mobile computing device, etc. In some examples, such as shown by programmable devices 2004a, 2004b, 2004c, 2004e, the programmable device can be directly connected to the network 2006. In other examples, such as shown by programmable device 2004d, the programmable device can be indirectly connected to the network 2006 via an associated computing device (such as programmable device 2004c). In this example, programmable device 2004c can act as an associated computing device to pass electronic communications between programmable device 2004d and network 2006. In other examples, as shown by programmable device 2004e, the computing device may be part of and / or within a vehicle, such as a car, truck, bus, boat or ship, airplane, etc. Fig. 20 In other examples not shown, programmable devices can be connected to network 2006 directly or indirectly.

[0087] As requested by programmable devices 2004a-2004e, server devices 2008, 2010 may be configured to perform one or more services. For example, server devices 2008 and / or 2010 may provide content to programmable devices 2004a-2004e. The content may include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or unencrypted. Other types of content are also possible.

[0088] As another example, server devices 2008 and / or 2010 may provide programmable devices 2004a-2004e with access to software for database, search, computing, graphics, audio, video, web / internet utilization, and / or other functions. Many other examples of server devices are possible.

[0089] Computing device architecture

[0090] Fig.21 is a block diagram of an example computing device 2100 according to an example embodiment. Specifically, Fig.21 The computing device 2100 shown in the figure can be configured to perform at least one function of convolutional neural network, confidence learning, predicting target image, predicting original lighting model, convolutional neural network 430, confidence learning 630 and / or method 2300 and / or functions related thereto.

[0091] The computing device 2100 may include a user interface module 2101, a network communication module 2102, one or more processors 2103, data storage 2104, one or more cameras 2118, one or more sensors 2120 and a power system 2122, all of which may be linked together via a system bus, network or other connection mechanism 2105.

[0092] User interface module 2101 can be operable to send data to external user input / output device and / or receive data from external user input / output device.For example, user interface module 2101 can be configured to send data to user input device and / or receive data from user input device, and user input device is such as touch screen, computer mouse, keyboard, keypad, touch pad, trackball, joystick, speech recognition module and / or other similar devices.User interface module 2101 can also be configured to provide output to user display device, such as one or more cathode ray tubes (CRT), liquid crystal display, light emitting diode (LED), display, printer, light bulb and / or other similar devices known or developed later using digital light processing (DLP) technology.User interface module 2101 can also be configured to utilize equipment such as loudspeaker, speaker jack, audio output port, audio output device, earphone and / or other similar devices to generate audible output.User interface module 2101 can also be configured with one or more tactile devices, which can generate tactile output, such as vibration and / or other outputs detectable by touch and / or with physical contact of computing device 2100. In some examples, user interface module 2101 may be used to provide a graphical user interface (GUI) for utilizing computing device 2100 .

[0093] The network communication module 2102 may include one or more devices that provide one or more wireless interfaces 2107 and / or one or more wired interfaces 2108 that may be configured to communicate via a network. The wireless interface 2107 may include one or more wireless transmitters, receivers, and / or transceivers, such as Bluetooth. TM Transceiver, Transceiver, Wi-Fi TM Transceiver, WiMAX TM The wired interface 2108 may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or a similar transceiver, which may be configured to communicate with a wired network via a twisted pair, a coaxial cable, a fiber optic link, or a similar physical connection.

[0094] In some examples, the network communication module 2102 can be configured to provide reliable, secure and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., guaranteed message delivery) can be provided, possibly as part of a message header and / or tail (e.g., packet / message sequencing information, encapsulation header and / or tail, size / time information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity check values). One or more encryption protocols and / or algorithms can be used to make communications secure (e.g., encoded or encrypted) and / or decrypted / decoded, the encryption protocols and / or algorithms, such as, but not limited to, data encryption standard (DES), advanced encryption standard (AES), Rivest-Shamir-Adelman (RSA) algorithm, Diffie-Hellman algorithm, secure socket protocols such as secure socket layer (SSL) or transport layer security (TLS) and / or digital signature algorithm (DSA). Other encryption protocols and / or algorithms may also be used, or in addition to those listed here, to protect (and then decrypt / decode) communications.

[0095] The one or more processors 2103 may include one or more general purpose processors, and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). The one or more processors 2103 may be configured to execute computer readable instructions 2106 contained in the data store 2104 and / or other instructions described herein.

[0096] Data storage 2104 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 2103. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage, which may be integrated in whole or in part with at least one of the one or more processors 2103. In some examples, data storage 2104 may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage unit), while in other examples, data storage 2104 may be implemented using two or more physical devices.

[0097] Data store 2104 may include computer-readable instructions 2106 and possibly additional data. In some examples, data store 2104 may include storage required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. In some examples, data store 2104 may include storage of trained neural network models 2112 (e.g., models of trained convolutional neural networks such as convolutional neural network 430). In certain examples of these examples, computer-readable instructions 2106 may include instructions that, when executed by processor 2103, enable computing device 2100 to provide some or all of the functionality of trained neural network model 2112.

[0098] In some examples, computing device 2100 may include one or more cameras 2118. Cameras 2118 may include one or more image capture devices, such as still and / or video cameras, that are equipped to capture light and record the captured light in one or more images; that is, camera 2118 may generate an image of the captured light. The one or more images may be one or more still images and / or one or more images used in a video image. Camera 2118 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or one or more other frequencies of light.

[0099] In some examples, computing device 2100 may include one or more sensors 2120. Sensors 2120 may be configured to measure conditions within computing device 2100 and / or conditions in the environment of computing device 2100 and provide data about these conditions. For example, sensors 2120 may include one or more of the following: (i) sensors for obtaining data about computing device 2100, such as, but not limited to, a thermometer for measuring the temperature of computing device 2100, a battery sensor for measuring the power of one or more batteries of power system 2122, and / or other sensors for measuring conditions of computing device 2100; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, radio frequency identification (RFID) readers, proximity sensors, one-dimensional barcode readers, two-dimensional barcode (e.g., quick response (QR) code) readers, and laser trackers, wherein the identification sensors may be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or objects that are configured to be read and provide at least identification information; (iii) sensors for measuring computing Sensors for the position and / or movement of the device 2100, such as, but not limited to, tilt sensors, gyroscopes, accelerometers, Doppler sensors, GPS devices, sonar sensors, radar devices, laser displacement sensors, and compasses; (iv) environmental sensors for obtaining data indicative of the environment of the computing device 2100, such as, but not limited to, infrared sensors, optical sensors, light sensors, biosensors, capacitive sensors, touch sensors, temperature sensors, wireless sensors, radio sensors, motion sensors, microphones, sound sensors, ultrasonic sensors, and / or smoke sensors; and / or (v) force sensors for measuring one or more forces (e.g., inertial forces and / or gravity) acting on the computing device 2100, such as, but not limited to, one or more sensors that measure: force in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor that identifies the location of the ZMP. Many other examples of sensors 2120 are also possible.

[0100] The power system 2122 may include one or more batteries 2124 and / or one or more external power interfaces 2126 for providing power to the computing device 2100. When electrically coupled to the computing device 2100, each battery of the one or more batteries 2124 may act as a source of stored power for the computing device 2100. The one or more batteries 2124 of the power system 2122 may be configured to be portable. Some or all of the one or more batteries 2124 may be easily removed from the computing device 2100. In other examples, some or all of the one or more batteries 2124 may be inside the computing device 2100 and may therefore not be easily removed from the computing device 2100. Some or all of the one or more batteries 2124 may be rechargeable. For example, a rechargeable battery may be recharged by a wired connection between the battery and another power source, such as by one or more power sources external to the computing device 2100 and connected to the computing device 2100 via one or more external power interfaces. In other examples, some or all of the one or more batteries 2124 may be non-rechargeable batteries.

[0101] The one or more external power interfaces 2126 of the power system 2122 may include one or more wired power interfaces, such as a USB cable and / or a power cord, which enables a wired power connection to one or more power sources external to the computing device 2100. The one or more external power interfaces 2126 may include one or more wireless power interfaces, such as a Qi wireless charger, which enables wireless power connection to one or more external power sources, for example, via a Qi wireless charger. Once a power connection to an external power source is established using the one or more external power interfaces 2126, the computing device 2100 can draw power from the external power source to the established power connection. In some examples, the power system 2122 may include associated sensors, such as battery sensors or other types of power sensors associated with one or more batteries.

[0102] Cloud-based servers

[0103] Fig. 22 A network 2006 of computing clusters 2209a, 2209b, 2209c arranged as a cloud-based server system according to an example embodiment is depicted. The computing clusters 2209a, 2209b, 2209c may be cloud-based devices storing program logic and / or data for cloud-based applications and / or services; for example, performing at least one of the functions of convolutional neural networks, confidence learning, predicting target images, predicting original lighting models, convolutional neural networks 430, confidence learning 630, and / or method 2300 and / or functions related thereto.

[0104] In some embodiments, computing clusters 2209a, 2209b, 2209c may be a single computing device residing in a single computing center. In other embodiments, computing clusters 2209a, 2209b, 2209c may include multiple computing devices in a single computing center, or even multiple computing devices in multiple computing centers located in different (diverse) geographical locations. For example, Fig. 22 Each of the computing clusters 2209a, 2209b, and 2209c are depicted as residing in a different physical location.

[0105] In some embodiments, the data and services of the computing clusters 2209a, 2209b, 2209c may be encoded as computer-readable information stored in a non-transitory tangible computer-readable medium (or computer-readable storage medium) and accessible by other computing devices. In some embodiments, the computing clusters 2209a, 2209b, 2209c may be stored on a single disk drive or other tangible storage medium, or may be implemented on multiple disk drives or other tangible storage media located in one or more different geographic locations.

[0106] Fig. 22 A cloud-based server system is depicted according to an example embodiment. Fig. 22 In the embodiment of the present invention, the functions of the convolutional neural network, confidence learning and / or computing devices can be distributed in computing clusters 2209a, 2209b, 2209c. The computing cluster 2209a can include one or more computing devices 2200a, a cluster storage array 2210a, and a cluster router 2211a connected by a local cluster network 2212a. Similarly, the computing cluster 2209b can include one or more computing devices 2200b, a cluster storage array 2210b, and a cluster router 2211b connected by a local cluster network 2212b. Similarly, the computing cluster 2209c can include one or more computing devices 2200c, a cluster storage array 2210c, and a cluster router 2211c connected by a local cluster network 2212c.

[0107] In some embodiments, each of computing clusters 2209a, 2209b, and 2209c may have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each computing cluster may have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster may depend on one or more computing tasks assigned to each computing cluster.

[0108] For example, in computing cluster 2209a, computing device 2200a may be configured to perform various computing tasks of convolutional neural networks, confidence learning, and / or computing devices. In one embodiment, various functions of convolutional neural networks, confidence learning, and / or computing devices may be distributed among one or more computing devices 2200a, 2200b, and 2200c. The computing devices 2200b and 2200c in the corresponding computing clusters 2209b and 2209c may be configured similarly to the computing device 2200a in the computing cluster 2209a. On the other hand, in some embodiments, computing devices 2200a, 2200b, and 2200c may be configured to perform different functions.

[0109] In some embodiments, computing tasks and stored data associated with convolutional neural networks, confidence learning, and / or computing devices may be distributed across computing devices 2200a, 2200b, and 2200c based at least in part on the processing requirements of the convolutional neural networks, confidence learning, and / or computing devices, the processing capabilities of computing devices 2200a, 2200b, 2200c, the latency of network links between computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that contribute to cost, speed, fault tolerance, resilience, efficiency, and / or other design goals of the overall system architecture.

[0110] The cluster storage arrays 2210a, 2210b, 2210c of the computing clusters 2209a, 2209b, 2209c may be data storage arrays that include disk array controllers configured to manage read and write access to the hard disk drive groups. The disk array controllers, alone or in combination with their respective computing devices, may also be configured to manage backup or redundant copies of data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0111] Similar to the way that the functionality of the convolutional neural network, confidence learning, and / or computing devices can be distributed across the computing devices 2200a, 2200b, 2200c of the computing clusters 2209a, 2209b, 2209c, various active and / or backup portions of these components can be distributed across the cluster storage arrays 2210a, 2210b, 2210c. For example, some cluster storage arrays can be configured to store a portion of the data of the convolutional neural network, confidence learning, and / or computing devices, while other cluster storage arrays can store other portions of the data of the convolutional neural network, confidence learning, and / or computing devices. In addition, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.

[0112] The cluster routers 2211a, 2211b, 2211c in the computing clusters 2209a, 2209b, 2209c may include networking devices configured to provide internal and external communications to the computing clusters. For example, the cluster router 2211a in the computing cluster 2209a may include one or more Internet switching and routing devices configured to provide (i) local area network communications between the computing device 2200a and the cluster storage array 2210a via a local cluster network 2212a, and (ii) wide area network communications between the computing cluster 2209a and the computing clusters 2209b and 2209c via a wide area network link 2213a to the network 2006. Cluster routers 2211b and 2211c may include network devices similar to cluster router 2211a, and cluster routers 2211b and 2211c may perform networking functions for computing clusters 2209b and 2209b similar to what cluster router 2211a performs for computing cluster 2209a.

[0113] In some embodiments, the configuration of the cluster routers 2211a, 2211b, 2211c can be based at least in part on the data communication requirements of the computing devices and the cluster storage array, the data communication capabilities of the network devices in the cluster routers 2211a, 2211b, 2211c, the latency and throughput of the local cluster network 2212a, 2212b, 2212c, the latency, throughput and cost of the wide area network links 2213a, 2213b, 2213c, and / or other factors that can help adjust the cost, speed, fault tolerance, resilience, efficiency and / or other design criteria of the system architecture.

[0114] Example methods of operation

[0115] Fig.232300 according to an example embodiment. The method 2300 may be performed by a computing device such as the computing device 2100. The method 2300 may begin at block 2310, where the computing device may train a neural network to apply an illumination model to the input image using confidence learning based on light predictions and prediction confidence values ​​associated with the illumination of the input image, as described above at least in Figure 2-Figure 6 discussed in the context of .

[0116] In some examples, training a neural network using confidence learning to apply a lighting model to an input image may include training a convolutional neural network using confidence learning to apply a lighting model to an input image, such as at least Figure 5 and Figure 6 In some of these examples, the illumination of the input image can be modeled using the original light model. In such examples, training the convolutional neural network with confidence learning to apply the illumination model to the input image can include training the convolutional neural network with confidence learning based on light predictions and prediction confidence values ​​associated with the original light model, as described above at least in Figure 5 and Figure 6 In some of these examples, training the convolutional neural network using confidence learning based on light predictions and prediction confidence values ​​associated with the original light model includes training the convolutional neural network using confidence learning based on light predictions of the original light model for a portion of the input image and prediction confidences of the illumination predictions for the portion of the input image, as described above at least in Figure 6 discussed in the context of .

[0117] In some examples, training the neural network to apply the illumination model to the input image may include training the neural network using multiple images of the object, wherein the multiple images illuminate the object using multiple illumination models, as described above at least in Figure 2-Figure 5 In some examples, training the neural network may include training the neural network at a computing device, as described above at least in Fig.19 discussed in the context of .

[0118] At block 2320, the computing device may receive an input image of an object and data regarding a particular lighting model to be applied to the input image, as described above at least in Figure 4-Figure 18 discussed in the context of .

[0119] In some examples, the illumination of the input image can be modeled using the original light model. In such examples, determining the output image can further include using a trained neural network to determine the output image and a prediction of the original light model, as described above at least in Figure 4-Figure 18 discussed in the context of .

[0120] In some examples, the object may include an object that diffusely reflects light, such as at least Figure 4-Figure 18 In some examples, the object may include a human face, such as at least Figure 4-Figure 18 discussed in the context of .

[0121] In some examples, the computing device may include a camera. In such examples, receiving the input image of the object may include generating the input image of the object using the camera, and receiving the generated input image from the camera at the computing device, as described above at least in Figure 2 and Figure 3 discussed in the context of .

[0122] In some examples, the input image of the object may be a single image of the object, as described above at least in Figure 4-Figure 18 discussed in the context of .

[0123] At block 2330, the computing device may determine an output image of the object by applying a particular lighting model to an input image of the object using a trained neural network, as described above at least in Figure 4-Figure 18 discussed in the context of .

[0124] In some examples, receiving an input image of the object and data about a specific illumination model to be applied to the input image may include receiving the input image of the object and data about a plurality of specific illumination models to be applied to the input image, and determining the output image may include determining a plurality of output images by applying each of the plurality of specific illumination models to the input image, such as at least Figure 2 and Figure 3 discussed in the context of .

[0125] In some examples, determining an output image of the object by using a trained neural network may include obtaining the trained neural network at a computing device; and determining, by the computing device, an output image of the object using the obtained neural network, as described above at least in Fig.19 discussed in the context of .

[0126] In some examples, method 2300 may further include using a computing device to provide an output image, as described above at least in Figure 4-Figure 18 discussed in the context of .

[0127] In some examples, the illumination of the input image can be modeled using an original light model. In such examples, method 2300 can further include providing, using a computing device, a prediction of the original light model, as described above at least in Figure 4-Figure 18 discussed in the context of .

[0128] In some examples, determining an output image of an object by using a trained neural network may include: determining, by a computing device, a request to apply a particular lighting model to an input image; sending, from the computing device to a second computing device, a request to apply the particular lighting model to the input image, the second computing device including the trained neural network; and after sending the request, receiving, from the second computing device, an output image of the object with the particular lighting model applied to the input image, such as at least Fig.19 discussed in the context of .

[0129] Additional Example Embodiments

[0130] The following clauses are provided as further description of the present disclosure.

[0131] Item 1 - A computer-implemented method comprising: training a neural network to apply an illumination model to an input image using confidence learning based on light predictions and prediction confidence values ​​associated with illumination of the input image; receiving, at a computing device, an input image of an object and data regarding a particular illumination model to be applied to the input image; and determining, by the computing device, an output image of the object by applying the particular illumination model to the input image of the object using the trained neural network.

[0132] Clause 2.—The computer-implemented method of clause 1, wherein the illumination of the input image is modeled using an original light model, and wherein determining the output image further comprises determining the output image and a prediction of the original light model using a trained neural network.

[0133] Paragraph 3.—A computer-implemented method as described in paragraph 1 or 2, wherein training a neural network using confidence learning to apply the illumination model to the input image comprises training a convolutional neural network using confidence learning to apply the illumination model to the input image.

[0134] Clause 4.—A computer-implemented method as described in Clause 3, wherein the illumination of the input image is modeled using an original light model, and wherein training the convolutional neural network using confidence learning to apply the illumination model to the input image comprises training the convolutional neural network using confidence learning based on light predictions and prediction confidence values ​​associated with the original light model.

[0135] Clause 5.—A computer-implemented method as described in Clause 4, wherein training the convolutional neural network using confidence learning based on light predictions and prediction confidence values ​​associated with the original light model comprises training the convolutional neural network using confidence learning based on light predictions of the original light model for a portion of an input image and prediction confidences of the illumination predictions for the portion of the input image.

[0136] Clause 6. A computer-implemented method according to any of Clauses 1-5, wherein training the neural network to apply the illumination model to the input image comprises training the neural network using a plurality of images of the object, wherein the plurality of images utilize a plurality of illumination models to illuminate the object.

[0137] Clause 7—A computer-implemented method as described in any of Clauses 1-6, wherein the object comprises an object that diffusely reflects light.

[0138] Clause 8—A computer-implemented method as described in any of Clauses 1-7, wherein the object comprises a human face.

[0139] Clause 9 —A computer-implemented method according to any of Clauses 1-8, wherein the computing device includes a camera, and wherein receiving an input image of the object comprises: generating an input image of the object using the camera; and receiving the generated input image from the camera at the computing device.

[0140] Clause 10—A computer-implemented method according to any of Clauses 1-9, further comprising: providing an output image using a computing device.

[0141] Clause 11 —A computer-implemented method according to any of Clauses 1-10, wherein the illumination of the input image is modeled using a raw light model, and wherein the method further comprises: providing a prediction of the raw light model using a computing device.

[0142] Clause 12. A computer-implemented method according to any of Clauses 1 to 11, wherein receiving an input image of the object and data regarding a specific illumination model to be applied to the input image comprises receiving an input image of the object and data regarding a plurality of specific illumination models to be applied to the input image, and wherein determining the output image comprises determining a plurality of output images by applying each of the plurality of specific illumination models to the input image.

[0143] Clause 13—A computer-implemented method as described in any of Clauses 1-12, wherein the input image of the object is a single image of the object.

[0144] Clause 14.—A computer-implemented method as described in any of Clauses 1-13, wherein determining the output image of the object by using the trained neural network comprises: obtaining the trained neural network at a computing device; and determining the output image of the object using the obtained neural network by the computing device.

[0145] Clause 15.—The computer-implemented method of clause 14, wherein training the neural network comprises training the neural network at a computing device.

[0146] Clause 16. A computer-implemented method according to any of Clauses 1 to 15, wherein determining an output image of the object using a trained neural network comprises: determining, by a computing device, a request to apply a particular lighting model to an input image; sending, from the computing device to a second computing device, a request to apply the particular lighting model to the input image, the second computing device comprising the trained neural network; and after sending the request, receiving, by the computing device from the second computing device, an output image in which the particular lighting model is applied to the input image of the object.

[0147] Item 17 - A computing device comprising: one or more processors; and a data storage device, wherein the data storage device has computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the computing device to perform functions including the computer-implemented method of any one of Items 1-16.

[0148] Item 18 - An article of manufacture comprising one or more computer-readable media having computer-readable instructions stored thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform functions including the computer-implemented method of any of Items 1-16.

[0149] Clause 19.—The article of manufacture of clause 18, wherein the one or more computer-readable media comprises one or more non-transitory computer-readable media.

[0150] Clause 20—A computing device comprising: means for performing the computer-implemented method of any of Clauses 1-16.

[0151] The present disclosure is not limited to the specific embodiments described in the present application, which are intended to be illustrations of various aspects. It is clear to those skilled in the art that many modifications and variations may be made without departing from its spirit and scope. In addition to those listed here, the functionally equivalent methods and devices within the scope of the present disclosure will be clear to those skilled in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0152] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the drawings, similar symbols generally identify similar components unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that, as generally described herein and shown in the accompanying drawings, the various aspects of the present disclosure may be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0153] With respect to any or all ladder diagrams, scenarios, and flow charts in the accompanying drawings and as discussed herein, each block and / or communication may represent the processing of information and / or the transmission of information, according to example embodiments. Alternative embodiments are also included within the scope of these example embodiments. In these alternative embodiments, for example, depending on the functionality involved, the functions described as blocks, transmissions, communications, requests, responses, and / or messages may not be performed in the order shown or discussed, including substantially concurrently or in reverse order. In addition, more or fewer blocks and / or functions may be used with any ladder diagrams, scenarios, and flow charts discussed herein. Figure 1 and these ladder diagrams, scenarios, and flow charts can be combined with each other in part or in whole.

[0154] The blocks representing information processing may correspond to circuits that may be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, the blocks representing information processing may correspond to a module, a segment, or a portion of a program code (including associated data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in a method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a disk or hard drive or other storage medium.

[0155] Computer readable media may also include non-transitory computer readable media, such as non-transitory computer readable media that store data for a short period of time, such as register memory, processor cache, and random access memory (RAM). Computer readable media may also include non-transitory computer readable media that store program code and / or data for a longer period of time, such as auxiliary or persistent long-term storage, such as read-only memory (ROM), optical or magnetic disks, compact disk read-only memory (CD-ROM). Computer readable media may also be any other volatile or non-volatile storage system. Computer readable media may be considered, for example, as a computer readable storage medium or a tangible storage device.

[0156] In addition, the blocks representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device. However, other information transfers may be performed between software modules and / or hardware modules in different physical devices.

[0157] Although various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the appended claims.

Claims

1. A computer-implemented method comprising: training a neural network to apply an illumination model to the input image using confidence learning based on light predictions and prediction confidence values ​​associated with illumination of the input image, wherein the light predictions include predictions of the original light model indicating positions of one or more ambient light sources relative to objects in the input image; receiving, at a computing device, an input image of a subject and data regarding a particular lighting model to be applied to the input image; and determining, by a computing device, an output image of the object by applying a particular illumination model to an input image of the object using a trained neural network, wherein the confidence value is based on weights of a trained neural network, and the confidence learning comprises combining the light prediction with the prediction confidence value to determine an updated light prediction.

2. The computer-implemented method of claim 1 , wherein: The illumination of the input image is modeled using the original light model, and wherein determining the output image further comprises determining the output image and a prediction of the original light model using the trained neural network.

3. A computer-implemented method according to claim 1 or claim 2, wherein: Training a neural network using confidence learning to apply an illumination model to an input image includes training a convolutional neural network using confidence learning to apply an illumination model to an input image.

4. The computer-implemented method of claim 3, wherein: The illumination of the input image is modeled using the original light model, and wherein training the convolutional neural network using confidence learning to apply the illumination model to the input image includes training the convolutional neural network using confidence learning based on light predictions and prediction confidence values ​​associated with the original light model.

5. The computer-implemented method of claim 4, wherein: Training the convolutional neural network using confidence learning based on light predictions and prediction confidence values ​​associated with the original light model includes training the convolutional neural network using confidence learning based on light predictions of the original light model for a portion of an input image and prediction confidences of illumination predictions for the portion of the input image.

6. The computer-implemented method of any one of claims 1, 2, 4, and 5, wherein: Training the neural network to apply the illumination model to the input image includes training the neural network using a plurality of images of the object, wherein the plurality of images illuminate the object using a plurality of illumination models.

7. The computer-implemented method of any one of claims 1, 2, 4, and 5, wherein: The objects include objects that diffusely reflect light.

8. The computer-implemented method of any one of claims 1, 2, 4, and 5, wherein: The object includes a human face.

9. The computer-implemented method of any one of claims 1, 2, 4, and 5, wherein: The computing device includes a camera, and wherein receiving an input image of an object includes: generating an input image of the object using a camera; and A generated input image is received at a computing device from a camera.

10. The computer-implemented method of any one of claims 1, 2, 4, and 5, further comprising: An output image is provided using a computing device.

11. The computer-implemented method of any one of claims 1, 2, 4, and 5, wherein: The illumination of the input image is modeled using a primitive light model, and wherein the method further comprises: A computing device is used to provide predictions of the original light model.

12. The computer-implemented method of any one of claims 1, 2, 4, and 5, wherein: Receiving an input image of the object and data about a specific illumination model to be applied to the input image includes receiving an input image of the object and data about multiple specific illumination models to be applied to the input image, and wherein determining the output image includes determining multiple output images by applying each of the multiple specific illumination models to the input image.

13. A computer-implemented method according to any one of claims 1, 2, 4, and 5, wherein: The input image of the object is a single image of the object.

14. The computer-implemented method of any one of claims 1, 2, 4, and 5, wherein: The output image of the object determined by using the trained neural network includes: obtaining, at a computing device, a trained neural network; and The obtained neural network is used by a computing device to determine an output image of the object.

15. The computer-implemented method of claim 14, wherein: Training the neural network includes training the neural network at a computing device.

16. A computer-implemented method according to any one of claims 1, 2, 4, and 5, wherein: The output image of the object determined by using the trained neural network includes: determining, by a computing device, a request to apply a particular illumination model to an input image; sending, from the computing device to a second computing device, a request to apply a particular illumination model to an input image, the second computing device comprising the trained neural network; and After sending the request, the computing device receives from the second computing device an output image that applies the particular lighting model to the input image of the object.

17. A computing device comprising: one or more processors; as well as A data storage device, wherein the data storage device has computer executable instructions stored thereon, and when the computer executable instructions are executed by the one or more processors, the computing device performs the functions of the computer-implemented method described in any one of claims 1, 2, 4, and 5.

18. An article of manufacture related to photo relighting using deep neural networks and confidence learning, comprising one or more computer-readable media having computer-readable instructions stored thereon, which, when executed by one or more processors of a computing device, causes the computing device to perform functions including the computer-implemented method of any one of claims 1, 2, 4, and 5.

19. The article of claim 18, wherein: The one or more computer-readable media include one or more non-transitory computer-readable media.

20. A computing device comprising: An apparatus for performing the computer-implemented method of any one of claims 1, 2, 4, and 5.

Images