Image light redistribution based on machine learning models
Patent Information
- Application Number
- JP2024523737
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-10-22
- Publication Date
- 2024-11-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Mobile devices lack computational resources to perform high-quality image enhancement features such as reducing specular highlights and redistributing light to shadow areas, which are typically available only on more powerful computing devices.
A computing device configured with a neural network that redistributes light energy per pixel without predicting albedo, using a pixel-aligned lighting representation and U-net architecture to adjust specular and diffuse components, enabling light diffusion techniques similar to a real-world scrim, suitable for mobile devices.
Enables high-quality image enhancement on mobile devices by reducing specular highlights and softening shadows without requiring additional computational resources, allowing users to create attractive portraits and images in real-time.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Background technology]
[0001] background Many modern computing devices, including mobile phones, personal computers, and tablets, include image capture devices, such as still and / or video cameras. The image capture devices may capture images, such as images that include people, animals, landscapes, and / or objects.
[0002] Some image capture devices and / or computing devices may enhance or otherwise modify a captured image. For example, some image capture devices may provide "red-eye" correction to remove artifacts such as red-appearing eyes of humans and animals that may be present in images captured using bright lights such as flash lighting. Additionally, for example, some image capture devices may provide image editing features such as relighting to modify the lighting in an image. After a captured image has been enhanced and / or edited, the enhanced image may be saved, displayed, transmitted, printed on paper, and / or otherwise utilized.
[0003] Portraits taken in casual photography settings may have undesirable environmental lighting conditions that result in undesirable portrait appearances in the captured photos. As an illustration, portraits under strong directional lighting or with objects that occlude the light source of the scene may have undesirable shadows that may detract from the overall appearance of the portrait. In another example, a portrait may exhibit strong specular highlights, which are bright spots that may generally be white spots that appear due to very bright point or spot light sources in the scene. For example, a spotlight lamp indoors, or the sun for outdoor photography, may be a very bright point or spot light source. In some aspects, photographers may be able to utilize tools such as "scrims" that can diffuse the scene lighting and reduce the harshness of the lighting on the subject. However, such features are not available on mobile devices.
[0004] Mobile phone applications are limited in the amount of computational resources available, so allowing users to take advantage of realistic, professional, and high-quality image enhancement features while utilizing low amounts of computational resources is an important technological improvement. Summary of the Invention
[0005] overview In one aspect, a computing device, such as an image capture device, may be configured to perform image adjustment tasks, such as reducing specular highlights and / or redistributing light to shadow regions of an image. Driven by a system of machine-learned components, the computing device may be configured to enable a user to create attractive portraits or other types of images. For example, the computing device may be configured to soften the appearance of unsightly shadows in a scene. Furthermore, for example, the computing device may be configured to soften or completely remove bright specular highlights using machine learning techniques. In general, to perform such functions, a neural network may have to predict the albedo, which requires additional computational resources. However, predicting the albedo may require additional computational resources. Furthermore, for example, an albedo predictor may change the color characteristics of the image, which may not be ideal. Thus, the computing device may be configured to reduce specular highlights and / or redistribute light to shadow regions of an image without the need to predict the albedo for the image. This facilitates the technique to be performed on a mobile device by utilizing a low amount of computational resources.
[0006] In some aspects, light diffusion techniques are described that may be a digital analogue to a real-world photography tool called a "scrim," the purpose of which is to diffuse scene lighting and reduce the harshness of the illumination falling on a subject. Further, for example, the computing device may be configured to allow a user to select the amount of light redistribution. In some aspects, the computing device may be configured to make inferences regarding the color composition underlying specular highlights in an image.
[0007] To perform such functions, the computing device may be configured to utilize one or more intermediate outputs of the machine learning model for model training supervision. A "pixel-aligned" lighting representation may be utilized that takes the target lighting environment and maps it directly into the coordinate space of the target image. Such a lighting representation may be based on computer graphics rendering techniques, such as, for example, a diffuse convolution operation and one or more specular convolution operations. A lighting representation as described herein is suitable for a U-net type neural network architecture, where the spatial alignment of the inputs is of high importance. In some embodiments, the U-net architecture may be configured to preserve the high frequency aspects of the input image. One of the advantages of this lighting representation is that there is no longer a need to learn a complex mapping from a representation of the lighting environment coordinates to a representation of the image coordinates. Another advantage of the lighting representation is that it allows the light redistribution model to generate complex non-Lambertian reflectance while correctly inferring the lower frequency colors and shades under the target illumination.
[0008] As described herein, machine learning models can be trained on relit images of a subject captured in a light stage computational lighting system that can record multiple lighting conditions, high quality geometry, and / or accurate alpha mattes. Per-pixel lighting representations in a deep learning framework can be utilized to model the diffuse and specular components of a subject's appearance.
[0009] In some aspects, a mobile device may be configured such that images may be enhanced in real time with these features. Furthermore, for example, the described methods may be applied to images in real environments, including images in video content. In some instances, images may be automatically enhanced by a mobile device. In other aspects, mobile phone users may non-destructively enhance images to match their user preferences. Furthermore, for example, pre-existing images in a user's image library may be enhanced based on the techniques described herein.
[0010] In one embodiment, a computer-implemented method is provided. The method includes a computing device receiving an input image including an object. The method further includes a neural network adjusting one or more of a specular component or a diffuse component associated with the input image. The adjusting includes redistributing light energy per pixel of the input image. The method additionally includes the neural network predicting an output image including the object with the adjusted one or more of the specular component or the diffuse component.
[0011] In another aspect, a computing device is provided. The computing device includes one or more processors and a data storage. The data storage stores computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform a function. The function includes the computing device receiving an input image including an object. The function further includes a neural network adjusting one or more of a specular component or a diffuse component associated with the input image. The adjusting includes redistributing light energy per pixel of the input image. The function further includes the neural network predicting an output image including the object with the adjusted one or more of the specular component or the diffuse component.
[0012] In another aspect, an article of manufacture is provided. The article of manufacture includes one or more computer-readable media storing computer-readable instructions that, when executed by one or more processors of the computing device, cause the computing device to perform a function. The function includes the computing device receiving an input image including an object. The function further includes the neural network adjusting one or more of the specular or diffuse components associated with the input image. The adjusting includes redistributing light energy per pixel of the input image. The function further includes the neural network predicting an output image including the object with the adjusted one or more of the specular or diffuse components.
[0013] In another aspect, a system is provided, the system including: means for a computing device to receive an input image including an object; and means for a neural network to adjust one or more of a specular component or a diffuse component associated with the input image, the adjusting including redistributing light energy per pixel of the input image, the system further including means for the neural network to predict an output image including the object with the adjusted one or more of the specular component or the diffuse component.
[0014] The foregoing summary is merely illustrative and is not intended to be in any way limiting. In addition to the exemplary aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description and accompanying drawings. [Brief description of the drawings]
[0015] [Figure 1] 1 illustrates an example inference phase of a neural network for adjusting one or more of the specular or diffuse components associated with an input image according to an example embodiment. [Diagram 2]1 illustrates an example training phase of a neural network for adjusting one or more of the specular or diffuse components associated with an input image according to an example embodiment. [Diagram 3] 1 illustrates linear interpolation of input and output images according to an example embodiment; [Figure 4] 1 is a diagram illustrating the training and inference phases of a machine learning model according to an example embodiment. [Diagram 5] FIG. 1 depicts a distributed computing architecture according to an example embodiment. [Figure 6] 1 is a block diagram of a computing device according to an example embodiment. [Figure 7] FIG. 1 depicts a network of computing clusters arranged as a cloud-based server system according to an example embodiment. [Figure 8] 4 is a flowchart of a method according to an example embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Detailed Description Overview The present application relates to enhancing images of objects, such as objects depicting human faces, by redistribution of light energy using machine learning techniques, such as, but not limited to, neural network techniques. The present application further relates to reducing specular highlights and / or softening shadows. When a mobile computing device user takes an image of an object, such as a person, the resulting image may not always have ideal specular or shadow components. For example, the image may include specular highlights or enhanced shadows. Furthermore, even if the image does have the desired specular and / or shadow components at one time, the user may want to modify these components at a later time. As such, technical problems related to image processing arise, including redistribution of light energy of an already acquired image.
[0017] To enable user control of light energy redistribution, such as in an image of a human face, the techniques described herein apply a model based on a convolutional neural network to redistribute light energy. The techniques described herein include receiving an input image and adjusting one or more of a specular or diffuse component associated with the input image, where adjusting includes redistributing light energy per pixel of the input image, and the techniques further include predicting an output image including the subject with the adjusted one or more of the specular or diffuse components. The input and output images may be high-resolution images, such as multi-megapixel size images captured by a camera of a mobile computing device. Convolutional neural networks may work well with input images captured under a variety of natural and artificial lighting conditions. In some examples, trained models of convolutional neural networks may work on a variety of computing devices, including, but not limited to, mobile computing devices (e.g., smartphones, tablet computers, cell phones, laptop computers), stationary computing devices (e.g., desktop computers), and server computing devices.
[0018] A neural network, such as a convolutional neural network, may be trained using training data including pairs of images to perform one or more aspects as described herein. In some examples, the neural network may be configured as an encoder / decoder neural network.
[0019] Human faces typically reflect light diffusely, but may also include some specular highlights due to directly reflected light. For example, specular highlights may result from direct light reflection from eye surfaces, eyeglasses, jewelry, etc. In many images of human faces, such specular highlights are relatively small in area compared to the area of the face surface that diffusely reflects light. Thus, a neural network can be trained to apply light redistribution to images of objects that reflect light diffusely and that include specular highlights. Images in the training dataset may show one or more particular objects using lighting provided under multiple different conditions, such as lighting provided from different directions, lighting provided of varying intensities (e.g., brighter and dimmer lighting), lighting provided by light sources of different colors, lighting provided by a different number of light sources, etc.
[0020] The trained neural network may process an input image to predict an output image including an object with one or more of the specular or diffuse components adjusted. In one example, (a copy of) the trained neural network may reside on a mobile computing device. The mobile computing device may include a camera that may capture an input image of an object, such as an image of a person's face. A user of the mobile computing device may review the input image and determine that the input image should be adjusted. In response, the trained neural network may generate a predicted output image indicative of an adjusted output image based on the techniques described herein, and subsequently output the output image (e.g., provide the output image for display by the mobile computing device). In another example, a user of the mobile computing device may review the input image and determine the amount of light redistribution to be applied. In response, the trained neural network may generate a predicted output image indicative of the input image with the indicated amount of redistribution.
[0021] In other examples, the trained neural network is not resident on the mobile computing device; rather, the mobile computing device provides input images to a remotely located trained neural network (e.g., over the Internet or another data network). The remotely located convolutional neural network may process the input images and provide an output image indicative of the redistribution of light in the input images to the mobile computing device. In other examples, non-mobile computing devices may use the trained neural network to redistribute light, even including images not captured by the computing device's camera.
[0022] As such, the techniques described herein may improve images by applying a more desirable and / or selectable redistribution of light to images, thereby enhancing the actual and / or perceived quality of those images. Improving the actual and / or perceived quality of images, including human portrait images, may provide emotional benefits to people who believe their pictures look better. These techniques are flexible and therefore may apply a wide variety of lighting representations of human faces and other objects, particularly other objects with similar lighting characteristics.
[0023] Techniques for light redistribution using neural networks. Light redistribution networks can be designed to computationally redistribute light for consumer photography or other applications. As described herein, these methods are applicable to any omnidirectional input and targeted lighting environments. Moreover, in addition to achieving realistic results for, for example, low frequency lighting, the network can also adjust shadows and specular highlights appropriately for lighting related to high frequency detail.
[0024] FIG. 1 illustrates an example inference phase of a neural network 100 for adjusting one or more of a specular or diffuse component associated with an input image according to an example embodiment. In some embodiments, the method includes a computing device receiving an input image 105 including a subject 105A. A light redistribution network 140 adjusts one or more of a specular or diffuse component associated with the input image 105, where adjusting includes redistributing light energy per pixel of the input image. In some aspects, the light redistribution network 140 calculates a per-pixel lighting representation or light map, such as a specular light map 135 that encodes a specular component and a diffuse light map 130 that encodes a diffuse component, of a surface reflection for a given omnidirectional high dynamic range (HDR) lighting environment 110. In some embodiments, adjusting one or more of a specular or diffuse component includes adjusting a specular component, and redistributing light energy per pixel includes reducing a specular highlight associated with the subject. In some embodiments, adjusting one or more of the specular or diffuse components includes adjusting the diffuse component, and redistributing the light energy per pixel includes reducing the light energy per pixel in shadow regions of the input image.
[0025] In some embodiments, the method includes a neural network predicting an output image 146 that includes the object with one or more of the specular or diffuse components adjusted.
[0026] Some embodiments include the neural network predicting a plurality of per-pixel surface normal representations. The surface normal is a representation of a surface normal vector stored as a red-green-blue (RGB) image, with the RGB components corresponding to the x, y, and z coordinates of the normal vector, respectively. Some embodiments include the neural network predicting a per-pixel albedo image based on the input image 105. The term "albedo" as used herein may generally refer to a representation of the proportion of incident light that is diffusely reflected by a surface without being absorbed by the surface. For example, an object perceived as being lighter in color reflects a greater proportion of incident light and therefore has a higher albedo. Similarly, an object perceived as being darker in color reflects a smaller proportion of incident light and therefore has a lower albedo.
[0027] Some embodiments include pre-filtering the target illumination. The pre-filtering may be based on a convolved specular light map and a convolved diffuse light map. The specular and diffuse components of the surface reflection may be determined by sampling the convolved specular light map and the convolved diffuse light map by using one or more of the surface normal or the reflection vector. For example, an HDR lighting environment such as an input HDR map 230 may be pre-filtered using a convolved light map 125 that includes a diffuse convolved light map 115 and one or more specular convolved light maps 120. In general, an object may have different light reflection characteristics that may depend, for example, on the object's surface geometry, color, and / or material. Furthermore, for example, the surface of an object may be composed of multiple materials, thus creating complex light reflection characteristics. The term "diffuse map" as used herein generally refers to an image of an object that represents the diffuse reflection of the object. Diffuse reflection is a type of surface reflectance where incident light is reflected and scattered in multiple directions (e.g., reflection from a rough surface). A diffuse map may be indexed by a set of color values indicating the texture (e.g., color and pattern) of an object. The term "specular map" as used herein generally refers to an image of an object that represents the specular reflection of that object. Specular reflection is a type of surface reflectance where incident light is reflected into unidirectional reflected light (e.g., reflection from a smooth and / or shiny surface). A specular map represents the glossiness characteristics of a surface and the highlight color of that surface.
[0028] The pre-filtered diffuse convolved light map 115 and one or more specular convolved light maps 110 may be sampled using the reflection vectors, which may produce a pixel-by-pixel representation of the diffuse and specular reflectance for the target illumination, represented as a diffuse light map 130 and one or more specular light maps 135.
[0029] For example, at training or inference time, after pre-computing a diffuse irradiance map, such as, for example, the diffuse convolved light map 115, and multiple pre-filtered HDR environment maps with different Phong exponents (e.g., n=1, 16, 32, 64), such as, for example, one or more specular light maps 120, diffuse and specular reflectance images or light maps, such as, for example, the diffuse light map 130 and one or more specular light maps 135, may be computed by indexing the diffuse light map 130 and one or more specular light maps 135 into the pre-filtered maps.
[0030] In some embodiments, the neural network may be configured to predict the specular light map 135. Such a neural network may model the uncertainty in the material properties of one or more materials of the input image 105. In some aspects, the one or more specular light maps 120 may be generated by multiple Phong indices n. In some embodiments, the neural network may be configured as a U-net architecture. For example, the neural network may be configured to have an architecture with 13 encoder-decoder layers with 8, 16, 32, 64, 128, 256 filters for the encoder, 256 for the bottleneck, and 256, 128, 64, 32, 16, 8 filters for the decoder, respectively. The neural network may be operated by a 3×3 convolution followed by a Leaky ReLU activation. The output of the neural network may be a four-channel per-pixel weight image w.
[0031]
number
[0032] A single specular map 135, represented as S n For example, for a pixel represented by pixel coordinates (u,v), the specular light map 135 may be determined as follows:
[0033]
number
[0034] where the summation is over multiple Phong indices n. A convolutional neural network, such as the light redistribution network 140 described herein, may be a fully convolutional neural network. During training, the convolutional neural network may receive one or more input training images as input. The convolutional neural network may include layers of nodes for processing the input images. Example layers may include, but are not limited to, an input layer, a convolutional layer, an activation layer, a pooling layer, and an output layer. The input layer may store input data, such as pixel data of the input image and inputs from other layers of the convolutional neural network. The convolutional layer may calculate the output of neurons connected to local regions at the input. In some examples, the predicted output may be fed back into the convolutional neural network as an input to perform iterative refinement. The activation layer may determine whether the output of the previous layer is "activated" or actually provided (e.g., provided to the next layer). The pooling layer may downsample the input. For example, a convolutional neural network may include one or more pooling layers to downsample an input by a predetermined factor (e.g., a factor of 2) in the horizontal and / or vertical dimensions. In some examples, some or all of the pooling layers in a convolutional neural network may downsample an input by a common factor in both the horizontal and vertical dimensions while not downsampling the depth dimension associated with the input. The depth dimension may store data about pixel colors (red, green, blue) and / or data representing scores. Other common factors for downsampling other than 2 may be used by one or more (pooling) layers of a convolutional neural network as well.
[0035] The output layer may provide the output of the convolutional neural network to software and / or hardware that interfaces with the convolutional neural network, such as hardware and / or software that is used to display, print, transmit, and / or otherwise provide the relighted image and / or the relighted image in a new background. Each layer of a convolutional neural network may include one or more input layers, output layers, convolutional layers, activation layers, pooling layers, and / or other layers described herein.
[0036] Loss Functions and Training As described herein, the ambient light in the portrait is diffused or blurred, which may result in the disappearance or reduction of the severity of the shadows and / or the disappearance or reduction of the severity of the specular highlights. In general, for images with high specular highlights, especially those that are already in a low dynamic range, the specular highlights may be clipped. For example, the image sensor may be saturated. Because the specular highlights have been clipped, the full dynamic range of the specular highlights may no longer be available for restoration. Thus, varying the exposure on the image, such as stepping down or darkening the image to remove the appearance of the specular highlights, may no longer be possible since the full amount of ambient light in the scene was not captured. Hence, the specular highlights cannot be adjusted by varying the exposure. Instead, this may be achieved by generating training pairs 210, as described herein.
[0037] FIG. 2 illustrates an example training phase of a neural network 200 for adjusting one or more of the specular or diffuse components associated with an input image 205 according to an example embodiment. In some embodiments, a light redistribution network 225 may receive individual input images 205 with individual objects and be trained to predict individual output images 230 including the objects with individual adjusted one or more of the specular or diffuse components. In some embodiments, a training dataset may include multiple image pairs 210, where a first image 215 of a given image pair includes an object in a lighting environment and a second image 220 of a given image pair includes an object in a diffused version of the lighting environment. In general, each training pair 210 includes two versions of an image in the same lighting environment, where the second image 220 is the same as the first image 210 in a blurred version of the environment.
[0038] For example, a variety of paired images 210 may be used to train a machine learning model using a teacher, where a first image 215 in the pair includes images of different objects lit in various lighting environments, and a second image 220 in the pair includes diffused versions of these images. In some implementations, to generate the first image 215, the reflectance field of the object may be photographed in a computer-controllable sphere of white LED light sources, for example, with the lights spaced 12° apart at the equator. In such implementations, the reflectance field is formed from a set of reflectance basis images photographing the object as each of the directional LED light sources is individually turned on, one at a time, in the spherical rig. Such One-Light-at-a-Time (OLAT) images are captured for multiple camera viewpoints. In some implementations, the light stage may have 331 programmable LED-based lights and 58 RGB cameras recording video at 60 Hz with 12.4 megapixel resolution, although these values and numbers of OLAT images, as well as the types of cameras used, may differ in some implementations.
[0039] In some implementations, reflectance fields for 70 diverse subjects may be used, each performing nine different facial expressions and wearing different clothing and accessories (e.g., hats, scarves, etc.), which produces approximately 700 sets of OLAT sequences from multiple camera viewpoints. In some implementations, approximately 10% of the OLAT sequences may be recorded with 58 cameras covering all 360° possible vantage points to provide training examples from arbitrary viewpoints covering the whole body. The remaining 90% of the OLAT sequences may be recorded with a subset of six frontal viewpoints intended to emulate the framing of casual photography. In one implementation, 7,560 unique sequences may be recorded for the 58 RGB cameras, which were then relit and composited using approximately 200 HDR panoramic lighting environments with random rotation, thus generating 8 million training examples. Other quantities of sets of OLAT sequences may be used. For evaluation purposes, the dataset may be split into a training and a testing set. In some instances, seven subjects with various skin tones may be selected for the testing set, along with ten lighting environments.
[0040] In some embodiments, the lighting environment may be a high dynamic range lighting environment, and training may include generating a specular convolution of a portion of the high dynamic range lighting environment. For example, a diffused version of an OLAT image, such as the first image 215, may be generated to generate the second image 220. In some embodiments, a diffuse convolution operation may be applied. For example, a blurred version of the environment convolved with Lambertian reflectance may be generated, and then the second image in the pair is the same subject in the blurred version of the environment. In some embodiments, generating the specular convolution may include applying a Phong Reflectance Model. In some embodiments, the second image 220 of the training pair 210 may be generated by diffusing only a portion of the environment, rather than the entire environment. This may allow for more targeted effects in specular falloff and / or shadow removal. For example, a diffused light environment corresponding to an HDR panorama is shown in image 210A. A diffuse or specular convolution of the input image 205, or a portion of the input image 205, may be generated for an HDR panorama of the input image 205. Image 210A illustrates a lighting panorama with different amounts of diffusion, for example based on a variant of the Phong reflectance model.
[0041] In some embodiments, training may include applying an adversarial loss function to a selected portion of the subject, such as the subject's face. In general, the adversarial loss for the face portion enables the light redistribution network 225 to learn to generate plausible, more photorealistic facial images.
[0042] Thus, the light redistribution network 225 may be trained to remove shadows and / or remove bright highlight areas. The input image 205 and the output image 230 have the same overall light energy, however, the light energy is redistributed over a larger area. Hence, strong shadows and specularity are softened. Such a technique is particularly advantageous for portraits, because unsightly shadows or specular highlights may cause undesirable effects in the image that a user may want to remove. In some embodiments, the light redistribution network 225 may maintain an average of the global color values associated with the input image 205. For example, the average overall color balance of the image may be preserved. This is particularly important because softening shadows and / or specularity may generally lead to an imbalance in the overall color attributes of the image.
[0043] In some embodiments, the light redistribution network 225 may predict one or more characteristics of the coloration associated with the specular component. For example, specular highlights generally do not contain information about the underlying diffuse colors. Thus, the light redistribution network 225 may learn to make inferences about the colors that lie beneath the specular highlights. Machine learning-based techniques enable such inferences to be made plausible.
[0044] In some embodiments, one of the evaluation objects may be evaluated when illuminated by one or more directional light sources, generating an HDR panoramic lighting environment to approximately match the positioning of the light in the light stage. This framework may be utilized to synthesize OLAT or reflectance field basis images. A single light source may be used to enhance complex light transport effects, such as specular highlights and subsurface scattering, enabling true photorealism. As described herein, the method may synthesize both diffuse and specular components, and may be configured to learn self-shadowing directly from the data.
[0045] Example Application Some embodiments include a graphical user interface of a computing device providing a user-adjustable slider bar to indicate an amount of adjustment of one or more of the specular or diffuse components. For example, a horizontal slider bar may be displayed, with the left end of the bar representing no image adjustment (e.g., an input image, etc.) and the right end of the bar representing a maximum predicted image adjustment (e.g., an output image, etc.). A middle portion of the bar may represent an intermediate image adjustment.
[0046] Such an embodiment further includes the graphical user interface receiving a user indication of an amount of adjustment of one or more of the specular or diffuse components. For example, a user may move a small virtual object from left to right along a slider bar and vice versa to indicate a position on the slider bar. An amount of image adjustment corresponding to the position of the small virtual object on the slider bar may be applied to the input image. Such an embodiment further includes the graphical user interface providing an output image based on the user-indicated amount of adjustment.
[0047] In some embodiments, providing the output image includes applying a linear interpolation of the input image and the output image. The linear interpolation may be based on a user instruction. As described herein, the user may indicate the amount of specularity to be reduced and / or the amount of light to be diffused. Thus, the computing device may linearly interpolate between the input image and the output image from the machine learning model. In general, the interpolation may be performed in image space. The amount of specularity may change as the image transitions from the input image to the output image. For example, based on a user instruction of the amount of diffusion and / or specularity desired, one of the intermediate images may be generated by interpolating between the input image and the output image. For example, pixels in the input image and the output image may be linearly interpolated to generate the intermediate image.
[0048] FIG. 3 illustrates linear interpolation of an input image 305 and an output image 330 according to an example embodiment. For example, one or more intermediate images 310, 315, 320, and 325 are illustrated. One or more intermediate images 310, 315, 320, and 325 may be generated by interpolating between the input image 305 and the output image 330. The input image 305 includes strong shadows and bright white highlights. As indicated, as the image transitions from the input image 305 through the intermediate images 310, 315, 320, and 325 to the output image 330, the specular highlights (e.g., the bright white on the subject's forehead) are gradually removed and the shadow areas (e.g., on the right side of the subject's face) are gradually brightened. As indicated in the output image 330, the bright white on the forehead is eliminated and the shadow areas under the eyes are reduced to a small extent. Furthermore, for example, the global color scheme is maintained.
[0049] In some embodiments, adjusting one or more of the specular or diffuse components includes a neural network predicting an output image based on a user-specified amount of adjustment. For example, the neural network may receive an amount of diffusion as an input. For example, a value of 80% diffusion may be provided to the neural network. Thus, the network may directly control the amount of diffusion in the output image. In some embodiments, a variant of the Phong reflectance model may be used to control the amount of diffusion in an HDR lighting environment used for training data, such as illustrated in image 210A of FIG. 2. Thus, the neural network may be trained to control the diffusion applied base on the training data.
[0050] Some embodiments include providing the output image as an input to another neural network configured to perform image relighting. For example, the output of the light redistribution network may be provided to an image relighting network that predicts a relighted image. For example, automatic directional lighting may be applied to the adjusted image output by the light redistribution network.
[0051] Some embodiments include providing the output image as an input to another neural network configured to perform portrait background replacement. For example, the output of the light redistribution network may be provided to a background replacement network that takes an input image with a first background and predicts an output image in a target background and target environment lighting. Additional and / or alternative image processing neural networks may be provided with the output of the light redistribution network.
[0052] As described herein, adjusting the specular or shadow components may be accomplished without predicting albedo, surface geometry, etc. Thus, such a simple neural network may be adaptable to mobile devices, as it removes significant constraints on computational resources.
[0053] Training a machine learning model to generate inferences / predictions FIG. 4 shows a diagram 400 illustrating a training phase 402 and an inference phase 404 of a trained machine learning model 432 according to an example embodiment. Some machine learning techniques include training one or more machine learning algorithms based on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions regarding (the patterns in) the training data. The resulting trained machine learning algorithms may be referred to as trained machine learning models. For example, FIG. 4 shows a training phase 402 in which one or more machine learning algorithms 420 are trained based on training data 410 to become trained machine learning models 432. Then, during the inference phase 404, the trained machine learning model 432 may receive input data 430 and one or more inference / prediction requests 440 (possibly as part of the input data 430) and responsively provide one or more inferences and / or predictions 450 as output.
[0054] As such, the trained machine learning models 432 may include one or more models of one or more machine learning algorithms 420. The machine learning algorithms 420 may include, but are not limited to, artificial neural networks (e.g., convolutional neural networks described herein, recurrent neural networks, Bayesian networks, hidden Markov models, Markov decision processes, logistic regression functions, support vector machines, suitable statistical machine learning algorithms, and / or heuristic machine learning systems). The machine learning algorithms 420 may be supervised or unsupervised and may perform any suitable combination of online and offline learning.
[0055] In some examples, the machine learning algorithm 420 and / or the trained machine learning model 432 may be accelerated using an on-device coprocessor, such as a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), and / or an application specific integrated circuit (ASIC). Such an on-device coprocessor may be used to speed up the machine learning algorithm 420 and / or the trained machine learning model 432. In some examples, the trained machine learning model 432 may be trained, reside, and execute to provide inferences and / or otherwise make inferences for a separate computing device.
[0056] During the training phase 402, the machine learning algorithm 420 may be trained by providing at least the training data 410 as a training input using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques. Unsupervised learning involves providing a portion (or all) of the training data 410 to the machine learning algorithm 420, and the machine learning algorithm 420 determining one or more output inferences based on the provided portion (or all) of the training data 410. Supervised learning involves providing a portion of the training data 410 to the machine learning algorithm 420, and the machine learning algorithm 420 determining one or more output inferences based on the provided portion of the training data 410, and the output inferences are accepted or otherwise corrected based on a correct result associated with the training data 410. In some examples, the supervised learning of the machine learning algorithm 420 may be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or the set of labels may be used to correct the inferences of the machine learning algorithm 420.
[0057] Semi-supervised learning involves having a correct result for some, but not all, of the training data 410. During semi-supervised learning, supervised learning is used for the portion of the training data 410 that has the correct result, and unsupervised learning is used for the portion of the training data 410 that does not have the correct result. Reinforcement learning involves the machine learning algorithm 420 receiving a reward signal for a prior inference, where the reward signal may be a numerical value. During reinforcement learning, the machine learning algorithm 420 may output an inference and receive a reward signal in response, where the machine learning algorithm 420 is configured to attempt to maximize the numerical value of the reward signal. In some examples, reinforcement learning further utilizes a value function that provides a numerical value that represents an expected total of the numerical value provided by the reward signal over time. In some examples, the machine learning algorithm 420 and / or the trained machine learning model 432 may be trained using other machine learning techniques, including, but not limited to, incremental learning and curriculum learning.
[0058] In some examples, the machine learning algorithm 420 and / or the trained machine learning model 432 may use transfer learning techniques. For example, the transfer learning techniques may include the trained machine learning model 432 being pre-trained on one set of data and additionally trained using the training data 410. More specifically, the machine learning algorithm 420 may be pre-trained on data from one or more computing devices and the resulting trained machine learning model provided to an individual computing device, which is intended to execute the trained machine learning model during the inference phase 404. Then, during the training phase 402, the pre-trained machine learning model may be additionally trained using the training data 410, which may be derived from kernel and non-kernel data of the individual computing device. This further training of the machine learning algorithm 420 and / or the pre-trained machine learning model using the training data 410 of the data of the individual computing device may be accomplished using either supervised learning or unsupervised learning. Once the machine learning algorithm 420 and / or the pre-trained machine learning model have been trained based on at least the training data 410, the training phase 402 may be completed. The resulting trained machine learning model may be utilized as at least one of the trained machine learning models 432.
[0059] In particular, once the training phase 402 has been completed, the trained machine learning model 432 may be provided to a computing device if not already on the computing device. The inference phase 404 may begin after the trained machine learning model 432 is provided to the respective computing device.
[0060] During the inference phase 404, the trained machine learning model 432 may receive the input data 430 and generate and output one or more corresponding inferences and / or predictions 450 regarding the input data 430. As such, the input data 430 may be used as input to the trained machine learning model 432 to provide the corresponding inferences and / or predictions 450 to the kernel and non-kernel components. For example, the trained machine learning model 432 may generate the inferences and / or predictions 450 in response to one or more inference / prediction requests 440. In some examples, the trained machine learning model 432 may be executed by another piece of software. For example, the trained machine learning model 432 may be executed by an inference or prediction daemon so that it is readily available to provide inferences and / or predictions upon request. The input data 430 may include data from a separate computing device executing the trained machine learning model 432 and / or input data from one or more computing devices other than the separate computing device.
[0061] The input data 430 may include a collection of images provided by one or more sources. The collection of images may include images of objects, such as a human face, where the images of the human face are taken under different lighting conditions, and the collection of images may further include images of multiple objects, images resident on a separate computing device, and / or other images. Other types of input data are possible as well.
[0062] The inferences and / or predictions 450 may include output foreground, alpha matte, albedo, surface normals, diffuse and specular maps, numerical, and / or other output data produced by the trained machine learning model 432 operating on the input data 430 (and training data 410). In some examples, the trained machine learning model 432 may use the output inferences and / or predictions 450 as input feedback 460. The trained machine learning model 432 may also rely on past inferences as input to generate new inferences.
[0063] The convolutional neural network 225 may be an example of a machine learning algorithm 420. After training, a trained version of the convolutional neural network 225, such as the light redistribution network 140, may be an example of a trained machine learning model 432. In this approach, an example of an inference / prediction request 440 may be a request to predict an output image including an object with adjusted one or more of the specular or diffuse components, and a corresponding example of an inference and / or prediction 450 may be the output image.
[0064] In some examples, a given computing device may include a trained version of convolutional neural network 140, perhaps after training convolutional neural network 225. The given computing device may then receive a request to predict an output image including an object with an adjusted one or more of the specular or diffuse components, and use the trained version of convolutional neural network 140 to generate the output image.
[0065] In some examples, more than one computing device may be used to provide an output image, for example, a first computing device may generate and send a request to a second computing device to predict an output image including an object with one or more of the specular or diffuse components adjusted. The second computing device may then use a trained version of the convolutional neural network 140, possibly after training the convolutional neural network 225, to generate the output image and respond to the request from the first computing device for the output image. The first computing device may then provide the requested output image (e.g., using a user interface and / or a display, a printed copy, electronic communication, etc.) upon receiving a response to the request.
[0066] Example Data Network 5 depicts a distributed computing architecture 500 according to an example embodiment. The distributed computing architecture 500 includes server devices 508, 510 configured to communicate with programmable devices 504a, 504b, 504c, 504d, 504e via a network 506. The network 506 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 506 may even correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public Internet.
[0067] 5 only shows five programmable devices, the distributed application architecture may serve tens, hundreds, or thousands of programmable devices. Moreover, programmable devices 504a, 504b, 504c, 504d, 504e (or any additional programmable devices) may be any type of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mountable device (HMD), a network terminal, a mobile computing device, and so on. In some examples, as illustrated by programmable devices 504a, 504b, 504c, 504e, the programmable devices may be directly connected to the network 506. In other examples, as illustrated by programmable device 504d, the programmable devices may be indirectly connected to the network 506 through an associated computing device, such as programmable device 504c. In this example, programmable device 504c may act as an associated computing device for carrying electronic communications between programmable device 504d and network 506. In other examples, as illustrated by programmable device 504e, the computing device may be part of and / or inside a vehicle, such as a car, truck, bus, boat or ship, airplane, etc. In other examples, not shown in FIG. 5, the programmable device may be connected to network 506 not only directly but also indirectly.
[0068] The server devices 508, 510 may be configured to perform one or more services as requested by the programmable devices 504a-504e. For example, the server devices 508 and / or 510 may provide content to the programmable devices 504a-504e. The content may include, but is not limited to, web pages, hypertext scripts, binary data such as compiled software, images, audio, and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or unencrypted. Other types of content are possible as well.
[0069] As another example, server device 508 and / or 510 may provide software for database, search, computational, graphical, audio, video, World Wide Web / Internet utilization, and / or other functions available to programmable devices 504a-504e. Many other examples of server devices are possible as well.
[0070] Computing Device Architecture 6 is a block diagram of a computing device 600 according to an example embodiment. In particular, the computing device 600 shown in FIG. 6 may be configured to perform at least one function of and / or associated with the convolutional neural network 140, 225, and / or method 800.
[0071] The computing device 600 may include a user interface module 601, a network communications module 602, one or more processors 603, data storage 604, one or more cameras 618, one or more sensors 620, and a power system 622, all of which may be linked together via a system bus, network, or other connection mechanism 605.
[0072] The user interface module 601 may be operable to send data to and / or receive data from external user input / output devices. For example, the user interface module 601 may be configured to send and / or receive data to and / or from user input devices such as a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, and / or other similar devices. The user interface module 601 may also be configured to provide output to a user display device, such as one or more cathode ray tubes (CRTs), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices now known or later developed. The user interface module 601 may also be configured to generate an audible output via devices such as a speaker, a speaker jack, an audio output port, an audio output device, an earphone, and / or other similar devices. User interface module 601 may further be configured with one or more haptic devices that may generate haptic outputs, such as vibrations and / or other outputs, detectable upon touch and / or physical contact with computing device 600. In some examples, user interface module 601 may be used to provide a graphical user interface (GUI) for utilizing computing device 600.
[0073] The network communication module 602 may include one or more devices providing one or more wireless interfaces 607 and / or one or more wireline interfaces 608 that are configurable to communicate over a network. The wireless interface 607 may include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™ transceiver, a Zigbee™ transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, an LTE™ transceiver, and / or other types of wireless transceivers that are configurable to communicate over a wireless network. The wireline interface 608 may include one or more wireline transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or similar transceiver that is configurable to communicate over a twisted pair of wires, a coaxial cable, a fiber optic link, or a similar physical connection to a wireline network.
[0074] In some examples, the network communication module 602 may be configured to provide reliable, secured, and / or authenticated communications. For each communication described herein, information to facilitate reliable communications (e.g., guaranteed message delivery) may be provided, possibly as part of the message header and / or footer (e.g., packet / message sequencing information, encapsulation header and / or footer, size / time information, and transmission verification information such as Cyclic Redundancy Check (CRC) and / or parity check values). Communications may be secured (e.g., encoded or encrypted) and / or decrypted / decoded using one or more cryptographic protocols and / or algorithms, such as, but not limited to, Data Encryption Standard (DES), Advanced Encryption Standard (AES), Rivest-Shamir-Adelman (RSA) algorithm, Diffie-Hellman algorithm, secure socket protocols such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms may be used to secure (and then decrypt / decode) communications as well as or in addition to those enumerated herein.
[0075] The one or more processors 603 may include one or more general purpose processors and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). The one or more processors 603 may also be configured to execute computer readable instructions 606 contained in data storage 604 and / or other instructions as described herein.
[0076] Data storage 604 may include one or more non-transitory computer-readable storage media that may be read and / or accessed by at least one of the one or more processors 603. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage, that may be fully or partially integrated with at least one of the one or more processors 603. In some examples, data storage 604 may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage unit), while in other examples, data storage 604 may be implemented using two or more physical devices.
[0077] The data storage 604 may include computer readable instructions 606 and possibly additional data. In some examples, the data storage 604 may include storage required to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functionality of the devices and networks described herein. In some examples, the data storage 604 may include storage for a trained neural network model 612 (e.g., a model of a trained convolutional neural network, such as convolutional neural network 140). In these examples, the computer readable instructions 606 may include instructions that, when executed by the processor 603, enable the computing device 600 to provide some or all of the functionality of the trained neural network model 612.
[0078] In some examples, computing device 600 may include one or more cameras 618. Camera 618 may include one or more image capture devices, such as a still and / or video camera, equipped to capture light and record the captured light in one or more images, i.e., camera 618 may generate an image of the captured light. The one or more images may be one or more still images and / or one or more images utilized in a video image. Camera 618 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light.
[0079] In some examples, computing device 600 may include one or more sensors 620. Sensors 620 may be configured to measure conditions within computing device 600 and / or in the environment of computing device 600 and provide data regarding these conditions.For example, the sensors 620 may include (i) sensors for obtaining data regarding the computing device 600, such as, but not limited to, a thermometer for measuring a temperature of the computing device 600, a battery sensor for measuring the power of one or more batteries of the power system 622, and / or other sensors for measuring a state of the computing device 600; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, a radio frequency identification (RFID) reader, a proximity sensor, a one-dimensional barcode reader, a two-dimensional barcode (e.g., a Quick Response (QR) Code) reader, and a laser tracker, which may be configured to read and provide at least information identifying an RFID tag, a barcode, a QR code, and / or an identifier of another device and / or object configured to be read; (iii) tilt sensors, gyroscopes, accelerometers, Doppler sensors, and other sensors for measuring the state of the computing device 600. The sensors 620 may include one or more of the following: (i) sensors for measuring the location and / or movement of the computing device 600, such as, but not limited to, a GPS device, a sonar sensor, a radar device, a laser displacement sensor, and a compass; (ii) environmental sensors for obtaining data indicative of the computing device 600's environment, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a motion sensor, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; and / or (iii) one or more sensors for measuring force, torque, ground force, friction in one or more dimensions, and / or a force sensor for measuring one or more forces (e.g., inertial forces and / or G-forces) acting on the computing device 600, such as, but not limited to, a zero moment point (ZMP) sensor that identifies a ZMP and / or a location of the ZMP. Many other examples of sensors 620 are possible as well.
[0080] The power system 622 may include one or more batteries 624 and / or one or more external power interfaces 626 to provide power to the computing device 600. Each battery of the one or more batteries 624 may serve as a source of stored power for the computing device 600 when electrically coupled to the computing device 600. The one or more batteries 624 of the power system 622 may be configured to be portable. Some or all of the one or more batteries 624 may be readily removable from the computing device 600. In other examples, some or all of the one or more batteries 624 may be internal to the computing device 600 and therefore may not be readily removable from the computing device 600. Some or all of the one or more batteries 624 may be rechargeable. For example, a rechargeable battery may be recharged by a wired connection between the battery and another power supply, such as by one or more power supplies that are external to the computing device 600 and connected to the computing device 600 via one or more external power interfaces. In other examples, some or all of the one or more batteries 624 may be non-rechargeable batteries.
[0081] The one or more external power interfaces 626 of the power system 622 may include one or more wired power interfaces, such as a USB cable and / or a power cord, that enable a wired power connection to one or more power supplies external to the computing device 600. The one or more external power interfaces 626 may include one or more wireless power interfaces, such as a Qi wireless charger, that enable a wireless power connection, such as via a Qi wireless charger, to one or more external power supplies. Once a power connection is established to an external power source using the one or more external power interfaces 626, the computing device 600 may draw power from the external power source, the established power connection. In some examples, the power system 622 may include associated sensors, such as a battery sensor associated with one or more batteries, or other types of power sensors.
[0082] Cloud-Based Server 7 depicts a network 506 of computing clusters 709a, 709b, 709c arranged as a cloud-based server system according to an example embodiment. The computing clusters 709a, 709b, and 709c may be cloud-based devices that store program logic and / or data for cloud-based applications and / or services, e.g., perform at least one function of and / or associated with the convolutional neural networks 140, 225, and / or method 800.
[0083] In some embodiments, computing clusters 709a, 709b, and 709c may be a single computing device resident in a single computing center. In other embodiments, computing clusters 709a, 709b, and 709c may include multiple computing devices in a single computing center, or even multiple computing devices located in multiple computing centers located in various geographic locations. For example, FIG. 7 depicts each of computing clusters 709a, 709b, and 709c resident in different physical locations.
[0084] In some embodiments, the data and services in computing clusters 709a, 709b, 709c may be encoded as computer-readable information stored on a non-transitory tangible computer-readable medium (or computer-readable storage medium) and accessible by other computing devices. In some embodiments, computing clusters 709a, 709b, 709c may be stored on a single disk drive or other tangible storage medium, or may run on multiple disk drives or other tangible storage media located in one or more diverse geographic locations.
[0085] 7, the functionality of the convolutional neural network 140, 225, and / or the computing devices may be distributed among computing clusters 709a, 709b, 709c. Computing cluster 709a may include one or more computing devices 700a, a cluster storage array 710a, and a cluster router 711a connected by a local cluster network 712a. Similarly, computing cluster 709b may include one or more computing devices 700b, a cluster storage array 710b, and a cluster router 711b connected by a local cluster network 712b. Similarly, computing cluster 709c may include one or more computing devices 700c, a cluster storage array 710c, and a cluster router 711c connected by a local cluster network 712c.
[0086] In some embodiments, each of computing clusters 709a, 709b, and 709c may have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each computing cluster may have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster may depend on the computing task or tasks assigned to each computing cluster.
[0087] In computing cluster 709a, for example, computing device 700a may be configured to perform various computing tasks of the convolutional neural network and / or computing device. In one embodiment, various functionality of the convolutional neural network and / or computing device may be distributed among one or more of computing devices 700a, 700b, and 700c. Computing devices 700b and 700c in each computing cluster 709b and 709c may be configured similarly to computing device 700a in computing cluster 709a. On the other hand, in some embodiments, computing devices 700a, 700b, and 700c may be configured to perform different functions.
[0088] In some embodiments, computing tasks and stored data associated with the convolutional neural network and / or computing devices may be distributed across the computing devices 700a, 700b, and 700c based at least in part on the processing requirements of the convolutional neural network and / or computing devices, the processing capabilities of the computing devices 700a, 700b, 700c, the latency of network links between the computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that may contribute to the cost, speed, fault tolerance, resiliency, efficiency, and / or other design goals of the overall system architecture.
[0089] The cluster storage arrays 710a, 710b, 710c of computing clusters 709a, 709b, and 709c may be data storage arrays including disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers, either alone or in conjunction with their respective computing devices, may also be configured to manage backup or redundant copies of data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.
[0090] Similar to the manner in which the functionality of the convolutional neural network and / or computing devices may be distributed across the computing devices 700a, 700b, 700c of the computing clusters 709a, 709b, 709c, various active and / or backup portions of these components may be distributed across the cluster storage arrays 710a, 710b, 710c. For example, some cluster storage arrays may be configured to store one portion of the data of the convolutional neural network and / or computing devices, while other cluster storage arrays may store other portions of the data of the convolutional neural network and / or computing devices. Further, for example, some cluster storage arrays may be configured to store data of a first convolutional neural network, while other cluster storage arrays may store data of a second and / or third convolutional neural network. In addition, some cluster storage arrays may be configured to store backup versions of data stored in other cluster storage arrays.
[0091] Cluster routers 711a, 711b, 711c in computing clusters 709a, 709b, and 709c may include networking equipment configured to provide internal and external communications for the computing clusters. For example, cluster router 711a in computing cluster 709a may include one or more Internet switching and routing devices configured to provide (i) local area network communications between computing device 700a and cluster storage array 710a via local cluster network 712a, and (ii) wide area network communications between computing cluster 709a and computing clusters 709b and 709c via wide area network link 713a to network 506. Cluster routers 711b and 711c may include network equipment similar to cluster router 711a, and cluster routers 711b and 711c may perform similar networking functions for computing clusters 709b and 709b that cluster router 711a performs for computing cluster 709a.
[0092] In some embodiments, the configuration of the cluster routers 711a, 711b, 711c may be based at least in part on the data communication requirements of the computing devices and cluster storage arrays, the data communication capabilities of the network equipment in the cluster routers 711a, 711b, 711c, the latency and throughput of the local cluster networks 712a, 712b, 712c, the latency, throughput, and cost of the wide area network links 713a, 713b, 713c, and / or other factors that may contribute to the cost, speed, fault tolerance, resiliency, efficiency, and / or other design criteria of the moderation system architecture.
[0093] Exemplary Methods of Operation 8 is a flowchart of a method 800 according to an example embodiment. The method 800 may be performed by a computing device, such as the computing device 600. The method 800 may begin at block 810, where the method includes the computing device receiving an image including an object.
[0094] At block 820, the method further includes the neural network adjusting one or more of the specular or diffuse components associated with the input image, where the adjusting includes redistributing light energy per pixel of the input image, as discussed above in the context of at least FIGS. 1-3.
[0095] At block 830, as discussed above in the context of at least FIGS. 1-3, the method further includes the neural network predicting an output image including the subject with an adjusted one or more of the specular or diffuse components.
[0096] In some embodiments, adjusting one or more of the specular component or the diffuse component includes adjusting the specular component, and redistributing the light energy per pixel includes reducing specular highlights associated with the subject.
[0097] In some embodiments, adjusting one or more of the specular or diffuse components includes adjusting the diffuse component, and redistributing the light energy per pixel includes reducing the light energy per pixel in shadow regions of the input image.
[0098] Some embodiments include the neural network maintaining an average of a global color value associated with the input image.
[0099] Some embodiments include predicting one or more characteristics of a coloration associated with the specular component.
[0100] In some embodiments, the input image is a portrait of a subject. In some embodiments, the neural network includes a U-net architecture configured to preserve high frequency aspects of the input image.
[0101] Some embodiments include a graphical user interface of the computing device providing a user-adjustable slider bar for indicating an amount of adjusting one or more of the specular or diffuse components. Such embodiments further include the graphical user interface receiving a user indication of an amount of adjusting one or more of the specular or diffuse components. Such embodiments further include the graphical user interface providing an output image based on the user-indicated amount of adjusting. In some embodiments, providing the output image includes applying a linear interpolation of the input image and the output image. The linear interpolation may be based on a user indication.
[0102] In some embodiments, adjusting one or more of the specular or diffuse components includes a neural network predicting an output image based on a user-specified amount of adjusting.
[0103] Some embodiments include training a neural network to receive individual input images with individual objects and predict individual output images including the objects with individual adjusted one or more of the specular or diffuse components.
[0104] In some embodiments, the training dataset includes multiple image pairs, where a first image of a given image pair may include a subject in a lighting environment and a second image of the given image pair may include a subject in a diffused version of the lighting environment.
[0105] In some embodiments, the lighting environment may be a high dynamic range lighting environment. Training includes generating a specular convolution of a portion of the high dynamic range lighting environment. In some embodiments, generating the specular convolution includes applying a Phong reflectance model.
[0106] In some embodiments, training includes applying an adversarial loss function to a selected portion of the subject, hi some embodiments, the selected portion may be a facial portion of the subject.
[0107] Some embodiments include providing the output image as an input to another neural network configured to perform image relighting.
[0108] Some embodiments include providing the output image as an input to another neural network configured to perform portrait background replacement.
[0109] The present disclosure should not be limited with respect to the specific embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from the spirit and scope of the present disclosure, as will be apparent to those skilled in the art. In addition to those mentioned herein, functionally equivalent methods and apparatuses within the scope of the present disclosure will be apparent to those skilled in the art from the above description. Such modifications and variations are intended to fall within the scope of the appended claims.
[0110] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, like symbols typically identify like components unless otherwise specified by context. The exemplary embodiments described in the detailed description, figures, and claims are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly contemplated herein.
[0111] With respect to any or all of the ladder diagrams, scenarios, and flow charts in the figures and as discussed herein, each block and / or communication may represent processing of information and / or transmission of information according to example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and / or messages may be executed out of the order shown or discussed, including substantially simultaneously or in reverse order, depending on the functionality involved. Additionally, more or fewer blocks and / or functions may be used with any of the ladder diagrams, scenarios, and flow charts discussed herein, and these ladder diagrams, scenarios, and flow charts may be combined with each other in part or in whole.
[0112] A block representing processing of information may correspond to circuitry that may be configured to perform certain logical functions of the methods or techniques described herein. Alternatively, or in addition, a block representing processing of information may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more instructions executable by a processor to perform certain logical functions or actions in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device, including a disk or hard drive or other storage medium.
[0113] The computer readable medium may further include non-transitory computer readable media, such as non-transitory computer readable media that store data for a short period of time, such as register memory, processor cache, and random access memory (RAM). The computer readable medium may further include non-transitory computer readable media that store program code and / or data for a longer period of time, such as secondary or permanent long-term storage, such as read-only memory (ROM), optical or magnetic disk, compact disk read-only memory (CD-ROM). The computer readable medium may further be any other volatile or non-volatile storage system. The computer readable medium may be considered, for example, a computer readable storage medium, or a tangible storage device.
[0114] Moreover, blocks representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device, however, other information transfers may be between software and / or hardware modules in different physical devices.
[0115] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those of ordinary skill in the art. The various aspects and embodiments disclosed herein are provided for illustrative purposes and are not intended to be limiting, with the true scope being indicated by the claims which follow.
Claims
1. 1. A computer-implemented method comprising: A computing device receiving an input image including an object; and a neural network adjusting one or more of a specular component or a diffuse component associated with the input image; said adjusting includes redistributing light energy per pixel of said input image; The computer implemented method comprises: The computer-implemented method further comprising the neural network predicting an output image including the object with the adjusted one or more of the specular or diffuse components.
2. 2. The computer-implemented method of claim 1, wherein the adjusting one or more of the specular component or the diffuse component comprises adjusting the specular component, and the redistributing the per-pixel light energy comprises reducing specular highlights associated with the object.
3. 2. The computer-implemented method of claim 1, wherein adjusting the one or more of the specular or diffuse components comprises adjusting the diffuse component, and wherein redistributing the per-pixel light energy comprises reducing per-pixel light energy in shadow regions of the input image.
4. The computer-implemented method of claim 1 or claim 2, further comprising maintaining, through the neural network, an average of a global color value associated with the input image.
5. The computer-implemented method of claim 1 , claim 2 , or claim 4 , further comprising predicting one or more characteristics of a coloration associated with the specular component.
6. The computer-implemented method of any of claims 1 to 5, wherein the input image is a portrait of the subject.
7. The computer-implemented method of any of claims 1 to 6, wherein the neural network comprises a U-net architecture configured to preserve high frequency aspects of the input image.
8. a graphical user interface of the computing device providing a user-adjustable slider bar for indicating an amount of the adjusting of the one or more of the specular or diffuse components; the graphical user interface receiving a user indication of the amount of the adjustment of the one or more of the specular or diffuse components; The computer-implemented method of any of claims 1 to 7, further comprising the graphical user interface providing the output image based on an amount of the user-instructed adjustment.
9. 9. The computer-implemented method of claim 8, wherein said providing of the output image comprises applying a linear interpolation of the input image and the output image, said linear interpolation being based on the user instruction.
10. 9. The computer-implemented method of claim 8, wherein adjusting the one or more of the specular or diffuse components includes the neural network predicting the output image based on an amount of the user-instructed adjustment.
11. 11. The computer-implemented method of claim 1, further comprising: training the neural network to receive individual input images with individual objects and to predict individual output images including the objects with individual adjusted one or more of the specular or diffuse components.
12. 12. The computer-implemented method of claim 11, wherein the training dataset includes a plurality of image pairs, a first image of a given image pair including a subject in a lighting environment and a second image of the given image pair including the subject in a diffused version of the lighting environment.
13. 13. The computer-implemented method of claim 12, wherein the lighting environment is a high dynamic range lighting environment, and the training comprises generating a specular convolution of a portion of the high dynamic range lighting environment.
14. The computer-implemented method of claim 13 , wherein the generating the specular convolution comprises applying a Phong reflectance model.
15. The computer-implemented method of claim 11 , wherein the training comprises applying an adversarial loss function to the selected portion of the subject.
16. The computer-implemented method of claim 15 , wherein the selected portion is a facial portion of the subject.
17. The computer-implemented method of claim 1 , further comprising providing the output image as an input to another neural network configured to perform image relighting.
18. The computer-implemented method of claim 1 , further comprising providing the output image as an input to another neural network configured to perform portrait-background replacement.
19. 1. A computing device comprising: one or more processors; and a data storage storing computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform functions including the computer-implemented methods of any one of claims 1 to 18.
20. A program which, when executed by one or more processors of a computing device, causes the computing device to perform functions including the computer implemented method of any one of claims 1 to 18.
