Image colorization using machine learning

Abstract

Embodiments described herein relate to methods, systems, and computer-readable media for training and using a machine learning model to colorize a grayscale image depicting a person. In some embodiments, a computer-implemented method includes receiving a grayscale image. The method further includes generating a colored image based on the grayscale image as an output of a trained convolutional neural network (CNN) by providing the grayscale image as an input to the trained CNN. In some embodiments, the trained CNN performs part segmentation to detect one or more parts of a person and colorizes the grayscale image.

Description

Background Art

[0001] Since cameras first became available, users have captured photographs to preserve memories. For example, photographs captured in the early days of photography were black and white images due to limitations in camera technology, due to the high cost of capturing color photographs, etc. Subsequently, as color photographs became available and less expensive, users took and stored color photographs.

[0002] Modern imaging applications allow users to store, view, and edit photos. Many users have scanned copies of old black and white photos in their library along with recently captured color photos. Users may like the old black and white photos if they were rendered as color photos.

[0003] Although image colorization techniques, including those based on machine learning, can be used to colorize black and white images, it is difficult to obtain a good output color image when such techniques are applied to old black and white photos, especially photos depicting one or more people. For example, errors such as skin areas not being consistent with skin tone color; one skin area, such as the face, getting color while another area, such as the hands, not getting color; etc. make the photo colorization unsatisfactory.

[0004] Furthermore, current image colorization techniques do not take into account the fact that people depicted in old black-and-white photos often wear different clothing styles than people in newer color photos. When adding color to a black-and-white photo depicting one or more people, current image colorization techniques cannot account for this difference. This is the domain adaptation problem.

[0005] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors, to the extent that it is described in this background section, and aspects of the specification that might not otherwise be prior art at the time of filing are not admitted, either explicitly or implicitly, as prior art against the present disclosure. Summary of the invention

[0006] Implementations described herein relate to methods, systems, and computer-readable media for training and using machine learning models to colorize grayscale images depicting one or more people.

[0007] In some embodiments, a computer-implemented method includes receiving a grayscale image including a person. The method further includes generating a colored image based on the grayscale image as an output of a trained convolutional neural network (CNN) by providing the grayscale image as an input to the trained CNN. In some embodiments, the trained CNN performs part segmentation to detect one or more parts of the person and colorizes the grayscale image.

[0008] In some embodiments, generating a shaded image includes obtaining an intermediate shaded image as an output of a latent layer of a trained CNN, obtaining a network output image as an output of an output layer of the trained CNN, and combining the intermediate shaded image and the network output image to obtain the shaded image. In some embodiments, the intermediate shaded image has a lower resolution than the network output image. In these embodiments, the method may further include upsampling the intermediate shaded image to the same resolution as the network output image before combining.

[0009] In some embodiments, the method may further include downsampling the grayscale image before providing the grayscale image as input to the trained CNN.

[0010] In some embodiments, the rendered image has a lower resolution than the grayscale image. In these embodiments, the method may further include upsampling the rendered image to the same resolution as the grayscale image and combining the rendered image with the grayscale image to obtain an output image. In some embodiments, the rendered image may be an sRGB image.

[0011] In some embodiments, a computer-implemented method for training a convolutional neural network to colorize a grayscale image includes providing training data, the training data including a plurality of training color images each depicting one or more persons, corresponding grayscale training images, and part annotations for the one or more persons in each training color image. The method further includes, for each grayscale training image, obtaining an output feature vector describing the colorized image as an output of the convolutional neural network based on the grayscale training image and the part annotations for the one or more persons in the grayscale training image. The method further includes, for each grayscale training image, obtaining a perceptual loss value from a pixel discriminator based on a label prediction by the pixel discriminator as to whether the colorized image is an original color image or a predicted color image. The method further includes, for each grayscale training image, modifying parameters of the convolutional neural network based on the perceptual loss value.

[0012] In some embodiments, the method of training a convolutional neural network may further include, for each grayscale training image, obtaining an intermediate feature vector describing an intermediate colored image from a potential layer of the convolutional neural network based on the grayscale training image and the intermediate part annotations for one or more people in the grayscale training image. The method may further include, for each grayscale training image, obtaining a feature loss value from a feature discriminator based on a prediction by the feature discriminator of whether there is a ground truth color image corresponding to the intermediate colored image. In these embodiments, the intermediate feature vector is provided as an input to the feature discriminator. The method may further include modifying parameters of the convolutional neural network based on the feature loss value. In some embodiments, the method may further include resizing the intermediate colored image, obtaining a network output image from an output layer of the convolutional neural network, and combining the intermediate colored image and the network output image to obtain a colored image.

[0013] In some embodiments, the method of training a convolutional neural network may further include, for each grayscale training image, obtaining a saturation loss from a saturation enhancer. In these embodiments, modifying the parameters of the convolutional neural network is further based on the saturation loss. In some embodiments, obtaining the saturation loss may include identifying one or more background regions of the colored image based on the part annotation, and for pixels identified as being in the background region, calculating the saturation loss based on the saturation of the colored image and the brightness value of the colored image.

[0014] In some embodiments, the method of training a convolutional neural network may further include, for each grayscale training image, obtaining a skin tone loss from a skin tone enhancer. In these embodiments, modifying the parameters of the convolutional neural network is further based on the skin tone loss. In some embodiments, obtaining the skin tone loss may include identifying one or more silhouettes in the colored image based on the part annotation and for each pixel of the colored image, calculating the skin tone loss based on the hue of the colored image and whether the pixel is in at least one of the one or more silhouettes in the colored image.

[0015] In some embodiments, a system includes one or more processors that implement a convolutional neural network having a plurality of layers. The convolutional neural network is configured to perform operations including receiving a grayscale image depicting one or more persons and generating a colored image based on the grayscale image. The colored image includes annotations of locations of the one or more persons in the grayscale image. The one or more processors further implement a pixel discriminator. The pixel discriminator is configured to perform operations including receiving a colored image and generating a label prediction indicating whether the colored image is an original color image or a predicted color image.

[0016] In some embodiments, the convolutional neural network is configured to perform further operations, which may include generating an intermediate colored image based on the grayscale image at a latent layer of the convolutional neural network. The intermediate colored image includes an annotation of intermediate parts of one or more people in the grayscale image. In these embodiments, the one or more processors are further implemented to include a pixel discriminator. The pixel discriminator is configured to perform operations including receiving the intermediate colored image and generating a prediction indicating whether there is a ground truth color image corresponding to the grayscale image.

[0017] In some embodiments, the intermediate colored image has a lower resolution than the grayscale image. In these embodiments, generating the colored image may further include resizing the intermediate colored image to the resolution of the grayscale image and, after resizing, combining the intermediate colored image with a network output image of an output layer of the convolutional neural network to obtain the colored image.

[0018] In some implementations, the predictions of the feature discriminator can be used to train the convolutional neural network by adjusting one or more parameters of the convolutional neural network.

[0019] In some embodiments, a pixel discriminator can be trained by providing as input a plurality of ground truth color images with associated ground truth labels indicating that the images are ground truth color images, and adjusting one or more parameters of the pixel discriminator using a loss based on the difference between the label prediction for each input image and the ground truth label.

[0020] In some implementations, the pixel discriminator can be trained based on a loss that compares label predictions to a ground truth label indicating whether the colorized image is the original color image or the predicted color image.

[0021] In some implementations, the label predictions of the pixel discriminator can be used to train the convolutional neural network by adjusting one or more parameters of the convolutional neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a block diagram of an example network environment that can be used with one or more implementations described herein.

[0023] Figure 2 is a block diagram illustrating an example generative adversarial network (GAN) configuration 200 for training a machine learning model to colorize images.

[0024] Figure 3 is a flow chart illustrating an example method of training a feature identifier according to some implementations.

[0025] Figure 4is a flow chart illustrating an example method of training a pixel discriminator in accordance with some implementations.

[0026] Figure 5 is a flow chart illustrating an example method of training a shader according to some implementations.

[0027] Figure 6 is a flow chart illustrating an example method for colorizing an image using a trained convolutional neural network in accordance with some embodiments.

[0028] Figure 7 is a block diagram of an example device that may be used with one or more implementations described herein. DETAILED DESCRIPTION

[0029] Embodiments described herein can generate color images from grayscale images using a trained machine learning model. The machine learning model can be trained to perform part segmentation to detect one or more parts of a person depicted in a grayscale image and colorize the grayscale image.

[0030] Some embodiments described herein relate to a generative adversarial network (GAN) configuration that can be used to train a generative model for executable part segmentation and image colorization. The GAN configuration can include a generative model, such as a convolutional neural network, and two adversarial models. In some embodiments, the generative model can be trained to generate a colored image from a grayscale image. A first adversarial model can be trained to determine whether the colored image is an image for which a ground truth color image exists. In some embodiments, the first adversarial model can receive an intermediate colored image generated at a potential layer of a generative model implemented using a convolutional neural network. In some embodiments, a second adversarial model can be trained to determine whether the colored image is a ground truth colored image or an image generated as an output of the generative model, such as an image generated from an output layer of a convolutional neural network.

[0031] In some embodiments, the loss values ​​generated by the first adversarial model and the second adversarial model are provided as training inputs to the generative model. In some embodiments, additional loss values ​​can be obtained based on heuristics related to image saturation (e.g., background areas in an image) and skin tone (e.g., for one or more people depicted in an image) and can be used to train the generative model.

[0032] Some embodiments described herein relate to a generative machine learning model that is trained and can generate a color image from an input grayscale image. The machine learning model can be trained to perform part segmentation to annotate one or more parts of a person depicted in an input image and color the input image. In some embodiments, the machine learning model is implemented using a convolutional neural network (CNN). In some embodiments, a color image is obtained by combining an intermediate colored image obtained from a potential layer of the CNN with a network output image obtained from an output layer of the CNN.

[0033] Figure 1 A block diagram illustrating an example network environment 100 that may be used in some implementations described herein. In some implementations, the network environment 100 includes one or more server systems, such as, Figure 1 102 and a second server system 140 in the example of FIG. 104. For example, the server systems 102 and 140 can communicate with the network 130. The server system 102 can include a server device 104 and a database 106 or other storage device. In some embodiments, the server device 104 can provide an image application 156b and / or a training application 158. The second server system 140 can include a second server device 142, which is configured to provide one or more applications, such as application A 144, application B 146, and application C 148. Figure 1 In the figures and the remaining figures, a letter following a reference number, e.g., "156a", indicates a reference to the element having that specific reference number. A reference number in the text without a following letter, e.g., "156", indicates a general reference to the embodiment of the element bearing that reference number.

[0034] The network environment 100 may also include one or more client devices, such as client devices 120, 122, 124, and 126, which may communicate with each other and / or with the server system 102 and / or the second server system 140 via a network 130. The network 130 may be any type of communication network, including one or more of the Internet, a local area network (LAN), a wireless network, a switch or hub connection, etc. In some embodiments, the network 130 may include peer-to-peer communications between devices, such as using a peer-to-peer wireless protocol (e.g., Wi-Fi Direct, etc.) etc. An example of peer-to-peer communication between two client devices 120 and 122 is shown by arrow 132.

[0035] For ease of illustration, Figure 1One box for server system 102, server device 104, database 106, second server system 140, and second server device 142 is shown, and four boxes for client devices 120, 122, 124, and 126 are shown. Server boxes 102, 104, 106, 140, and 142 can represent multiple systems, server devices, and network databases, and these boxes can be provided in different configurations than shown. For example, server system 102 and / or second server system 140 can represent multiple server systems that can communicate with other server systems via network 130. In some embodiments, for example, server system 102 and / or second server system 140 can include cloud hosting servers. In some examples, database 106 and / or other storage devices can be set in a server system box separate from server device 104 and can communicate with server device 104 and other server systems via network 130.

[0036] Moreover, any number of client devices may exist.Each client device may be any type of electronic device, for example, desktop computer, laptop computer, portable or mobile device, mobile phone, smart phone, tablet computer, television, TV set-top box or entertainment device, wearable device (for example, display glasses or goggles, wristwatch, headset, armband, jewelry, etc.), personal digital assistant (PDA), media player, gaming device, etc.Some client devices may also have a local database similar to database 106 or other storages.In some embodiments, network environment 100 may not have all components shown and / or may have other elements including other types of elements to replace those described herein or other elements including other types of elements in addition to those described herein.

[0037] In various embodiments, end users U1, U2, U3, and U4 may communicate with server system 102 and / or each other using respective client devices 120, 122, 124, and 126. In some examples, users U1, U2, U3, and U4 may interact with each other via network services implemented on server system 102 or second server system 140—e.g., social networking services, other types of network services—via applications running on respective client devices and / or server system 102 or second server system 140. For example, respective client devices 120, 122, 124, and 126 may communicate data to and from one or more server systems (e.g., system 102, second server system 140).

[0038] In some embodiments, the server system 102 or the second server system 140 can provide appropriate data to the client devices so that each client device can receive the communication content or shared content uploaded to the server system 102 or the second server system 140 and / or the network service. In some examples, the users U1-U4 can interact via audio or video conferencing, audio, video or text chat, or other communication modes or applications.

[0039] The network services implemented by the server system 102 or the second server system 140 may include systems that allow users to perform various communications, form links and associations, upload and publish shared content such as images, text, video, audio and other types of content, and / or perform other functions. For example, a client device may display received data, such as content posts that are sent or streamed to a client device and originate from different client devices (or directly from different client devices) or originate from a server system and / or network service via a server and / or network service. In some embodiments, client devices may communicate directly with each other, for example, using peer-to-peer communications between client devices as described above. In some embodiments, a "user" may include one or more programs or virtual entities, as well as people that interface with a system or network.

[0040] In some implementations, any of client devices 120, 122, 124, and / or 126 may provide one or more applications. Figure 1 As shown in , the client device 120 may provide an image application 156a and one or more other applications 154. The client devices 122-126 may also provide similar applications. The image application 156a may be implemented using the hardware and / or software of the client device 120. In different embodiments, the image application 156a may be an independent client application, for example, executed on any one of the client devices 120-124, or may work in conjunction with the image application 156b provided on the server system 102. The image application 156a and the image application 156b may provide various functions related to images. For example, such functions may include one or more of capturing images using a camera, analyzing images to associate one or more tags, modifying images, scanning images, coloring grayscale images, etc.

[0041] In some embodiments, the image application 156 may include a trained machine learning model. For example, a trained machine learning model, also referred to as a shader, may be trained to colorize a grayscale image. In some embodiments, the server device 104 may also include a model training application 158. In some embodiments, the image application 156 and / or the model training application 158 may perform training of the shader. In some embodiments, one or more additional machine learning models, such as discriminators, may be used to perform training. For example, a shader and one or more discriminators may be arranged in a generative adversarial network configuration for training.

[0042] In some embodiments, the client device 120 (or any of the other client devices 122-126) may include a shader (e.g., a machine learning model trained to colorize grayscale images) as part of the image application 156a and / or other applications 154. In some embodiments, the client device 120 (or any of the other client devices 122-126) may also include other machine learning models (e.g., a discriminator) and training instructions. In some embodiments, the client devices 120-126 may receive the training model from the server device 104 and may not include the training instructions and other machine learning models.

[0043] In some embodiments, the training instructions and discriminators can be provided as a separate application (e.g., model training application 158) executed on the server device 104. In these embodiments, the server device 106 can execute the training instructions to obtain a generative adversarial network configuration, which includes a shader and one or more discriminators and perform training. After completing the training, the model training application 158 can provide the trained shader for use in the image application 156.

[0044] In some implementations, the client device 120 may include one or more other applications 154. For example, the other applications 154 may be applications that provide various types of functionality, such as calendars, address books, email, web browsers, shopping, transportation (e.g., taxi, train, flight reservations, etc.), entertainment (e.g., music players, video players, game applications, etc.), social networks (e.g., messaging or chat, audio / video calls, sharing images / videos, etc.), and the like. In some implementations, one or more of the other applications 154 may be stand-alone applications executed on the client device 120. In some implementations, one or more of the other applications 154 may access a server system, such as the server system 102 and / or the second server system 140, to provide data and / or functionality of the other applications 154. For example, any of the applications 144, 146, and 148 shown as provided by the second server system 140 may provide data and / or commands to one or more of the other applications 154. In some implementations, the server applications 144-148 may be stand-alone applications accessed by a client device, such as via a web browser or other client program.

[0045] The user interface on the client device 120, 122, 124 and / or 126 can enable the display of user content and other content, including images, videos, data, and other content as well as communications, privacy settings, notifications, and other data. Such user interfaces can be displayed using software on the client device, software on the server device, and / or a combination of client software and server software executed on the server device 104 and / or the second server device 142, for example, application software or client software that communicates with the server system 102 and / or the second server device 142. The user interface can be displayed by a display device of the client device or server device, for example, a touch screen or other display screen, a projector, etc. In some embodiments, an application running on a server system can communicate with a client device to receive user input at the client device and output data such as visual data, audio data, etc. at the client device.

[0046] Other implementations of the features described herein may use any type of system and / or service. For example, other networking services (e.g., connected to the Internet) may be used instead of or in addition to social networking services. Any type of electronic device may utilize the features described herein. Some implementations may provide one or more features described herein on one or more client or server devices that are disconnected from or intermittently connected to a computer network. In some examples, a client device that includes or is connected to a display device may display content posts stored on a storage device local to the client device, e.g., previously received over a communication network.

[0047] As referred to herein, an image may include a digital image having pixels with one or more pixel values ​​(e.g., color values, brightness values, etc.). An image may be a still image (e.g., a still photograph, an image having a single frame, etc.), a dynamic image (e.g., an animation, an animated GIF, a film photograph of a portion of an image including motion while other portions are static, etc.), or a video (e.g., an image or sequence of image frames that may include audio). Although the remainder of this document relates to images that are still images, it is understood that the techniques described herein are applicable to dynamic images, videos, etc. For example, the embodiments described herein may be used with still images (e.g., photographs or other images), videos, or dynamic images.

[0048] Figure 2 is a block diagram illustrating an example generative adversarial network (GAN) configuration 200 for training a machine learning model to colorize an image. The GAN configuration can be implemented on a computer including one or more processors and a memory with software instructions. In some embodiments, the one or more processors can include a general-purpose central processing unit (CPU), a graphics processing unit (GPU), a machine learning processor—e.g., a TensorFlow processing unit (TPU) or other processor), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any other type of processor.

[0049] The GAN configuration 200 may include a plurality of machine learning models. In some implementations, the GAN configuration 200 may include a shader convolutional neural network (CNN) 210, a combiner 216, a feature discriminator 218, a resizer 220, and a pixel discriminator 222. In some implementations, the GAN configuration 200 may further include a saturation enhancer 224 and / or a skin tone enhancer 226.

[0050] like Figure 2As illustrated in , grayscale images including a real grayscale image 202 and a training grayscale image 204 are provided as input to a shader CNN 210. The real grayscale image 202 may be an image for which no color data is available. For example, such an image may include an old photo, such as captured using black and white film or other black and white images. The training grayscale image 204 may include an image that has been modified to remove color information. For example, the training grayscale image 204 may be obtained by removing color information from a ground truth color image 206. For example, a ground truth color image may include a color photo, such as captured using color film, a digital color camera, or other image capture technology that records color information. The ground truth color image may have an associated ground truth label indicating that the image is a ground truth color image (uncolored).

[0051] In some embodiments, the shader CNN 210 may be a ladder network or a U-Net model. In some embodiments, the shader CNN 210 may be a fully convolutional CNN model, for example, an autoencoder plus a skip connection. The shader CNN 210 may be organized into multiple layers. Each layer may include multiple neural network nodes. In some embodiments, each layer may include three convolution nodes (the remaining convolution nodes). In some embodiments, one or more nodes in a particular layer may be connected to a node in the immediately preceding layer and a node in the immediately following layer. In some embodiments, a skip connection that connects one or more nodes between non-continuous layers may be used. The first layer of the shader CNN 210 (referred to as the input layer) may receive a real grayscale image 202 and a training grayscale image 210 (collectively referred to as an input grayscale image) as input.

[0052] In addition, the real grayscale image 202 and the ground truth color image 206 can each depict one or more people. During the training phase of the shader CNN 210, ground truth part annotations for each of the one or more people can be provided as training input. The ground truth part annotations can be automatically generated, for example, using any suitable part segmentation technique, or manually generated, for example, by a human annotator. Because the generation of the ground truth part segmentation is independent of the color of the image, the ground truth part annotations can be generated for the real grayscale image (e.g., historical black and white photos) and the training grayscale or color images (e.g., modern photos).

[0053] CNN 210 can be trained to predict part annotations that match ground truth part annotations provided as training input. Thus, the trained shader CNN 210 generates a network output image that includes color values ​​predicted for pixels of the input image and part annotations for one or more persons depicted in the input image.

[0054] The shader CNN 210 is trained to analyze the input grayscale image to detect one or more features of the input grayscale image. Specifically, the shader CNN 210 may perform part segmentation to detect one or more parts of a human being depicted in the input grayscale image. In some embodiments, the one or more parts detected may include a face, a torso, a right arm, a left arm, a left leg, and a right leg. In some embodiments, the one or more parts detected may include a face, a torso, a right upper arm, a right lower arm, a left upper arm, a left lower arm, a right hand, a left hand, a right upper leg, a right lower leg, a left upper leg, a left calf, a left foot, and a right foot. Still other embodiments may detect more or fewer parts.

[0055] In some embodiments, the detection of one or more parts may include generating part annotations. For example, the part annotation may indicate whether a pixel of an image belongs to a particular part of a particular person, or whether a pixel is not part of a person depicted in the image. When multiple persons are depicted in the input image, the detection of parts may be performed such that a part is detected for each person depicted in the input image.

[0056] The shader CNN 210 is also trained to generate a color (chrominance) value for each pixel of the input image. The layers of the shader CNN 210 can be arranged so that a feature vector corresponding to the shaded image is generated by multiple layers of the shader CNN 210. For example, Figure 2 As shown in , the latent layer 212 and the output layer 214 can each generate a colored image from an input grayscale image.

[0057] like Figure 2 , the output of the latent layer 212 may include an intermediate colored image 230. In some embodiments, the intermediate colored image 230 may be output by the latent layer 212 as an intermediate feature vector. In some embodiments, the image resolution of the intermediate colored image 230 may be lower than the image resolution of the input grayscale image.

[0058] In addition, the output layer 214 can generate a network output image 232. In some embodiments, a residual network can be used as the output layer 214. The use of a residual network can accelerate the training of the shader CNN 210. In some embodiments, the image resolution of the network output image 232 can be the same as the resolution of the input grayscale image. In some embodiments, the network output image 232 can be output by the output layer 214 as a feature vector.

[0059] Although Figure 2 A colorizer CNN is illustrated, but it will be appreciated that any other type of generative machine learning model may be used to colorize an input grayscale image, such as an autoencoder, variational autoencoder, or other type of model.

[0060] In some embodiments, the GAN configuration 200 can further include a feature discriminator 218. The feature discriminator 218 can be implemented using any type of machine learning model, e.g., a classifier neural network. In some embodiments, a gradient reversal layer can be used. For example, the gradient reversal layer can provide feedback from the feature discriminator to the shader CNN 210, e.g., based on a loss value output by the feature discriminator. In some embodiments, the intermediate colored image 230 is provided as an input to the feature discriminator 218.

[0061] The feature discriminator 218 can be trained to determine whether an input image, e.g., any intermediate colored image, is an image for which a ground truth color image is available. For example, the feature discriminator 218 can analyze a feature vector of the input image to make such a determination. The feature discriminator 218 can generate a label prediction 250 as its output. In some embodiments, the label prediction 250 can be a binary value indicating whether the input image has a corresponding ground truth color image (“yes”) or does not have (“no”). In some embodiments, the label prediction 250 can be the likelihood that the input image has a corresponding ground truth color image, e.g., a probability value between 0 and 1. In some embodiments, the label prediction 250 can be a Wasserstein metric.

[0062] In some embodiments, the GAN configuration 200 may further include a resizer 220. The resizer 220 may receive the intermediate colored image 230 as an input and generate a resized intermediate image 234 as an output. For example, the resizer 220 may upsample the intermediate colored image from a low resolution to a higher resolution, such as the same resolution as the ground truth color image. The resizer 220 may be implemented using any type of image upsampling technique, including machine learning techniques, image upscaling algorithms, and the like.

[0063] In some embodiments, the GAN configuration 200 may include a combiner 216. The combiner 216 may receive the network output image 232, output by the output layer 214 of the shader CNN 210, and the resized intermediate image 234 as input, and may generate a colored image 236 as output. In some embodiments, each of the network output image 232, the resized intermediate image 234, and the colored image 236 may be represented as a corresponding feature vector representing the image (including pixel locations and color values) and the part annotation.

[0064] In some embodiments, the GAN configuration 200 can further include a pixel discriminator 222. The pixel discriminator 222 can be implemented using any type of machine learning model, such as a classifier neural network. For example, the pixel discriminator 222 can be implemented using a convolutional neural network that extracts features from a colorized image and classifies the features as belonging to a true color image or a predicted color image. An output convolutional layer can be used to produce a one-dimensional output, such as a label prediction.

[0065] The pixel discriminator 222 can be implemented to have a smaller receptive field and can account for image texture. Image texture is similar to ground truth color images (e.g., modern color photographs captured in color) and colorized images (e.g., generated from grayscale historical photographs). In some embodiments, the colorized image 236 and / or the ground truth color image 206 are provided as input to the feature discriminator 218.

[0066] The pixel discriminator 222 can be trained to analyze an input image, e.g., any of the colored images 236 and / or the ground truth color image 206, to determine whether the image is a true color image, e.g., one of the ground truth color images 206, or a predicted color image, e.g., one of the colored images 236. For example, the pixel discriminator 222 can analyze a feature vector of the input image to make such a determination. The pixel discriminator 222 can generate a label prediction 260 as its output. In some embodiments, the label prediction 260 can be a binary value indicating whether the input image is a true color image ("yes") or not ("no"). In some embodiments, the label prediction 260 can be a likelihood that the input image is a true color image, e.g., a probability value between 0 and 1.

[0067] The GAN configuration 200 is utilized to train the shader CNN 210 (generator network) by using the feature discriminator 218 and the pixel discriminator 222 as an adversarial network. One or more parameters of the shader CNN 210 are adjusted based on the label prediction 250 of the feature discriminator 218 and the label prediction 260 of the pixel discriminator 222. For example, the one or more parameters may include weights of one or more nodes of one or more layers of the shader CNN 210 and / or connections between one or more pairs of nodes of the shader CNN 210. A feature loss value may be determined based on the label prediction 250 and provided as a training input to the shader CNN 210. Additionally, a perceptual loss value may be determined by the pixel discriminator and provided as a training input to the shader CNN 210.

[0068] One goal of the shader CNN 210 is to generate intermediate shaded images that will cause the feature discriminator 218 to fail, e.g., be incorrect for the label prediction 250. In other words, the trained shader CNN 210 is configured to generate intermediate shaded images (intermediate feature vectors) from true grayscale input images that are indistinguishable to the feature discriminator 218 from the intermediate shaded images (intermediate feature vectors) from the training grayscale images.

[0069] Another goal of the shader CNN 210 is to generate a network output image that, when combined with the resized intermediate image, produces a shaded image that causes the pixel discriminator to fail, e.g., be incorrect for the label prediction 260. In other words, the trained shader CNN 210 is configured to generate a shaded image (feature vector) from a true grayscale input image that is indistinguishable from a ground truth color image by the pixel discriminator 222.

[0070] The GAN configuration 200 is also utilized to train a feature discriminator 218. During training, the intermediate colored images 230 from the shader CNN 210 are provided as input to the feature discriminator 218, and the accuracy of the label prediction 250 is evaluated. For example, the label prediction 250 is accurate if it indicates that the ground truth color is available for the intermediate colored image only when the intermediate colored image is generated from one of the training grayscale images 204, or equivalently indicates that the ground truth color is not available for the intermediate colored image only when the intermediate colored image is generated from one of the real grayscale images 202.

[0071] The GAN configuration 200 is also utilized to train a pixel discriminator 222. During training, the colored images 236 and the ground truth color images 206 from the shader CNN 210 are provided as inputs to the pixel discriminator 222, and the accuracy of the label predictions 260 is evaluated. For example, if the label prediction 260 indicates that the image color of the input image is realistic only when the input image is one of the ground truth color images 206, or equivalently indicates that the image color is not realistic only when the input image is one of the colored images 236 generated by the shader CNN 210, then the label prediction 260 is accurate for the input image.

[0072] In some implementations, the GAN configuration 200 may include a saturation enhancer 224. The colored image obtained from the combiner 216 may be provided as an input to the saturation enhancer 224.

[0073] The saturation enhancer 224 may be configured to generate a saturation loss 260 for training the shader CNN 210. When colorizing a grayscale image, one technical problem is that the resulting image often appears desaturated. Many users prefer images with higher saturation, such as modern images captured using color film cameras, digital cameras, etc. Using the saturation loss 260 to train the shader CNN 210 may ensure that the colored images produced by the trained shader CNN 210 have sufficient saturation.

[0074] In some embodiments, the saturation enhancer 224 may be configured to calculate a saturation penalty 260 such that low saturation in the HSV (hue, saturation, value) color space is penalized. The saturation penalty 260 may be a heuristic loss for training the shader CNN. The shaded image 236 generated by the shader CNN 210 is provided as input to the saturation enhancer 224. For example, the shaded image 236 may be in sRGB format (standard red, green, blue) and include predicted color values ​​for each pixel of the shaded image. The saturation enhancer 224 is configured to convert the pixel value to the HSV color space using an RGB to HSV conversion function RGB_to_HSV, as illustrated in the following formula:

[0075] (hue, saturation, value) = RGB_to_HSV(predicted_color)

[0076] In some implementations, the saturation enhancer 224 may calculate the saturation loss 260 from the "saturation" and "value" of each pixel of the rendered image (obtained using the formula above) by using the following formula:

[0077] Saturation loss = (1-saturation) × pow (value, weight)

[0078] Where pow represents the power function, value is the base, and weight is the exponent. The pow function can apply variable amounts of penalties to different pixels, for example, penalizing dark pixels of an image that are smaller than bright pixels based on the value.

[0079] In some embodiments, the value may represent the brightness of the shading image. In some embodiments, the weight may be set to 0.5. In some embodiments, the weight may be adjusted based on the brightness of the value. The weight controls the sensitivity of the calculated saturation loss to the brightness of the value.

[0080] In some embodiments, the saturation enhancer 224 may calculate a saturation loss 260 for a subset of pixels of the shaded image. For example, the saturation loss 260 may be calculated only for background regions of the shaded image. In these embodiments, the part annotations generated by the shader CNN 210 may be used to identify background regions, e.g., an image mask identifying foreground and background regions of the shaded image. For example, the background region may correspond to pixels of the shaded image that are not annotated as being part of a person depicted in the image. In other words, pixels of the shaded image that are part of a depiction of a person in the image may be excluded when calculating the saturation loss 260.

[0081] One or more parameters of the shader CNN 210 are adjusted based on the saturation loss 260 generated by the saturation enhancer 224. For example, the one or more parameters may include weights of one or more nodes of one or more layers of the shader CNN 210 and / or connections between one or more pairs of nodes of the shader CNN 210.

[0082] In some embodiments, the GAN configuration 200 may include a skin tone enhancer 224. The colorized image obtained from the combiner 216 may be provided as an input to the skin tone enhancer 226. A problem with colorized images generated from grayscale images is that skin pixels in the colorized image are sometimes assigned colors that are not in the value range corresponding to human skin color. For example, the colorized image may sometimes include skin pixels of purple or non-warm skin tones. Such colorized images may be considered unsatisfactory because the colors assigned to the skin pixels are inaccurate.

[0083] In some embodiments, the skin tone enhancer 226 can be configured to calculate a skin tone loss 270 for training the shader CNN 210 to eliminate or reduce the cases where the shaded image generated by the shader CNN 210 has skin pixels with wrong or inappropriate colors. The skin tone loss 270 may be a heuristic loss for training the shader CNN. In some embodiments, a specific hue value, for example, hue=20, can be set as a default skin tone value. The skin tone loss 270 is then calculated based on this value.

[0084] First, the shaded image 236 generated by the shader CNN 210 may be converted to the HSV color space, as described above with reference to the saturation enhancer 224. The skin tone loss 270 may be calculated for each pixel of the shaded image using the following formula:

[0085] Skin tone loss = (hue - 20) / 20 × silhouette x is_gray

[0086] Wherein, hue represents the hue of the pixel of the colored image, silhouette is a binary value indicating whether the pixel depicts a person, and is_gray indicates whether the source image of the colored image is a training grayscale image (e.g., any training grayscale image 204). Calculating the skin tone loss in this manner enables the shader CNN 210 to be trained in a supervised manner by comparing the predicted color, e.g., the color value of a pixel generated by the shader CNN 210, with the ground-truth color value of the pixel obtained from the corresponding ground-truth color image from the ground-truth color image 206.

[0087] The silhouette value may be obtained using the part annotations generated by the shader CNN 210. For example, pixels of the shaded image that are annotated as parts of a person (e.g., head, torso, arms, legs, etc.) may be identified as being within the silhouette (silhouette=1) and other pixels may be identified as being outside the silhouette (silhouette=0). In other words, pixels of the shaded image that are part of the depiction of a person in the image may be included while other pixels may be excluded when calculating the skin tone loss 270.

[0088] One or more parameters of the shader CNN 210 are adjusted based on the skin tone loss 270 generated by the skin tone enhancer 226. For example, the one or more parameters may include weights of one or more nodes of one or more layers of the shader CNN 210 and / or connections between one or more pairs of nodes of the shader CNN 210.

[0089] References Figure 3 , 4 5 describe example methods for training feature discriminator 218, pixel discriminator 222, and shader CNN 210.

[0090] Figure 3 is a flow chart illustrating an example method 300 for training a feature identifier (e.g., feature identifier 218) according to some embodiments. In some embodiments, method 300 may be performed, for example, in Figure 1 102 as shown. In some embodiments, some or all of method 300 may be implemented in a Figure 1, one or more client devices 120, 122, 124 or 126, one or more server devices, and / or both server devices and client devices as shown in . In the described examples, the implementation system includes one or more digital processors or processing circuits ("processors"), and one or more storage devices (e.g., database 106 or other storage). In some embodiments, different components of one or more servers and / or clients can execute different blocks or other parts of method 300. In some examples, the first device is described as a block that executes method 300. Some embodiments may have one or more blocks of method 300 that can send results or data to one or more other devices (e.g., other client devices or server devices) of the first device for execution.

[0091] In some embodiments, the method 300 or a portion of the method can be automatically initiated by the system. In some embodiments, the implementation system is the first device. For example, the method (or a portion thereof) can be executed periodically, or based on one or more specific events or conditions, such as the accuracy of the tag prediction below a threshold, the predetermined time period that has expired since the last execution of the method 300, and / or the occurrence of one or more other conditions that can be specified in the settings read by the method.

[0092] Method 300 may begin at block 302. At block 302, a shaded image, e.g., a feature vector representing a shaded image, is obtained. For example, intermediate shaded image 230 may be obtained from shader CNN 210 and provided as input. Block 302 may be followed by block 304.

[0093] At block 304 , a feature identifier, such as the feature identifier 218 , is applied to generate a label prediction, such as the label prediction 250 . Block 304 may be followed by block 306 .

[0094] At block 306, the label predictions are evaluated. For example, the accuracy of the label predictions is determined for each intermediate colorized image, e.g., evaluating whether the feature discriminator successfully distinguishes between intermediate colorized images for which ground truth colors are available, e.g., intermediate colorized images generated from training grayscale images 204, and intermediate colorized images for which ground truth colors are not available, e.g., intermediate colorized images generated from real grayscale images 202. A loss value is determined based on the accuracy of the label predictions. Block 306 may be followed by block 308.

[0095] At block 308, the loss value is used to adjust one or more parameters of the feature identifier. For example, when the feature identifier is implemented using a neural network, weights associated with one or more nodes of the neural network and / or connections between one or more pairs of nodes of the neural network can be adjusted based on the loss value.

[0096] Method 300 or portions thereof may be repeated any number of times using additional inputs. For example, method 300 may be repeated until a threshold level of label prediction accuracy is reached.

[0097] Figure 4 is a flow chart illustrating an example method 400 for training a pixel discriminator, such as pixel discriminator 222, according to some embodiments. In some embodiments, method 400 may be performed, for example, in Figure 1 4. In some embodiments, some or all of method 400 may be implemented on a server system 102 as shown. Figure 1 , one or more client devices 120, 122, 124, or 126, one or more server devices, and / or both server devices and client devices as shown in . In the described examples, the implementation system includes one or more digital processors or processing circuits ("processors"), and one or more storage devices (e.g., database 106 or other storage). In some embodiments, different components of one or more servers and / or clients can execute different blocks or other parts of method 400. In some examples, the first device is described as a block that executes method 400. Some embodiments may have one or more blocks of method 400 executed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0098] In some embodiments, the method 400 or a portion of the method may be automatically initiated by the system. In some embodiments, the implementing system is the first device. For example, the method (or a portion thereof) may be executed periodically, or based on one or more specific events or conditions, such as, the accuracy of the tag prediction below a threshold, the predetermined time period that has expired since the last execution of the method 400, and / or the occurrence of one or more other conditions that may be specified in the settings read by the method.

[0099] Method 400 may begin at block 402. At block 402, a ground truth color image is provided as input to a pixel discriminator, such as, for example, pixel discriminator 222. Block 402 may be followed by block 404.

[0100] At block 404, features of the ground truth color image are learned. For example, the pixel discriminator can learn an embedding, e.g., a set of values, that represents a distribution or pattern determined from the ground truth color image. Block 404 can be followed by block 406.

[0101] At block 406, a shaded image is obtained. For example, additional images may be obtained from the ground truth color image 206 and / or the shaded image 236 generated by the shader CNN 210. Block 406 may be followed by block 408.

[0102] At block 408, the pixel discriminator is applied to the colored image to generate a label prediction indicating whether each image in the colored image has a true color (e.g., the image is a ground truth color image) or a predicted color (e.g., the image is an image that has been colored). In some embodiments, the colored image provided as input can be an sRGB (standard red, green, blue) image, e.g., an image with three channels.

[0103] In some embodiments, a heat map based on one or more faces detected in the colored image can be provided as an input to the pixel discriminator. For example, the heat map can be a single channel indicating pixels of the colored image that are part of a person's face (including or excluding the upper body or other parts of the body). For example, the heat map can be generated using facial detection techniques and then generating a Gaussian heat map for each detected face in the colored image. In some embodiments, the pixel discriminator can generate a segmentation mask based on the colored image (sRGB) and the heat map. The segmentation mask can include one or more people depicted in the image, for example, a human body (silhouette) in the image can be distinguished from a background area of ​​the image. Block 408 can be followed by block 410.

[0104] At block 410, the label predictions are evaluated. For example, the accuracy of the label predictions is determined for each colorized image, e.g., evaluating whether the pixel discriminator successfully distinguishes between the colorized image (e.g., generated by the shader CNN 210) and the ground truth color image (e.g., the ground truth color image 206). A loss value is determined based on the accuracy of the label predictions. Block 410 may be followed by block 412.

[0105] At block 412, the loss value is used to adjust one or more parameters of the pixel discriminator. For example, when the pixel discriminator is implemented using a neural network, weights associated with one or more nodes of the neural network and / or connections between one or more pairs of nodes of the neural network may be adjusted based on the loss value.

[0106] In some embodiments, one or more additional inputs may be provided to the pixel discriminator. For example, an image classifier may be used to generate a feature vector generated based on an input grayscale image. In some embodiments, the classifier may be a machine learning classifier. In some embodiments, the feature vector may be a set of numbers specific to an image, such as a floating point number. The feature vector of an image may indicate the similarity of the image to one or more other images. For example, a classifier may generate a feature vector (e.g., a set of numbers) such that semantically and / or visually similar images are assigned sets that are close to each other. Thus, the feature vector may be used to identify images that are similar to an image.

[0107] Method 400 or portions thereof may be repeated any number of times using additional inputs. For example, blocks 402 and 404 may be repeated with a set of multiple ground truth color images. In another example, blocks 406-412 may be repeated with additional ground truth images and / or additional colorized images. Method 400 may be repeated until a threshold level of label prediction accuracy is reached.

[0108] Figure 5 is a flow chart illustrating an example method 500 for training a shader, such as shader CNN 210, according to some embodiments. In some embodiments, method 500 may be performed, for example, in Figure 1 102 as shown. In some embodiments, some or all of method 500 may be implemented in a Figure 1 , one or more client devices 120, 122, 124, or 126, one or more server devices, and / or both server devices and client devices as shown in . In the described examples, the implementation system includes one or more digital processors or processing circuits ("processors"), and one or more storage devices (e.g., database 106 or other storage). In some embodiments, different components of one or more servers and / or clients can execute different blocks or other parts of method 500. In some examples, the first device is described as a block that executes method 500. Some embodiments may have one or more blocks of method 500 executed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0109] In some embodiments, the method 500 or a portion of the method can be automatically initiated by the system. In some embodiments, the implementing system is the first device. For example, the method (or a portion thereof) can be executed periodically, or based on one or more specific events or conditions, such as the accuracy of the tag prediction below a threshold, the predetermined time period that has expired since the last execution of the method 500, and / or the occurrence of one or more other conditions that can be specified in the settings read by the method.

[0110] Method 500 may begin at block 502. At block 502, a grayscale image is provided as an input to a shader, such as shader CNN 210. The grayscale image may include a real grayscale image 202 and / or a training grayscale image 204. The grayscale image may include an image depicting one or more people. Block 502 may be followed by block 504.

[0111] At block 504, an intermediate shaded image and corresponding part annotations are generated. In some embodiments, the intermediate shaded image and corresponding part annotations may be represented by an intermediate feature vector generated by a latent layer, such as latent layer 212 of shader CNN 210. The part annotations may indicate a part segmentation of one or more persons depicted in the grayscale image, such as classifying each pixel of the input image as not depicting a person, or depicting a particular part of a person, such as a head, torso, arm, leg, etc. The intermediate feature vector is provided as an input to a feature discriminator, such as feature discriminator 218. Block 504 may be followed by block 506.

[0112] At block 506, a network output shaded image and corresponding part annotations are generated. In some embodiments, the network output shaded image and corresponding part annotations may be represented by a feature vector generated by an output layer, such as the output layer 214 of the shader CNN 210. Block 506 may be followed by block 508.

[0113] At block 508, the intermediate shaded image generated at block 506 is resized. For example, the intermediate shaded image generated at block 506 may be upsampled (or upconverted) to obtain a resized intermediate shaded image having the same resolution as the network output image generated at block 508. In some implementations, the resolution of the resized intermediate shaded image and the network output image may be the same as the resolution of the ground truth color image. Block 508 may be followed by block 510.

[0114] At block 510, the resized intermediate shaded image and the network output image are combined to obtain a shaded image. Block 510 may be followed by block 512.

[0115] At block 512, the colored image obtained in block 510 and the corresponding part annotations are provided as input to a pixel identifier, such as pixel identifier 222. Block 512 may be followed by block 514.

[0116] At block 514, label predictions from the feature discriminator and the pixel discriminator are obtained. In some embodiments, respective loss values ​​may be obtained from the feature discriminator and the pixel discriminator based on their respective label predictions. For example, a feature loss value may be obtained from the feature discriminator, and a perceptual loss value may be obtained from the pixel discriminator.

[0117] In some embodiments, a gradient reversal layer may be used. For example, the gradient reversal layer may provide feedback from the feature discriminator to the shader CNN 210. In some embodiments, the loss value output by the feature discriminator may be provided to the shader CNN via the gradient reversal layer.

[0118] In some embodiments, the perceptual loss value may be based on the difference between the ground truth color image and the corresponding shaded image generated by the shader CNN 210. The perceptual loss may be obtained using a trained neural network that determines the perceptual loss based on a comparison of an image pair. In some embodiments, for example, when the pixel discriminator is implemented using a multi-layer neural network, the output layer of the pixel discriminator may generate a perceptual loss value as an L2 loss between the ground truth image and the corresponding shaded image generated by the shader CNN 210. In some embodiments, the pixel discriminator may generate a label prediction based on the perceptual loss value.

[0119] The label prediction or loss value is utilized to train a shader, e.g., shader CNN 210. In some embodiments, the loss value may be an L1 loss value. For example, when the shader is implemented as a ladder CNN k, weights associated with one or more nodes of a neural network and / or connections between one or more pairs of nodes or layers of a neural network may be adjusted based on the loss value.

[0120] In some embodiments, for example, backpropagation from the network output image 232 and / or the shaded image 236 may be sent directly to the latent layer 212. Such backpropagation may speed up the training of the shader CNN 210.

[0121] Method 500 or portions thereof may be repeated any number of times, for example, until a trained shader generates a shaded image that causes label predictions for a feature discriminator and / or a pixel discriminator to meet a threshold level of inaccuracy.

[0122] Thus, a trained shader is obtained using method 500, which can take a grayscale image as input and generate a shaded image that adds color (chromaticity) to the grayscale image and part annotations of one or more people depicted in the grayscale image as output. Training using a feature discriminator can ensure that the shaded image generated by the trained shader has a structure similar to the ground truth color image. Using a feature discriminator that takes the output of the potential layer of the shader as input can ensure that the shader generates similar feature vectors for the real grayscale image and the training grayscale image. For example, a real grayscale image may include historical black and white photos for which no color data is available. In addition, training using a pixel discriminator can ensure that the shaded image generated by the trained shader has a texture similar to the ground truth color image.

[0123] The shader CNN 210 trained using the method 500 described above may be provided with a trained machine learning model that may be utilized to colorize an image. Thus, a trained shader CNN 210 may be provided without reference to Figure 2Other blocks described, for example, combiner 216, feature identifier 218, resizer 220, and / or pixel identifier 222. For example, the trained shader CNN 210 can be provided so that image shading can be performed on any device. For example, the trained shader CNN 210 can be provided on any of the client devices 120-126 as part of the image application 156a, as part of the other application 154, as part of the operating system of the client device 120-126.

[0124] In some embodiments, one or more additional blocks of the GAN configuration 200 or the entire GAN configuration 200 may be provided on the device, for example, when training (or retraining) is performed on the device executing the shader CNN 210. For example, the server device 104 may implement the GAN configuration 200 in the image application 156b and / or the model training application 158.

[0125] Figure 6 6 is a flowchart illustrating an example method 600 for coloring an image using a trained convolutional neural network according to some embodiments. For example, the trained convolutional neural network may be the colorizer CNN 210, which is generated using the above referenced Figure 5 The described technique is obtained after training.

[0126] In some embodiments, method 600 may be performed, for example, Figure 1 102 as shown. In some embodiments, some or all of method 600 may be implemented in a Figure 1 The method 600 is implemented on one or more client devices 120, 122, 124, or 126, one or more server devices, and / or both a server device and a client device. In the described examples, the implementation system includes one or more digital processors or processing circuits ("processors"), and one or more storage devices (e.g., database 106 or other storage). In some embodiments, different components of one or more servers and / or clients can perform different blocks or other parts of the method 600. In some examples, the first device is described as a block that performs the method 600. Some embodiments may have one or more blocks of the method 600 performed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0127] In some embodiments, method 600 or a portion of the method can be automatically initiated by the system. In some embodiments, the implementing system is the first device. For example, the method (or a portion thereof) can be executed periodically, or based on one or more specific events or conditions, such as the accuracy of the tag prediction below a threshold, the predetermined time period that has expired since the last execution of method 400, and / or the occurrence of one or more other conditions that can be specified in the settings read by the method.

[0128] Method 600 may begin at block 602. At block 602, a grayscale image is received. For example, the grayscale image may be an old black and white photograph depicting one or more persons, such as a scanned photograph. For example, the grayscale image may be received as user input, such as when a user activates an image colorization feature of a software application, such as by selecting a user interface option displayed on a user interface. For example, the user may activate the user interface option by clicking or tapping the user interface, by a voice command, etc.

[0129] In another example, where the colorization feature is provided via an application programming interface (API) or otherwise provided as a service, the grayscale image can be received via a program call to the API, uploaded to an image colorization service, etc. The API can be provided, for example, as an operating system API for a smartphone, digital camera, tablet computer, or any other device with image capture capabilities. The API can be utilized, for example, by an application executing on the device to issue a request for image colorization.

[0130] In some embodiments, the grayscale image can be an image captured using a digital camera. For example, the digital camera may capture a grayscale image when the scene being photographed is dark. For example, such capture can be performed by illuminating the scene with infrared light and capturing the image with an infrared-enabled sensor. In this type of capture, the person activating the camera and the people in the scene cannot see the illuminated light because it is outside the visible light range (infrared), but the infrared imaging sensor of the digital camera can capture a black and white photo. For example, the digital camera can be a smartphone camera. In another example, the digital camera can be a security camera with an infrared emitter and an infrared-enabled sensor.

[0131] Although the above description relates to the specific example of a grayscale image, it will be appreciated that any grayscale image may be utilized, for example, a digital image without chrominance or color information. Block 602 may be followed by block 604 .

[0132] At box 604, the grayscale image is downsampled. Downsampling the grayscale image can include reducing the image resolution. For example, the image resolution can be changed to 256x256 pixels, 352x352 pixels or other suitable resolutions, and the received grayscale image can have a higher resolution, for example, 1024x1024 pixels, 2048x2048 pixels or any other resolution. Downsampling the image can reduce the computational complexity of image coloring, for example, coloring the downsampled image may require lower processor resources, lower memory, etc. The reduction in computational load can also enable coloring to be performed on devices with relatively low computing power, such as smartphones, tablet computers, digital cameras, etc. In some embodiments, the target resolution achieved after downsampling can be selected based on the configuration of the shader convolutional neural network (e.g., shader CNN210) used to color the image. Block 604 can be followed by block 606.

[0133] At block 606, an intermediate colored image based on the grayscale image is obtained from a latent layer of a shader CNN (e.g., shader CNN 210) that has been trained to colorize grayscale images, e.g., as described above with reference to Figure 5 In some embodiments, the resolution of the intermediate shaded image may be lower than the resolution of the input grayscale image or the downsampled grayscale image. For example, the intermediate shaded image may have a resolution of 16x16 pixels, 32x32 pixels, 64x64 pixels, etc. In some embodiments, the resolution of the intermediate shaded image may be based on a specific potential layer of the shader CNN, and the intermediate shaded image is obtained from the specific potential layer. Block 606 may be followed by block 608.

[0134] At block 608, the intermediate rendered image is resized. For example, the intermediate rendered image may be resized (upsampled) to the same resolution as the grayscale image or the downsampled grayscale image. Block 608 may be followed by block 610.

[0135] At block 610, a network output image based on a grayscale image is obtained from an output layer of a shader CNN (e.g., shader CNN 210). In some embodiments, the resolution of the network output image may be the same as the resolution of the grayscale image or the downsampled grayscale image. Block 610 may be followed by block 612.

[0136] At block 612, the resized intermediate shading image and the network output image are combined to obtain a color image. Block 612 may be followed by block 614.

[0137] At block 614, the rendered image is upsampled. For example, the rendered image may be upsampled to the same size as the grayscale image. Further, the rendered image is combined with the grayscale image, for example, the grayscale image or the downsampled grayscale image. An output image is obtained as a result of the combination. The output image may have the same resolution as the grayscale image received in block 602.

[0138] Method 600 can be used to colorize any grayscale image. For example, method 600 can be used to colorize a grayscale image depicting one or more people. For example, method 600 can be used to colorize a black and white photo, for example, a photo captured using a camera that has been digitized. In some embodiments, a user interface including an output image can be displayed. In some embodiments, the user interface can enable a user to view and edit the output image, for example, to adjust saturation, brightness, white balance, etc. In some embodiments, the user interface can show the output image and the input grayscale image, for example, side by side or in other configurations. In some embodiments, the user interface can include options that enable a user to switch the display of the output image and the input grayscale image; save the output image to storage; print the output image; upload the output image to an image library or image sharing service; send the output image to other users via a messaging application; and the like.

[0139] Method 600 may be repeated for any number of grayscale images. In some embodiments, one or more blocks of method 600 may be omitted, or combined with other blocks. For example, in some embodiments, block 604 may be omitted, for example, when the shader CNN can process the grayscale image directly (e.g., when method 600 is implemented on a computing device (e.g., a server) with high computing power; when the shader CNN can accept high-resolution input; etc.). In some embodiments, for example, when the shaded image generated by the shader CNN has the same or similar resolution as the grayscale image, block 614 may be performed to combine the shaded image and the grayscale image without first upsampling the shaded image.

[0140] In some embodiments, a training model, such as a shader CNN 210, a feature identifier 218, or a pixel identifier 222, can be updated, for example, based on the use of additional training data. Additional training may help improve the accuracy and / or speed of the model. In some embodiments, for example, when a user agrees to use user data to update the training model, the training model can be updated locally based on user activity or response, for example, on a particular client device. For example, such user data may include one or more images, such as a color image or a grayscale image. In these embodiments, the user data is not transmitted or shared. In addition, if the user agrees, updates to the trained model can be provided to, for example, a server device 104 or one or more other client devices via the network 130.

[0141] In some embodiments, the training model can be customized based on the device (e.g., client devices 120-124). For example, based on the available hardware, different client devices may have different processing capabilities, such as a single-core processor, a multi-core processor, one or more parallel processors, a graphics processor with multiple graphics processing units (GPUs), a reconfigurable processor (e.g., an FPGA), a neural network processor, etc. In addition, different client devices may have different levels of availability of other resources, such as memory, battery capacity, network bandwidth, etc.

[0142] For example, a trained model with relatively high complexity, such as that used by an inference engine, may be provided in a device having a neural network processor or GPU optimized for parallel processing (e.g., any of client devices 120-124 or server device 104). In this example, the greater processing power of the client device is utilized to implement the trained model.

[0143] In another example, a simplified training model (e.g., having fewer neural network nodes or layers) can be provided in a client device with greater resource constraints, such as a single core processor with limited capabilities. In some embodiments, multiple training models can be provided on a client device. When calling, for example, to colorize an image, apply a local classifier, the appropriate trained model can be utilized, for example, based on contextual factors such as network connectivity, available battery capacity, available memory, etc.

[0144] One problem when automatically colorizing grayscale images is that it is difficult to achieve the correct white balance in the generated output image. Implementations described herein can colorize images with proper white balance. A shader convolutional neural network that has been trained using a ground truth color image can generate a colored image with properly white balanced pixel values.

[0145] Another problem when automatically colorizing grayscale images is ensuring that the colorized image has suitable and diverse colors. For example, if the grayscale image is a black and white photo depicting multiple people, it is best not to have different people depicted in similar colors, for example, all wearing clothes with similar hues. The described technique can produce a variety of colors by ensuring that there is no 1:1 match between the pixel values ​​of the input grayscale pixels and the pixel values ​​of the colorized image.

[0146] Another problem arises when the grayscale image is an old image depicting one or more people, for example, captured in an earlier era, capturing subjects from a different culture, etc. In this case, ground truth training images from a similar era or culture may not be available. Machine learning models trained using modern photos or photos from a different culture may not be able to adapt to such images. People from previous eras often wore different clothing styles and different color shades compared to those in modern photos. Lacking a sufficient number of training images from such eras, various types of errors may occur in the colorized images generated by such machine learning models. For example, errors such as inconsistent colorization of skin areas, for example, facial areas are colored differently from hands; skin areas are assigned the same color as clothes; and so on may occur. This can be called the domain adaptation problem.

[0147] Some embodiments described herein address these technical issues and can produce colored images with good white balance and color diversity, and can adapt to different domains of input images. Specifically, as described herein, training a machine learning model to perform both part segmentation and colorization can provide a trained model that overcomes the described problems. In addition, the embodiments described herein use a generative adversarial network (GAN) configuration that includes two adversarial discriminator networks that generate different label predictions and / or loss values. Training the machine learning model with these two loss values ​​can ensure that the machine learning model is robust to different types of grayscale input images and generates colored images based on both image structure and image texture.

[0148] There are several technical advantages to using a machine learning model that is trained to perform both part segmentation and image colorization. For example, the part segmentation information generated by the trained model identifies parts of an input image that correspond to clothing worn by one or more people depicted in the input image, and the model generates the color of the clothing based on the color image used to train the model.

[0149] Another benefit is that the trained model can more accurately identify skin regions based on the part annotations, which can ensure that all skin regions of the person depicted in the input image are colored. Using skin tone loss can ensure that the skin regions do not have colors that are not skin color.

[0150] Furthermore, the use of saturation loss can generate an output image with a large amount of saturation. In addition, in some embodiments, the use of saturation loss can be limited to areas that do not correspond to people, for example, areas of the input image that are not identified in the part annotation. Limiting the use of saturation loss in this way can prevent the skin areas of people depicted in the input image from being oversaturated in color.

[0151] While the foregoing description relates to a shader CNN that performs part segmentation and colorization, other models may also be trained to colorize images depicting people. For example, a model may be trained to detect the pose of a person depicted in an image and perform colorization. Pose detection may include, for example, detecting multiple key points of a person. For example, a key point may include image pixels corresponding to one or more of a head, shoulder, elbow, wrist, hip, knee, and ankle of a person depicted in an image. Pose detection may also include detecting connections between these points, such as skeletal connections.

[0152] Furthermore, when different datasets have different distributions, for example, depicting the frequency of people wearing certain types of headwear, such as hats, a dataset classifier can be used during training to ensure that a shader CNN that generates shaded images does not have colored images based on specific features of the input dataset, or does not learn features from the training dataset that are not present in the input image. For example, a discriminator (e.g., a pixel discriminator) can receive the dataset type as input) and can use it to penalize the generative model by generating training feedback, thereby reducing the likelihood that the generative model learns such features.

[0153] In addition, as an alternative to or in addition to the perceptual loss value, a neighborhood color loss or chroma loss of one or more pixels of the shaded image can be calculated based on a per-pixel calculation of the mean squared error. For example, for each pixel for which the neighborhood color loss is calculated, a mask can be created to identify other pixels of the image, for example, neighboring pixels with similar pixel colors. Masks can be created for the shaded image and the ground truth color image generated by the shader CNN. The mean squared errors of the respective masks can be compared to obtain the neighborhood color loss. The neighborhood color loss can be provided as feedback to the shader CNN and can ensure that the shaded image output by the shader CNN has appropriate white balance and color diversity.

[0154] Figure 7 is a block diagram of an example device 700 that can be used to implement one or more features described herein. In one example, the device 700 can be used to implement a client device, such as, Figure 1101. Alternatively, device 700 may implement a server device, such as server 101. In some embodiments, device 700 may be a computer system, a server device, or both a client device and a server device. Device 700 may be any suitable computer system, server, or other electronic or hardware device as described above.

[0155] One or more methods described herein may be executed in a standalone program executable on any type of computing device, a program running on a web browser, a mobile application ("app") running on a mobile computing device (e.g., a cell phone, a smart phone, a tablet computer, a wearable device (a watch, an armband, jewelry, headwear, virtual reality goggles or glasses, augmented reality goggles or glasses, a head mounted display, etc.), a laptop computer, etc. In one example, a client / server architecture may be used, for example, a mobile computing device (as a client device) sends user input data to a server device and receives final output data from the server for output (e.g., for display). In another example, all computations may be performed within a mobile app (and / or other apps) on the mobile computing device. In another example, computations may be split between the mobile computing device and one or more server devices.

[0156] In some embodiments, the device 700 includes a processor 702, a memory 704, and an input / output (I / O) interface 706. The processor 702 may be one or more processors and / or processing circuits for executing program code and controlling the basic operation of the device 700. A "processor" includes any suitable hardware system, mechanism, or component for processing data, signals, or other information. The processor may include a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single core, dual core, or multi-core configuration), multiple processing units (e.g., in a multi-processor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit for implementing functionality, a dedicated processor for implementing processing based on a neural network model, a neural circuit, a system of processors optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, the processor 702 may include one or more coprocessors that implement neural network processing. In some embodiments, the processor 702 may be a processor that processes data to generate a probabilistic output, for example, the output generated by the processor 702 may be imprecise or may be accurate within a range from the expected output. Processing need not be limited to a particular geographic location or have time limits. For example, a processor may perform its functions in "real time," "offline," in "batch mode," etc. Portions of a process may be performed at different times and in different locations by different (or the same) processing systems. A computer may be any processor in communication with a memory.

[0157] The memory 704 is typically provided in the device 700 for access by the processor 702, and may be any suitable processor-readable storage medium suitable for storing instructions executed by the processor and located separate from and / or integrated with the processor 702, such as a random access memory (RAM), a read-only memory (ROM), an electrically erasable read-only memory (EEPROM), a flash memory, etc. The memory 704 may store software operated by the processor 702 on the server device 700, including an operating system 708, a machine learning application 730, other applications 712, and application data 714. The other applications 712 may include, for example, a camera application, an image library, an image library application, a data display engine, a web hosting engine, an image display engine, a notification engine, a social networking engine, and the like. In some embodiments, the machine learning application 730 and the other applications 712 may each include enabling the processor 702 to perform the functions described herein—for example, Figure 2 , Figure 3 , Figure 4 as well as Figure 7 Some or all of the methods - instructions.

[0158] Other applications 712 may include, for example, image editing applications, media display applications, communication applications, web hosting engines or applications, mapping applications, media sharing applications, etc. One or more methods disclosed herein may operate in a number of environments and platforms, for example, as a stand-alone computer program that can run on any type of computing device, as a web application with a web page, as a mobile application ("app") running on a mobile computing device, etc.

[0159] In various embodiments, the machine learning application can utilize a Bayesian classifier, a support vector machine, a neural network, or other learning techniques. In some embodiments, the machine learning application 730 may include a training model 734, an inference engine 736, and data 732. In some embodiments, the data 732 may include training data, e.g., data used to generate the training model 734. For example, the training data may include any type of data, such as text, images, audio, video, etc. When the trained model 734 is a shader, e.g., a shader CNN 210 or other type of shader model, the training data may include a ground truth color image 206.

[0160] The training data may be obtained from any source, such as a data repository specifically labeled for training data that has provided permission for use as training data for machine learning, etc. In embodiments where one or more users have permitted the use of their respective user data to train a machine learning model (e.g., training model 734), the training data may include such user data. In embodiments where users have permitted the use of their respective user data, data 732 may include permitted data, such as images (e.g., photographs or other user-generated images).

[0161] In some embodiments, the training data may include synthetic data generated for training purposes, such as data that is not based on user input or activity in the context being trained, for example, data generated from simulated photos or other computer-generated images. In some embodiments, the machine learning application 730 excludes data 732. For example, in these embodiments, the training model 734 may be generated, for example, on a different device, and the training model 734 is provided as part of the machine learning application 730. In various embodiments, the training model 734 may be provided as a data file including a model structure or form and associated weights. The inference engine 736 may read the data file for the training model 734 and implement a neural network with node connections, layers, and weights based on the model structure or form specified in the training model 734.

[0162] In some embodiments, the training model 734 may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep neural network that implements multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., splits or divides the input data into multiple parts or tiles, processes each tile separately using one or more neural network layers, and aggregates the results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that takes sequential data (such as words in a sentence, frames in a video, etc.) as input and produces a sequence of results as output), etc. The model form or structure may specify the connectivity between various nodes and the organization of nodes into layers.

[0163] For example, nodes of a first layer (e.g., an input layer) may receive data as input data 732 or application data 714. For example, when the training model 734 is a shader, the input data may include a grayscale image, such as a real grayscale image 202 and / or a training grayscale image 204. Such data may include, for example, one or more pixels of each node, for example, when the training model is used for image analysis or image generation. Subsequent intermediate layers may receive the output of the previous layer nodes as input according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers or potential layers.

[0164] The last layer (e.g., output layer) produces the output of the machine learning application. For example, the output can be a colored image with part annotations identifying one or more parts of a person depicted in the image. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0165] In various embodiments, the training model 734 may include multiple nodes arranged in layers according to a model structure or form. In some embodiments, the node may be a computing node without memory, for example, a unit configured to process an input to produce an output. The calculation performed by the node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to produce a node output. In some embodiments, the calculation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such calculations may include operations such as matrix multiplication. In some embodiments, calculations performed by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, using individual processing units of a GPU, or dedicated neural circuits. In some embodiments, the node may include memory, for example, it may be possible to store and use one or more earlier inputs when processing subsequent inputs. For example, a node with memory may include a long short-term memory (LSTM) node. The LSTM node may use memory to maintain a "state" that allows the node to act like a finite state machine (FSM). Models with such nodes can be useful when processing sequential data—e.g., words in a sentence or paragraph, frames in a video, speech or other audio, etc.

[0166] In some embodiments, the training model 734 can include embeddings or weights for various nodes. For example, the model can be started as a plurality of nodes organized into layers as specified by the model form or structure. At initialization, corresponding weights can be applied to the connections between each pair of nodes connected in the model form, e.g., nodes in successive layers of a neural network. For example, corresponding weights can be randomly assigned or initialized to default values. The model can then be trained, e.g., using data 732, to produce results.

[0167] For example, training can include applying supervised learning techniques. In supervised learning, the training data can include multiple inputs (e.g., a set of grayscale images) and corresponding expected outputs for each input (e.g., a set of ground truth images corresponding to grayscale images or other color images). Based on a comparison of the output of the model with the expected output, the value of the weight is automatically adjusted, for example, in a manner that increases the probability that the model will produce the expected output when provided with similar inputs.

[0168] In some embodiments, training can include applying unsupervised learning techniques. In unsupervised learning, only input data can be provided and the model can be trained to distinguish the data, for example, to cluster the input data into a plurality of groups, where each group includes input data that is similar in some manner. For example, the model can be trained to distinguish images, such that the model distinguishes a colorized image - e.g., a colorized image generated from an input grayscale image using a colorizer - from a ground truth color image - e.g., a color photograph. Feature discriminator 218 and / or pixel discriminator 222 can be trained to distinguish the data, as described above with reference to Figure 2 , Figure 3 and 4 as described.

[0169] In some embodiments, unsupervised learning can be used to generate a knowledge representation, for example, that can be used by the machine learning application 730. For example, unsupervised learning can be used to generate an embedding utilized by the pixel discriminator 222, as shown in FIG. Figure 2 and Figure 4 As described above. In various embodiments, the training model includes a set of weights or embeddings corresponding to the model structure. In embodiments where data 732 is omitted, the machine learning application 730 may include a training model 734 based on prior training (e.g., by the developer of the machine learning application 730, by a third party, etc.). In some embodiments, the training model 734 may include a fixed set of weights (e.g., downloaded from a server that provides weights).

[0170] The machine learning application 730 also includes an inference engine 736. The inference engine 736 is configured to apply the training model 734 to data, such as application data 714 (e.g., video), to provide inferences. In some embodiments, the inference engine 736 may include software code to be executed by the processor 702. In some embodiments, the inference engine 736 may specify a circuit configuration (e.g., for a programmable processor, for a field programmable gate array (FPGA), etc.) that enables the processor 702 to apply the training model. In some embodiments, the inference engine 736 may include software instructions, hardware instructions, or a combination. In some embodiments, the inference engine 736 may provide an application programming interface (API) that can be used by the operating system 708 and / or other applications 712 to call the inference engine 736, for example, to apply the training model 734 to the application data 714 to generate inferences. For example, inferences for a shader model may be to shade an image, while inferences for a feature identifier or a pixel identifier may be to predict a label, as described above with reference to Figure 2-Figure 5 described.

[0171] Machine learning application 730 may provide several technical advantages. For example, when training model 734 is generated based on unsupervised learning, inference engine 736 may apply trained model 734 to generate knowledge representation (e.g., digital representation) from input data (e.g., application data 714). For example, a model trained for image analysis may generate an image representation with a smaller data size (e.g., 1KB) than the input image (e.g., 10MB). In some embodiments, such representations may help reduce processing costs (e.g., computational costs, memory usage, etc.) to generate outputs (e.g., labels, classifications, sentences describing images, colored images from grayscale images, etc.).

[0172] In some embodiments, such a representation can be provided as an input to a different machine learning application that generates an output from the output of the inference engine 736. In some embodiments, the knowledge representation generated by the machine learning application 730 can be provided, for example, over a network, to a different device for further processing. In such embodiments, providing a knowledge representation rather than an image can provide technical advantages, such as enabling faster data transfer at reduced costs. In another example, a model trained to cluster documents can generate document clusters from input documents. The document clusters can be suitable for further processing (e.g., determining whether documents are related to a topic, determining a classification category for documents, etc.) without accessing the original documents, and thus saving computational costs.

[0173] In some embodiments, the machine learning application 730 may be implemented in an offline manner. In these embodiments, the training model 734 may be generated in a first phase and provided as part of the machine learning application 730. In some embodiments, the machine learning application 730 may be implemented in an online manner. For example, in such embodiments, an application that calls the machine learning application 730 (e.g., the operating system 708, one or more of the other applications 712, etc.) may utilize the reasoning generated by the machine learning application 730, for example, provide the reasoning to a user, and may generate a system log (e.g., if permitted by the user, an action taken by the user based on the reasoning; or if used as an input to further processing, a result of further processing). The system log may be generated periodically, for example, hourly, monthly, quarterly, etc., and may be used to update the training model 734, for example, to update the embedding used for the training model 734, with the permission of the user.

[0174] In some embodiments, the machine learning application 730 may be implemented in a manner that may be suitable for a particular configuration of the device 700 on which the machine learning application 730 is executed. For example, the machine learning application 730 may determine a computation graph that utilizes available computing resources, e.g., the processor 702. For example, if the machine learning application 730 is implemented on multiple devices as a distributed application, the machine learning application 730 may determine the computations to be performed on each device in a manner that optimizes the computations. In another example, the machine learning application 730 may determine that the processor 702 includes a GPU with a particular number of GPU cores (e.g., 1000) and implement the inference engine accordingly (e.g., as 1000 individual processes or threads).

[0175] In some embodiments, the machine learning application 730 may implement a collection of training models. For example, the training model 734 may include multiple training models that are each applicable to the same input data. In these embodiments, the machine learning application 730 may select a specific training model based on available computing resources, a success rate with a priori reasoning, etc. In some embodiments, the machine learning application 730 may execute an inference engine 736 so that multiple training models are applied. In these embodiments, the machine learning application 730 may combine the outputs from the application of each model, for example, using a voting technique that scores the outputs from each training model applied or by selecting one or more specific outputs. In addition, in these embodiments, the machine learning application may apply a time threshold (e.g., 0.5ms) for applying each training model and only utilize those individual outputs that are available within the time threshold. Outputs that are not received within the time threshold may not be utilized, for example, discarded. For example, when a time limit is specified while calling a machine learning application, for example, by an operating system 708 or one or more applications 712, such a method may be suitable.

[0176] In different embodiments, the machine learning application 730 can generate different types of outputs. For example, the machine learning application 730 can provide representations or clusters (e.g., digital representations of input data), labels (e.g., for input data including images, documents, etc.), phrases or sentences (e.g., descriptions of images or videos, suitable for use as responses to input sentences, etc.), images (e.g., colored or otherwise stylized images generated by the machine learning application in response to input images (e.g., grayscale images), audio or video (e.g., in response to input videos), when the trained model 734 is trained using training data from comic books, the machine learning application 730 can generate an output video with a specific effect applied, for example, rendered in the style of a comic book or a specific artist. In some embodiments, the machine learning application 730 can generate output based on a format specified by a calling application, such as an operating system 708 or one or more applications 712. In some embodiments, the calling application can be another machine learning application. For example, such a configuration can be used in a generative adversarial network, where the output from the machine learning application 730 is used to train the calling machine learning application and vice versa.

[0177] Any of the software in memory 704 may alternatively be stored in any other suitable storage location or computer-readable medium. In addition, memory 704 (and / or other connected storage devices) may store one or more messages, one or more taxonomies, electronic encyclopedias, dictionaries, thesauruses, knowledge bases, message data, grammars, user preferences, and / or other instructions and data used in the features described herein. Memory 704 and any other type of storage (disk, optical disk, tape, or other tangible medium) may be considered "storage" or "storage devices".

[0178] The I / O interface 706 may provide functionality for enabling the server device 700 to be docked with other systems and devices. A docking device may be included as part of the device 700 or may be separate and communicate with the device 700. For example, a network communication device, a storage device (e.g., memory and / or database 106), and an input / output device may communicate via the I / O interface 706. In some embodiments, the I / O interface may be connected to an interface device such as an input device (keyboard, pointing device, touch screen, microphone, camera, scanner, sensor, etc.) and / or an output device (display device, speaker device, printer, motor, etc.).

[0179] Some examples of docking devices that can be connected to I / O interface 706 may include one or more display devices 720, which can be used for display content, such as images, videos, and / or user interfaces of output applications as described herein. Display device 720 can be connected to device 700 via local connection (e.g., display bus) and / or via networking connection and can be any suitable display device. Display device 720 may include any suitable display device, such as LCD, LED or plasma display screen, CRT, television, monitor, touch screen, 3-D display screen or other visual display device. For example, display device 720 can be a flat panel display screen provided on a mobile device, a plurality of display screens provided in goggles or head-mounted headset devices, or a monitor screen for a computer device.

[0180] The I / O interface 706 can interface with other input and output devices. Some examples include one or more cameras that can capture images. Some embodiments may provide a microphone for capturing sound (e.g., as part of a captured image, voice command, etc.), an audio speaker device for outputting sound, or other input and output devices.

[0181] For ease of illustration, Figure 7 A box for each of processor 702, memory 704, I / O interface 706, and software box operating system 708, other applications 712, and machine learning application 730 is shown. These boxes can represent one or more processors or processing circuits, operating systems, memories, I / O interfaces, applications, and / or software modules. In other embodiments, device 700 may not have all of the components shown and / or may have other elements including other types of elements instead of those shown herein or in addition to those shown herein. Although some components are described as performing the boxes and operations described in some embodiments herein, any suitable component or combination of components of environment 100, device 700, similar systems, or any suitable one or more processors associated with such systems can perform the described boxes and operations.

[0182] The methods described herein can be implemented by computer program instructions or codes that can be executed on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and can be stored on a computer program product including a non-transitory computer-readable medium (e.g., storage medium), such as magnetic, optical, electromagnetic, or semiconductor storage media, including semiconductor or solid-state memory, tape, removable computer disk, random access memory (RAM), read-only memory (ROM), flash memory, hard disk, optical disk, solid-state memory drive, etc. The program instructions can also be included in an electronic signal and provided as an electronic signal, for example, in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented with hardware (logic gates, etc.) or with a combination of hardware and software. Example hardware can be a programmable processor (e.g., a field programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), etc. One or more methods can be executed as a part or component of an application running on a system, or as an application or software running in combination with other applications and an operating system.

[0183] Although the specification has been described with respect to specific embodiments thereof, these specific embodiments are merely illustrative and not restrictive. The concepts illustrated in the examples may be applied to other examples and embodiments.

[0184] In certain embodiments discussed herein, where personal information about a user may be collected or used (e.g., user data, information about a user's social network, a user's location and time at that location, a user's biometric information, a user's activities and demographic information), one or more opportunities are provided to the user to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information, particularly after receiving explicit authorization from the relevant user to do so. For example, the user is provided with control over whether a program or feature collects user information about the particular user or other users associated with the program or feature. One or more options are presented to each user for whom personal information is to be collected to allow control over the collection of information related to the user to provide permission or authorization as to whether information is collected and which parts of the information are to be collected. For example, one or more such control options may be provided to the user via a communication network. In addition, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. As an example, the identity of the user may be processed so that personally identifiable information cannot be determined. As another example, the user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

[0185] It is noted that, as will be known to those skilled in the art, the functional blocks, operations, features, methods, devices and systems described in this disclosure may be integrated or divided into different combinations of systems, devices and functional blocks. Any suitable programming language and programming techniques may be used to implement the routines of a particular embodiment. Different programming techniques may be employed, such as procedural or object-oriented. The routines may be executed on a single processing device or multiple processors. Although steps, operations or calculations may be presented in a specific order, the order may be changed in different specific embodiments. In some embodiments, multiple steps or operations shown as being in order in this specification may be performed simultaneously.

Claims

1. A computer-implemented method for training a convolutional neural network to colorize a grayscale image, the method comprising: providing training data, the training data comprising a plurality of grayscale images as input to the neural network and a plurality of training color images each depicting one or more persons, the plurality of training color images corresponding to the plurality of grayscale training images and part annotations for the one or more persons in each training color image; as well as For each grayscale training image, Obtaining an intermediate feature vector describing an intermediate colored image from a latent layer of the convolutional neural network based on the grayscale training image and the intermediate part annotations for the one or more persons in the grayscale training image; as well as Based on the grayscale training image and the part annotations for the one or more persons in the grayscale training image, obtaining an output feature vector describing a colored image as an output of the convolutional neural network; obtaining a perceptual loss value from a pixel discriminator, wherein the pixel discriminator generates the perceptual loss value by comparing the colorized image to a training color image of the plurality of training color images that corresponds to the grayscale image, wherein the output feature vector and the part annotation are provided as inputs to the pixel discriminator; obtaining a feature loss value from a feature discriminator based on a prediction by the feature discriminator of whether there is a ground truth color image corresponding to the intermediate shaded image, wherein the intermediate feature vector is provided as an input to the feature discriminator; and Parameters of the convolutional neural network are modified based on the perceptual loss value and the feature loss value.

2. The computer-implemented method of claim 1 , further comprising: resizing the intermediate shaded image; Obtaining a network output image from an output layer of the convolutional neural network; as well as The resized intermediate shaded image and the network output image are combined to obtain the shaded image.

3. The computer-implemented method of claim 1 , further comprising, for each grayscale training image, obtaining a saturation loss from a saturation enhancer, and wherein, Modifying the parameters of the convolutional neural network is further based on the saturation loss.

4. The computer-implemented method of claim 3, wherein: Obtaining the saturation loss includes: identifying one or more background regions of the colored image based on the region annotations; and For pixels identified as being in the background region, the saturation loss is calculated based on the saturation of the shaded image and the brightness value of the shaded image.

5. The computer-implemented method of claim 1 , further comprising, for each grayscale training image, obtaining a skin tone loss from a skin tone enhancer, and wherein, Modifying the parameters of the convolutional neural network is further based on the skin tone loss.

6. The computer-implemented method of claim 5, wherein: Obtaining the skin tone loss includes: identifying one or more silhouettes in the colored image based on the part annotations; and For each pixel of the rendered image, the skin tone loss is calculated based on the hue of the rendered image and whether the pixel is in at least one of the one or more silhouettes in the rendered image.

7. A computer-implemented method for colorizing a grayscale image depicting a person, the method comprising: receiving the grayscale image; as well as by providing the grayscale image as an input to a trained convolutional neural network (CNN), generating a colored image based on the grayscale image as an output of the trained CNN, wherein the trained CNN is trained according to the method of any one of claims 1 to 6, wherein the trained CNN performs part segmentation to detect one or more parts of the person and colorizes the grayscale image, The grayscale image is a photo captured by a camera.

8. The computer-implemented method of claim 7, wherein: Generating the colored image includes: Obtaining an intermediate colored image as an output of a latent layer of the trained CNN; Obtaining a network output image as an output of an output layer of the trained CNN; and The intermediate shaded image and the network output image are combined to obtain the shaded image.

9. The computer-implemented method of claim 8, wherein: The intermediate shading image has a lower resolution than the network output image, and the method further comprises, prior to the combining, upsampling the intermediate shading image to the same resolution as the network output image.

10. The computer-implemented method of claim 7, further comprising downsampling the grayscale image before providing the grayscale image as input to the trained CNN, or wherein, The rendered image is an sRGB image.

11. The computer-implemented method of claim 7, wherein: The colored image has a lower resolution than the grayscale image, and the method further comprises: Upsampling the rendered image to the same resolution as the grayscale image; and The shaded image is combined with the grayscale image to obtain an output image.

12. A system for coloring a grayscale image, comprising: A convolutional neural network, the convolutional neural network being implemented by one or more processors, the convolutional neural network having a plurality of layers and being configured to perform operations including: receiving the grayscale image depicting one or more persons; as well as generating a colored image based on the grayscale image, wherein the colored image includes part annotations of the one or more persons in the grayscale image; and a pixel discriminator implemented by the one or more processors, the pixel discriminator being configured to perform operations including: receiving the rendered image; and generating a label prediction indicating whether the colorized image is an original color image or a predicted color image; The operations performed by the convolutional neural network further include: generating an intermediate colored image based on the grayscale image at a latent layer of the convolutional neural network, wherein the intermediate colored image includes annotations of intermediate parts of the one or more persons in the grayscale image, The system further comprises: A feature identifier, the feature identifier being implemented by the one or more processors, the feature identifier being configured to perform operations including: receiving the intermediate rendered image; and A prediction is generated indicating whether a ground truth color image exists corresponding to the grayscale image.

13. The system according to claim 12, wherein: The intermediate shaded image has a resolution lower than a resolution of the grayscale image, and wherein generating the shaded image comprises: resizing the intermediate shaded image to the resolution of the grayscale image; and After the resizing, the intermediate colored image is combined with a network output image of an output layer of the convolutional neural network to obtain the colored image.

14. The system according to claim 12, wherein: The predictions of the feature discriminator are used to train the convolutional neural network by adjusting one or more parameters of the convolutional neural network.

15. The system of claim 12, wherein: The pixel discriminator is trained by providing as input a plurality of ground truth color images having associated ground truth labels indicating that the images are ground truth color images, and adjusting one or more parameters of the pixel discriminator using a loss based on the difference between the label prediction for each input image and the ground truth label.

16. The system of claim 12, wherein: The pixel discriminator is trained based on a loss comparing the label prediction to a ground truth label indicating whether the colorized image is the original color image or the predicted color image.

17. The system of claim 12, wherein: The label predictions of the pixel discriminator are used to train the convolutional neural network by adjusting one or more parameters of the convolutional neural network.

Images