Generate colorized images based on interactive color edges using colorization neural network

Abstract

Embodiments of the present disclosure relate to generating a colorized image based on interactive color edges using a colorization neural network. Methods, systems, and non-transitory computer-readable storage media for converting a grayscale digital image into a colorized digital image using an edge prediction neural network and an edge-guided colorization neural network are disclosed. In one or more embodiments, the disclosed system applies a color edge prediction neural network to a grayscale image to generate a color edge map indicating predicted chromaticity edges. The disclosed system may present the color edge map to a user via a colorization graphical user interface, and receive user color points and color edge modifications. The disclosed system may apply a second neural network, an edge-guided colorization neural network, to the color edge map or a modified edge map, the user color points, and the grayscale image to generate an edge-constrained colorized digital image.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of digital images, and more particularly to generating digital images reflecting selected colors. Background Art

[0002] In recent years, significant improvements have been achieved in hardware and software platforms for editing and generating digital images. For example, developers have created technologies for modifying or improving digital images, including photographs and illustrations. To illustrate, a conventional image colorizing system can automatically add color to a complex black and white or grayscale image based on a user selection of a particular fill color. For example, a client device can select a color corresponding to a particular area of ​​a grayscale image, and the image colorizing system can generate a digital image reflecting the selected color.

[0003] Although conventional image colorization systems can add color to digital images, they have several technical disadvantages. Specifically, conventional image editing systems are often inaccurate, inflexible, and inefficient when applying color to digital images. For example, conventional image colorization systems often add color that results in an unbelievable, unrealistic digital image. More specifically, conventional systems often fail to accurately determine the desired color zone for a particular area of ​​an image and inaccurately generate an unbelievable resulting digital image. In fact, images colored by conventional image colorization systems often suffer from the problem of color bleeding between segments of uncertainty.

[0004] Additionally, conventional image colorization systems are oftentimes inflexible. In particular, conventional image colorization systems rigidly generate a colorized digital image that is difficult to modify or adjust. For example, after generating the colorized digital image, conventional image colorization systems are oftentimes inflexible in that they are unable to flexibly adjust the colorized digital image to reflect different color regions (without extensive user input). More specifically, conventional systems oftentimes include algorithms that rigidly map a single black and white image into a single set of fill boundaries of a color image. Thus, conventional image colorization systems are oftentimes inflexible and fail to provide smooth options for modifying the colorized digital image and the colorized regions.

[0005] In addition, conventional image colorization systems are often inefficient. In particular, conventional systems must often invest additional computing resources to correct for bleeding or other inaccuracies. For example, conventional systems often require a large amount of user interaction and corresponding memory and processing power to modify digital images in incorrectly colored areas. Therefore, conventional image colorization systems are often required to reanalyze pixels or areas and adjust the desired colorization. Even so, by adding color in one segment, conventional systems can often incorrectly color other segments. Alternatively, many conventional systems invest additional computing and time resources by requiring the user to explicitly segment the desired area and perform flood fill.

[0006] These and other disadvantages exist relative to conventional image colorization systems. Summary of the invention

[0007] One or more embodiments provide benefits and / or solve one or more of the aforementioned problems or other problems in the art using systems, methods, and non-transitory computer-readable storage media that utilize a colorization neural network to efficiently, accurately, and flexibly perform user-guided grayscale image colorization. In particular, the disclosed system can incorporate color edges as inputs to help guide the colorization neural network to accurately propagate color inputs. To illustrate, the disclosed system can utilize a color edge prediction neural network to generate and present user-selectable color edges. The disclosed system can flexibly add or remove color edges and then utilize a colorization neural network to analyze the color edges and selected color points to generate a colorized image. By using a color edge prediction neural network to generate interactive color edges that can be utilized with the colorization neural network, the disclosed system can efficiently and flexibly generate accurate colorized digital images.

[0008] Additional features and advantages of one or more embodiments of the present disclosure will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of such example embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The various embodiments will be described and explained with additional specificity and detail through use of the drawings that are summarized below.

[0010] Figure 1 An environment is illustrated in which an edge-guided colorization system may operate according to one or more embodiments of the present disclosure.

[0011] FIG. 2A to FIG. 2B Illustrated are example colorized images generated by a conventional system and an edge-guided colorization system in accordance with one or more embodiments of the present disclosure.

[0012] Figure 3 An overview for generating a colorized image according to one or more embodiments of the present disclosure is illustrated.

[0013] FIG. 4A to FIG. 4B An edge-guided colorization system is illustrated for training and applying a color edge prediction neural network to generate a color edge map in accordance with one or more embodiments of the present disclosure.

[0014] Figure 5 An overview of an edge-guided colorization system for generating a modified edge map from a color edge map according to one or more embodiments of the present disclosure is illustrated.

[0015] FIG. 6A to FIG. 6B An edge-guided colorization system for training and applying an edge-guided colorization neural network according to one or more embodiments of the present disclosure is illustrated.

[0016] FIG. 7A to FIG. 7B An example colored graphical user interface is illustrated in accordance with one or more embodiments of the present disclosure.

[0017] Figure 8 A schematic diagram illustrating an example architecture of an edge-guided colorization system according to one or more embodiments of the present disclosure is illustrated.

[0018] Fig. 9 A series of actions for generating a colorized image according to one or more embodiments of the present disclosure are illustrated.

[0019] Fig.10 A block diagram of an example computing device is illustrated in accordance with one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] One or more embodiments of the present disclosure include an edge-guided colorization system that utilizes a deep colorization neural network to efficiently and accurately generate a colorized image using user-guided color edges. For example, the edge-guided colorization system can utilize a two-stage approach to generate a colorized digital image. In the first stage, the edge-guided colorization system can utilize an edge prediction neural network to analyze a grayscale image and generate an interactive edge map including a set of color edges. In particular, the edge-guided colorization system can present the interactive edge map via a graphical user interface that includes various dynamic elements for modifying color edges (such as a global slider element that modifies the number of color edges, or a suggestion element that recommends additional color edges). In the second stage, the edge-guided colorization system can utilize an edge-guided colorization neural network to analyze the edge map to generate a colorized digital image. In particular, the edge-guided colorization system can utilize the selected color edges together with the grayscale image and the selected color points as channel inputs to the edge-guided colorization neural network to generate a colorized digital image. By predicting and rendering interactive color edges using a color edge prediction neural network and then combining the interactive color edges with an edge-guided colorization neural network, the edge-guided colorization system can flexibly and efficiently generate accurate colorized digital images.

[0021] To illustrate, in one or more embodiments, an edge-guided colorization system utilizes a color edge prediction neural network to generate a color edge map including color edges for a grayscale digital image. The edge-guided colorization system may provide a colorized graphical user interface for display to a client device, the colorized graphical user interface including the color edge map, a color point input element, and the grayscale digital image. The edge-guided colorization system may generate a colorized image using the edge-guided colorization neural network based on the grayscale digital image, the color edge map, and a user interaction with the color point input element. The edge-guided colorization system may then provide the colorized image for display to a user via the colorized graphical user interface.

[0022] As mentioned, the edge-guided colorization system can train and apply a color edge prediction neural network to generate a color edge map. More specifically, the edge-guided colorization system can train a color edge prediction neural network based on comparing predicted color edges to ground truth color edges. In particular, the edge-guided colorization system can utilize the Canny edge detection algorithm to generate ground truth edges from a chrominance image. The edge-guided colorization system can then apply the color edge prediction neural network to a corresponding grayscale image to predict color edges, and then train the edge-guided colorization system by comparing the predicted color edges to the ground truth edges.

[0023] Once trained, the edge-guided colorization system can use the color edge prediction neural network to predict the color edges of the grayscale digital image. More specifically, the edge-guided colorization system can use the color edge prediction neural network to generate initial edges and determine a confidence score for each of the initial edges. The edge-guided colorization system can identify initial edges with confidence scores that meet a color edge threshold and generate a color edge map using the identified edges.

[0024] As discussed, the edge-guided colorization system can provide a colorized graphical user interface for display at a client device, the colorized graphical user interface including a color edge map, a color point input element, and a grayscale digital image. The edge-guided colorization system can utilize the colorized graphical user interface to collect input for the edge-guided colorization neural network. In particular, the edge-guided colorization system can receive a user color point via the color point input element, the user color point identifying a location of the digital image that corresponds to a particular color.

[0025] As mentioned above, in some embodiments, the edge-guided colorization system also provides a dynamic user interface element to collect user edge guidance and modify color edges. In some embodiments, the edge-guided colorization system can provide a color edge threshold element, an edge suggestion element, and / or a user-drawn edge element. For example, based on interaction with a color edge threshold element (e.g., a slider or some other type of element), the edge-guided colorization system can modify the color edge threshold utilized by the color edge prediction neural network when generating color edges. Similarly, based on user interaction with the edge suggestion element, the edge-guided colorization system can apply the Canny edge detection algorithm to identify the brightness edges of the grayscale image to make suggestions to the client device. Moreover, based on user interaction with the user-drawn edge element, the edge-guided colorization system can identify the edges drawn by the user via the client device.

[0026] In addition, the edge-guided colorization system can also train and utilize the edge-guided colorization neural network to generate a colorized image. The edge-guided colorization system can use the training data to train the edge-guided colorization neural network. More specifically, the edge-guided colorization system can use the training grayscale image, the training color points, and the training color edges extracted from the ground-truth color image as training inputs of the edge-guided colorization neural network to generate a predicted colorized image. The edge-guided colorization system can train the edge-guided colorization neural network by comparing the predicted colorized image with the ground-truth colorized image, and adjusting the network parameters to minimize the loss.

[0027] Once trained, the edge-guided colorization system can apply the edge-guided colorization neural network to a target grayscale digital image to generate a colorized image. For example, in some embodiments, the edge-guided colorization system utilizes the target grayscale image, the user-selected color points, and the color edge map as input channels of the edge-guided colorization neural network to generate a colorized digital image. In some embodiments, the edge-guided colorization system directly cascades the edge map into the feature maps analyzed at various layers of the edge-guided colorization neural network to further incorporate edge guidance when predicting the color of the grayscale image. Thus, the edge-guided colorization neural network can generate a colorized image, and the edge-guided colorization system can present the colorized image to the user via a colorization graphical user interface.

[0028] The edge-guided colorization system provides several technical benefits relative to conventional systems. For example, the edge-guided colorization system can improve accuracy relative to conventional systems. In particular, the edge-guided colorization system can reduce color bleeding by utilizing a color edge prediction neural network to generate an interactive color edge that can be used in conjunction with the edge-guided colorization neural network. In fact, by utilizing a color edge prediction neural network to generate a color edge map, the edge-guided colorization system more accurately defines the chromaticity edges of the colorized image. In addition, the edge-guided colorization system can improve the accuracy of the predicted color edge map by receiving user edge guidance regarding the predicted color edges. For example, the edge-guided colorization system can improve accuracy by generating a modified color edge map based on user interaction with the interactive color edge.

[0029] The edge-guided colorization system can also provide increased flexibility relative to conventional systems. In fact, by providing various edge modification interface elements, the edge-guided colorization system can generate several different colorized images from a single grayscale digital image. For example, the edge-guided colorization system can generate colorized images with different numbers of chromaticity edges by modifying the edge thresholds to include more or fewer color edges within the color edge map. Similarly, by suggesting additional color edges and allowing the user to modify the color edges, the edge-guided colorization system can flexibly generate colorized images and provide various options to modify edges or color regions in a smooth manner.

[0030] Additionally, the edge-guided colorization system can improve efficiency relative to conventional systems. More specifically, the edge-guided image colorization system can reduce the number of interactions and corresponding resources required to generate an accurate colorized image. For example, by performing a small number of interactions with edge modification interface elements, the edge-guided colorization system can modify color edges and efficiently correct errors in the resulting digital image. In fact, the edge-guided colorization system can generate an edge map, suggest additional edges, and quickly add or remove edges, thereby generating a colorized digital image with minimal user input.

[0031] As illustrated by the foregoing discussion, the present disclosure utilizes various terms to describe the features and advantages of edge-guided colorization systems. Additional details about the meaning of such terms are now provided. For example, as used herein, the term "grayscale digital image" refers to a digital image that includes pixels that reflect shades of gray. In particular, a grayscale digital image may include an image in which each pixel has an intermediate tone gradation value between white and black. More specifically, each pixel in a grayscale digital image may represent a brightness or intensity value (e.g., not a color or hue). For illustration, a grayscale image may include pixels having values ​​in the L or brightness channel in the CIE Lab color space.

[0032] As used herein, the term "color edge map" refers to a collection of color edges. In particular, a color edge map may include an array, an image, or other digital representations of the locations of color edges within an indication digital image. For example, a color edge map may include a binary image having values ​​indicating the locations of color edges or chromaticity edge boundaries of a corresponding target digital image. In addition, a color edge map may include detected edges within a chromaticity map. In contrast to a grayscale digital image, a chromaticity map is an image whose pixels reflect color values. For example, a chromaticity map may include a and / or b channels in a CIE Lab color space.

[0033] As used herein, the term "neural network" refers to a machine learning model that can be trained to roughly estimate an unknown function based on training inputs. In particular, the term "neural network" may include a model of interconnected digital neurons that communicate and learn to roughly estimate complex functions and generate outputs based on multiple inputs provided to the model. For example, the term "neural network" includes one or more machine learning algorithms, including convolutional neural networks, recursive neural networks (e.g., LSTM), generative neural networks, adversarial neural networks, support vector machines, feedforward neural networks, autoencoders, or variational autoencoders. As described in more detail below, an edge-guided colorization system can utilize a color edge prediction neural network (i.e., a neural network for predicting color edges) and an edge-guided colorization neural network (i.e., a neural network for generating a colored digital image based on color edges).

[0034] As used herein, the term "color edge" refers to a boundary between areas of different colors. In particular, a color edge refers to a chromaticity boundary or an area where chromaticity values ​​change within a digital image. A color edge indicates a difference in color values ​​within a digital image. For example, the term "color edge" refers to a line that indicates a boundary between areas with different color values ​​within a digital image.

[0035] As used herein, the term "color point input element" refers to a user interface element for receiving color input. More particularly, the term "color point input element" refers to a graphical user interface element for user input of a point of a digital image that corresponds to a particular color. For example, the color point input element may include a color gamut that provides the entire range of colors available for a particular pixel of a digital image.

[0036] As used herein, the term "colorized image" refers to a digital image that includes pixels having colors or color values. In particular, the term "colorized image" refers to an image to which colors (e.g., RBG values) have been added. For example, the term "colorized image" may refer to an image having pixel values ​​in a and / or b channels in the CIE Lab color space. More specifically, a colorized image includes pixels having brightness and color values.

[0037] Additional details regarding the edge-guided colorization system will now be provided with respect to illustrative figures depicting example embodiments and implementations of the edge-guided colorization system. Figure 1 1 shows a schematic diagram of a system environment (or "environment") 100 in which an edge-guided colorization system 106 may operate according to one or more embodiments. As shown, the environment 100 includes one or more server devices 102 connected to user client devices 108 via a network 112. Figure 1An embodiment of an edge-guided colorization system 106 is shown, but alternative embodiments and configurations are possible.

[0038] like Figure 1 As shown in FIG. 1 , server device(s) 102 and user client device 108 are connected via network 112. Each of the components of environment 100 can communicate via network 112. Network 112 includes any suitable network over which computing devices can communicate. Figure 1 An example network is discussed with additional detail.

[0039] As shown, the environment 100 includes (multiple) server devices 102. The (multiple) server devices 102 can generate, store, receive and transmit electronic data, such as digital video, digital images, digital audio, metadata, etc. The (multiple) server devices 102 can receive data from user client devices 108. For example, the (multiple) server devices 102 can collect and / or receive digital images, including grayscale digital images, metadata, color points, color edges, color edge modifications, and other data. The (multiple) server devices 102 can communicate with the user client devices 108 via the network 112. For example, the (multiple) server devices 102 can send digital images including colorized images, color edge maps, and other data to the user client devices 108. In some embodiments, the (multiple) server devices 102 include distributed servers, wherein the (multiple) server devices 102 include multiple server devices distributed across the network 112 and located in different physical locations. The (multiple) server devices 120 may include content servers, application servers, communication servers, web hosting servers, or digital content management servers.

[0040] like Figure 1 As further shown in , (multiple) server devices 102 may include an image editing system 104. The image editing system 104 may create, modify, edit, share, distribute and / or manage digital images. For example, the image editing system 104 stores a repository of grayscale digital images and colorized images received from a user client device 108 and one or more other user client devices. Additionally, the image editing system 104 may access a database storing ground truth images including color digital images and corresponding grayscale images. The image editing system 104 may receive digital images, color points, color edge modifications and other data from the user client device 108. In addition, the image editing system 104 may distribute the colorized images to the user client device 108 (e.g., via an application or website accessed by the user client device 108).

[0041] like Figure 1As shown in , the edge-guided colorization system 106 can be implemented as part of the image editing system 104. The edge-guided colorization system 106 can receive a grayscale digital image and generate a colorized image. More specifically, the edge-guided colorization system 106 can implement a color edge prediction neural network for predicting color edges and an edge-guided colorization neural network for coloring an image in conjunction with edge constraints. The edge-guided colorization system 106 can communicate with a user client device 108. The edge-guided colorization system 106 can receive a grayscale digital image, a color point, and a color edge modification. Additionally, the edge-guided colorization system 106 can distribute the colorized image (e.g., via a network 112).

[0042] like Figure 1 , environment 100 includes a user client device 108. The user client device 108 can generate, store, receive, and transmit digital data. In particular, the user client device 108 can transmit digital data including digital images to the server device(s) 102 via the network 112. The user client device 108 can present the digital images to a user associated with the user client device 108.

[0043] Figure 1 The user client device 108 illustrated in the figure may include various types of client devices. For example, in some embodiments, the user client device 108 includes a mobile device, such as a laptop, a tablet computer, a mobile phone, a smart phone, etc. In other embodiments, the user client device 108 includes a non-mobile device, such as a desktop computer or a server or other type of client device. Figure 1 Additional details regarding user client device 108 are discussed.

[0044] like Figure 1 As further illustrated in FIG. 1 , the user client device 108 includes an application 110. The application 110 may be a web application or a native application (e.g., a mobile application, a desktop application, etc.) on the user client device 108. The application 110 may interface with the edge-directed colorization system 106 to provide digital content including digital images to the server device(s) 102. In practice, the application 110 may receive data from the edge-directed colorization system 106 and may render a colorized image generated by the edge-directed colorization system 106 for display at the user client device 108.

[0045] although Figure 1The edge-guided colorization system 106 is depicted as being located on the server device(s) 102, but in some embodiments, the edge-guided colorization system 106 may be implemented on (e.g., completely or partially located on) one or more other components of the environment 100. For example, the edge-guided colorization system 106 may be implemented completely (or partially) on the user client device 108. For example, the server device(s) 102 and / or the user client device 108 may have digital images stored thereon.

[0046] Additionally, the user client device 108 can communicate directly with the edge-guided colorization system 106, bypassing the network 112. Moreover, the edge-guided colorization system 106 can access one or more databases (e.g., digital document databases) housed on the server device(s) 102 or elsewhere in the environment 100. Further, the edge-guided colorization system 106 can include one or more machine learning models (e.g., neural networks), and the edge-guided colorization system 106 can be implemented across the server device(s) 102, the network 112, and the user client device 108 in a variety of different ways.

[0047] As previously mentioned, the edge-guided colorization system 106 can generate a colorized image. For example, FIG. 2A to FIG. 2B An edge-guided colorization system 106 is illustrated for generating a colorized image according to one or more embodiments. In particular, FIG. 2A to FIG. 2B One or more embodiments of the edge-guided colorization system 106 are contrasted with conventional systems. For example, Figure 2A A conventional color image 202 including a conventional color boundary 206 is illustrated. Figure 2B A colorized image 204 generated by the edge-guided colorization system 106 and including a color edge 208 is illustrated.

[0048] Figure 2A The diagram shows how conventional systems often produce color images that suffer from bleeding. Bleeding occurs when colors do not stay within exact boundaries. Figure 2A, so the effect is illustrated using dashed lines, which show the color boundaries that delineate different shading areas. As illustrated, some conventional systems map grayscale images to conventional color images 202. However, conventional systems often fail to predict the correct color for uncertainty areas and fill uncertainty areas with incorrect colors. Conventional color image 202 includes conventional color boundaries 206 that delineate pixel color values. For example, pixels within the area defined by conventional color boundaries 206 share the same color value. Additionally, areas on either side of conventional color boundaries 206 contain pixels with different color values. As illustrated, conventional color boundaries 206 of conventional color image 202 are inaccurate and fail to follow the outline of a flamingo or other object.

[0049] Figure 2B A colorized image 204 generated by the edge-guided colorization system 106 is illustrated. As mentioned above, the edge-guided colorization system 106 incorporates edge guidance to more accurately colorize an image. The edge-guided colorization system 106 can colorize an image based on predicted color edges and modified color edges. By incorporating edge guidance, the edge-guided colorization system 106 relies on defined color edges to more accurately resolve regions of uncertainty. The colorized image 204 includes a color edge 208. The color edge 208 is more accurate than the color edge 208. Figure 2A . In particular, pixels within a region bounded by a color edge 208 have the same color value. For example, the color of a flamingo is pink, the color of water is turquoise, and the color of the sky is blue. As illustrated, the edge-guided colorization system 106 generates a colorized image 204 that is more accurate than a conventional color image 202 generated by a conventional system. The following figures and accompanying discussion provide additional details on how the edge-guided colorization system 106 accurately generates a colorized image and incorporates edge guidance.

[0050] For example, Figure 3 An edge-guided colorization system 106 is illustrated that generates a colorized image using a two-stage approach according to one or more embodiments. In a first stage, the edge-guided colorization system 106 generates a color edge map using a color edge prediction neural network. In a second stage, the edge-guided colorization system 106 generates a colorized image using an edge-guided colorization neural network.

[0051] As illustrated, the edge-guided colorization system 106 receives a grayscale digital image 302. The edge-guided colorization system 106 analyzes the grayscale digital image 302 using a color edge prediction neural network 304. For example, the color edge prediction neural network 304 may include a deep convolutional neural network having multiple layers. At various layers of the convolutional neural network, the color edge prediction neural network 304 may generate feature maps (e.g., feature vectors) that allow the network to analyze different levels of abstraction. By analyzing these feature maps, the color edge prediction neural network 304 predicts a color edge map 306 based on the grayscale digital image 302. For example, in some embodiments, the color edge prediction neural network 304 utilizes the neural network architecture and methods described in "Colorful Image Colorization" by Richard Zhang, Phillip Isola, and Alexei A. Efros, ECCV 2016, 649-666 (2016), and the neural network architecture and methods described in "Real-Time User-Guided Image Colorization with Learned Deep Priors" by Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros, 2017 Computer Graphics Specialty Group Conference, Preprint Library: 1705:02999, the above documents are incorporated herein in their entirety.

[0052] Relative to Figure 3 , the color edge prediction neural network 304 generates a color edge map 306 using the color edge threshold. The edge-guided colorization system 106 utilizes the color edge prediction neural network 304 to generate an initial color edge map including the initial edges. As part of generating the initial color edge map, the color edge prediction neural network 304 determines a confidence score for each of the initial edges. The edge-guided colorization system 106 determines the color edge threshold. In one or more embodiments, the edge-guided colorization system 106 utilizes a predetermined color edge threshold. In addition, the edge-guided colorization system 106 may receive the color edge threshold from a user. Based on determining that the confidence score associated with the initial edge satisfies the color edge threshold, the edge-guided colorization system 106 includes the initial edge as a color edge in the color edge map 306. The edge-guided colorization system 106 excludes the initial edges having a confidence score below the color edge threshold from the color edge map 306. Figure 5 FIG. 6 and the accompanying discussion provide additional details regarding color edge thresholds.

[0053] like Figure 3 , the edge-guided colorization system 106 presents a color edge map 306 to the user. In at least one embodiment, the edge-guided colorization system 106 receives user input indicating one or more color edge modifications 310. For example, with respect to Figure 3 The edge-guided colorization system 106 receives the color edge modifications 310 via the color edge threshold component 318, the edge suggestion component 320, and / or the user-drawn edge component 322. Based on the color edge modifications 310, the edge-guided colorization system 106 generates a modified edge map 324.

[0054] For example, the edge-guided colorization system 106 may detect a user interaction with the color edge threshold element 318 and modify the color edge threshold. Based on the modified edge threshold, the edge-guided colorization system 106 includes a modified set of color edges in the modified edge map 324. For example, if the edge-guided colorization system 106 utilizes a modified edge threshold that is above the color edge threshold, the edge-guided colorization system 106 reduces the number of color edges in the color edge map 306 by removing color edges whose confidence scores are below the modified edge threshold. Thus, the modified set of color edges of the modified edge map 324 includes fewer color edges than the color edge map 306. Similarly, when the edge-guided colorization system 106 achieves a modified edge threshold that is below the color edge threshold, the edge-guided colorization system 106 increases the number of edges in the modified set of color edges.

[0055] As illustrated, the edge-guided colorization system 106 may also receive the color edge modification 310 via the edge suggestion element 320. As mentioned, the edge-guided colorization system 106 may present the suggested edges to the user. In at least one embodiment, the edge-guided colorization system 106 presents the luminance edges as the suggested edges via the colorization graphical user interface. For example, the edge-guided colorization system 106 may apply the Canny edge detection algorithm to the grayscale digital image 302 to generate predicted luminance edges. The edge-guided colorization system 106 may then present the predicted luminance edges as the suggested edges. Based on the user selection of one or more of the luminance edges, the edge-guided colorization system 106 includes the selected edges in the modified edge map 324. Figure 5 , and the accompanying discussion provide additional details regarding the edge proposal component 320 .

[0056] Additionally, the edge-guided colorization system 106 can detect user interactions with the user-drawn edge elements 322 to determine the color edge modification 310. In at least one embodiment, the edge-guided colorization system 106 presents the user-drawn edge elements 322 to the user. The user can manually input additional user-drawn edges via the user-drawn edge elements 322. The edge-guided colorization system 106 includes the user-drawn edges in the modified edge map 324. Figure 5 and FIG. 7A to FIG. 7B and the accompanying discussion provide additional details regarding user-drawn edge elements 322 .

[0057] like Figure 3 As further illustrated in FIG. 3 , the edge-guided colorization system 106 utilizes the edge-guided colorization neural network 314 to generate a colorized image 316. The edge-guided colorization system 106 utilizes the modified edge map 324, the user color points 312, and the grayscale digital image 302 as inputs to the edge-guided colorization neural network 314. For example, the edge-guided colorization system 106 can generate a grayscale channel reflecting the grayscale digital image 302 (e.g., a vector representation of pixels of the grayscale digital image 302). Similarly, the edge-guided colorization system 106 can generate a color point channel reflecting the user color points 312 (e.g., a vector representation of pixel locations and user-selected colors). Moreover, the edge-guided colorization system can generate an edge map channel reflecting the modified edge map 324 (e.g., a vector representation of pixels corresponding to color edges). The edge-guided colorization system 106 can analyze these input channels via the edge-guided colorization neural network 314 to generate the colorized image 316.

[0058] like Figure 3 , the edge-guided colorization system 106 receives a user color point 312 via a colorization graphical user interface. The user color point 312 includes user-indicated color samples at various locations within a colorized image 316. As discussed above, the edge-guided colorization system 106 can utilize an edge-guided colorization neural network 314 to analyze the color point 312 to generate a colorized image that accurately reflects the selected color point.

[0059] As discussed above, the edge-guided colorization neural network 106 may include a deep convolutional neural network with multiple layers. These intermediate layers can generate and analyze feature maps at different abstraction levels to generate a predicted colored image. For example, different convolutional layers can generate feature vectors of different dimensions, and these potential vectors are fed to additional convolutional layers. At each layer, the learned weighting parameters can emphasize different features that are important for generating accurate predicted outputs. In some embodiments, the edge-guided colorization system 106 connects the modified edge map 324 to all feature maps (or a subset thereof) analyzed at the intermediate layers of the edge-guided colorization neural network 314. By emphasizing the edge map at multiple intermediate layers of the edge-guided colorization neural network 314, the edge-guided colorization system 106 can reduce bleeding and improve the accuracy of the colored image 316. FIG. 6A to FIG. 6B , and the accompanying discussion further describe how the edge-guided colorization system 106 trains and applies the edge-guided colorization neural network 314 .

[0060] Therefore, if Figure 3 , the edge-guided colorization system 106 utilizes an edge-guided colorization neural network 314 to analyze a modified edge map 324 (or a color edge map 306 if the edge-guided colorization system 106 does not receive a color edge modification 310), a user color point 312, and a grayscale digital image 302. The edge-guided colorization system 106 utilizes the edge-guided colorization neural network 314 to generate a colorized image 316. Because the edge-guided colorization neural network 314 relies on input of color edges, the color edge prediction neural network 304 can be considered a pre-processing step. The color edge prediction neural network 304 improves efficiency because (1) requiring the user to manually input color edges is both time-consuming and computationally resource-intensive and (2) the user may not intuitively understand the concept of "edge" when starting from a blank canvas. Therefore, the color edge prediction neural network 304 provides initialization by predicting the color edge map 306 based on the grayscale digital image 302.

[0061] although Figure 3 The modified edge map 324 is illustrated as being used as an input to the edge-guided colorization neural network 314, but the edge-guided colorization system 106 may also use the color edge map 306 as an input to the edge-guided colorization neural network 314. More specifically, in one or more embodiments, the edge-guided colorization system 106 may automatically generate the colorized image 316 without receiving the color edge modification 310. The edge-guided colorization system 106 may present the colorized image 316 to a user as a preview or as a finished product.

[0062] FIG. 4A to FIG. 4BAdditional details are provided regarding how the edge-guided colorization system 106 trains and applies the color edge prediction neural network 304 according to one or more embodiments of the present disclosure. More specifically, Figure 4A The edge-guided colorization system 106 for training a color edge prediction neural network 304 is illustrated, and Figure 4B Illustrated is the edge-guided colorization system 106 applying a color edge prediction neural network 304 to generate a color edge map 306 including color edges.

[0063] like Figure 4A As illustrated in FIG. 4 , during training, the edge-guided colorization system 106 applies canny edge detection 404 to the chrominance map Y 402 to generate ground truth chrominance edges V 406. Additionally, the edge-guided colorization system 106 uses the ground truth grayscale digital image X 408 as input to the color edge prediction neural network 304 to generate predicted edges V 406. The edge-guided colorization system 106 compares the ground truth chrominance edge V 406 with the predicted edge The comparison is made and a loss function 412 is applied. Based on the loss function 412, the edge-guided colorization system 106 trains parameters of the color edge prediction neural network 304.

[0064] The edge-guided colorization system 106 extracts chromaticity values ​​from the ground truth color image to generate a chromaticity map Y 402. As used herein, the term "chromaticity map" refers to an image that reflects chromaticity values. In particular, the chromaticity map conveys the color information of the image (e.g., compared to the brightness information). For example, Figure 4A The chromaticity map Y 402 represents the chromaticity map of the ground truth color image of a rose.

[0065] During training, the edge-guided colorization system 106 runs Canny edge detection 404 on the chromaticity map Y 402 to generate ground truth chromaticity edges V 406. With respect to FIG. 4, Canny edge detection 404 includes three parts: filtering, non-maximum suppression (NMS), thresholding, and hysteresis. During filtering, the edge-guided colorization system 106 applies a filter to smooth the chromaticity map Y 402 to remove noise. For example, the edge-guided colorization system 106 applies a Gaussian filter. After the edge-guided colorization system 106 applies filtering, the resulting image may include edges of varying thickness. Therefore, the edge-guided colorization system 106 applies NMS to thin the edge to a more uniform thickness. In at least one embodiment, the edge-guided colorization system 106 applies NMS to thin the edge to a 1-pixel width. The edge-guided colorization system 106 also applies thresholding to remove filter responses below a certain threshold. Additionally, in at least one embodiment, the edge-guided colorization system 106 applies hysteresis to produce continuous edges. For example, the edge-guided colorization system 106 can utilize an edge tracing algorithm to remove floating weak edges and connect weak edges to strong edges.

[0066] By applying Canny edge detection 404 to the chromaticity map Y 402, the edge-guided colorization system 106 generates a ground truth chromaticity edge V 406. The ground truth chromaticity edge V 406 indicates a chromaticity boundary or a location where a chromaticity value (i.e., color) changes in the chromaticity map Y 402. As mentioned, the ground truth chromaticity edge V 406 comprises a binary edge of uniform thickness.

[0067] like Figure 4A As further illustrated in FIG. 1 , the edge-guided colorization system 106 inputs the ground truth grayscale digital image X 408 into the color edge prediction neural network 304. The edge-guided colorization system 106 extracts intensity values ​​from the ground truth color image to generate the ground truth grayscale digital image X 408. The edge-guided colorization system 106 uses the ground truth grayscale digital image X 408 as input to the color edge prediction neural network 304. The color edge prediction neural network 304 generates predicted edge

[0068] The edge-guided colorization system 106 also applies a loss function 412. The edge-guided colorization system 106 compares the ground truth chrominance edge V 406 with the predicted edge For example, the edge-guided colorization system 106 can adjust the parameters of the color edge prediction neural network 304 so that the binary cross entropy loss function given by the following formula Minimize:

[0069]

[0070] Where X represents the ground truth grayscale digital image X 408, V represents the ground truth chrominance edge V 406, and 304. The edge-guided colorization system 106 trains the network Let the classification loss on the training set of ground truth color images be In some embodiments, the edge-guided colorization system 106 can utilize alternative loss functions, such as mean square error, quadratic loss, L2 loss, L1 loss, or mean deviation error. Moreover, the edge-guided colorization system 106 can update the parameters of the neural network based on the loss function (e.g., through back propagation).

[0071] Figure 4A The edge-guided colorization system 106 is illustrated for training a color edge prediction neural network 304 . Figure 4B The edge-guided colorization system 106 is illustrated as applying a color edge prediction neural network 304 to generate a color edge map (i.e., color edge map 306). The edge-guided colorization system 106 applies the color edge prediction neural network 304 to a grayscale digital image X 420. The color edge prediction neural network 304 generates an initial color edge map 422 based on the grayscale digital image X 420. The edge-guided colorization system 106 applies non-maximum suppression (NMS) 430 (or simply “NMS 430”) and thresholding 426 to the initial color edge map 422 to generate a color edge map 434.

[0072] like Figure 4B As illustrated in , the color edge prediction neural network 304 generates an initial color edge map 422. The initial color edge map 422 includes initial edges 424. However, the ground truth chrominance edge V 406 used during training includes binary edges, and the initial edges 424 of the initial color edge map 422 are soft values. For example, Figure 4B , the set of initial edges 424 of the initial color edge map 422 includes weak edges, shading, and other soft features that contrast with the continuous and uniform thickness of the ground truth chrominance edge V 406. Therefore, the edge-guided colorization system 106 applies post-processing steps including NMS 430 and thresholding 426 to binarize the initial color edge map 422.

[0073] like Figure 4B, the edge-guided colorization system 106 applies NMS 430 to initial edges 424 of the initial color edge map 422. As previously mentioned, applying NMS 430 adjusts the pixel width to uniform thickness. Thus, the edge-guided colorization system 106 generates a uniform thickness edge map 428 in which the color edges are uniform thickness. In at least one embodiment, the edge-guided colorization system 106 applies NMS 430 so that the color edges of the uniform thickness edge map 428 are 1 pixel wide.

[0074] like Figure 4B As further illustrated in , the edge-guided colorization system 106 applies thresholding 426 to the uniform thickness edge map 428. The color edge prediction neural network 304 generates a confidence score (e.g., for each color edge that ultimately appears in the uniform thickness edge map 428). During thresholding 426, the edge-guided colorization system 106 determines a threshold t∈[0,1], where t represents a threshold and has a value in the range of 0 to 1. As part of the thresholding 426, the edge-guided colorization system 106 removes edges from the color edges whose confidence scores are below the determined threshold. Based on the thresholding 426, the edge-guided colorization system 106 generates a color edge map 434 that includes only the edges in the uniform thickness edge map 428 whose confidence scores satisfy the determined threshold. For example, as Figure 4B As illustrated in , color edge map 434 contains fewer shadows and other soft values ​​than uniform thickness edge map 428 .

[0075] The edge-guided colorization system 106 may utilize a predetermined threshold as a color edge threshold. For example, in at least one embodiment, the edge-guided colorization system 106 utilizes a predetermined color edge threshold (e.g., t=0.17) to generate the color edge map 434. As previously described with respect to Figure 3 As mentioned, the edge-guided colorization system 106 can present the color edge map 434 resulting from the color edge threshold via the colorization graphical user interface. The colorization graphical user interface can include the color edge threshold element 318. In at least one embodiment, the color edge threshold element is a sliding bar indicating the color edge threshold. Based on user interaction with the color edge threshold element (e.g., sliding the bar to indicate different thresholds), the edge-guided colorization system 106 can determine a modified edge threshold. In such cases, the edge-guided colorization system 106 can utilize the modified edge threshold to generate a modified thresholded edge map, and subsequently utilize the modified edge map (after NMS, thresholding, and hysteresis). Figure 5 and the accompanying discussion provide additional details regarding how the edge-guided colorization system 106 generates the modified edge map.

[0076] Despite Figure 4B 430 and thresholding 426, the edge-guided colorization system 106 also applies hysteresis. As mentioned, by utilizing hysteresis, the edge-guided colorization system 106 ensures that the color edges of the color edge map 434 are 432 is continuous and connected.

[0077] Similarly, although the color edge prediction neural network is generally described with respect to applying it to the grayscale digital image X 420 Figure 4B , but the edge-guided colorization system 106 can also utilize NMS, thresholding, and other post-processing actions during training. , the edge-guided colorization system 106 may apply similar post-processing so that the output of the color edge prediction neural network 304 can be easily compared with the ground truth chrominance edge V 406.

[0078] As described above, the edge-guided colorization system 106 can present a color edge map to a user via a colorization graphical user interface and receive color edge modifications. Based on the color edge modifications, the edge-guided colorization system 106 can generate a modified edge map. Figure 5 The edge-guided colorization system 106 is illustrated generating a modified edge map (ie, modified edge map 324 ) in accordance with one or more embodiments.

[0079] like Figure 5 , the edge-guided colorization system 106 presents the color edge map 306 via the colorization graphical user interface 504. The colorization graphical user interface 504 includes the color edge threshold element 318, the edge suggestion element 320, and the user-drawn edge element 322. Based on the user interaction with the color edge threshold element 318, the edge suggestion element 320, and the user-drawn edge element 322, the edge-guided colorization system 106 generates a modified edge map 324.

[0080] like Figure 5 , the color edge threshold element 318 includes an interactive sliding scale. In at least one embodiment, the edge-guided colorization system 106 presents the color edge threshold via the color edge threshold element 318. For example, the color edge threshold element 318 may instruct the edge-guided colorization system 106 to generate the color edge map 306 using the color edge threshold t=0.17. The edge-guided colorization system 106 receives user interaction with the color edge threshold element 318 to determine a modified edge threshold. For example, Figure 5, the modified edge threshold t=0.50. The edge-guided colorization system 106 applies thresholding using the modified edge threshold to the uniform thickness edge map 428 to generate a modified thresholded edge map. As illustrated, because the modified edge threshold (t=0.50) is higher than the color edge threshold (t=0.17), the modified thresholded edge map contains fewer edges than the color edge map 306. Additionally, the edge-guided colorization system 106 applies hysteresis to the modified thresholded edge map to generate the modified edge map 324.

[0081] The colorization graphical user interface 504 also includes an edge suggestion element 320. The edge-guided colorization system 106 presents the suggested edges to the user, and the edge-guided colorization system 106 includes one or more user-selected edges from the suggested edges in the modified edge map 324. The color edge scan may include a subset of luminance edges. Therefore, in at least one embodiment, the edge-guided colorization system 106 presents luminance edges as suggested edges. As used herein, the term "luminance edge" refers to an edge within a luminance map. In particular, the luminance map conveys brightness information of an image corresponding to a brightness value for each pixel. A luminance edge indicates a location or boundary where a brightness value changes.

[0082] like Figure 5 As illustrated in FIG. 1 , the edge-guided colorization system 106 applies a Canny edge detector 506 to the colorized image to generate an illumination edge 508 of the grayscale digital image 302. Figure 5 , the Canny edge detector 506 performs the same Figure 4A The same process as in the Canny edge detection 404 of the embodiment is performed. Specifically, the edge-guided colorization system 106 applies filtering, NMS, thresholding, and hysteresis to the grayscale digital image 302 using the Canny edge detector 506 .

[0083] The edge-guided colorization system 106 presents the luma edges 508 as suggested edges to the user via the colorization graphical user interface 504. Based on the user selection of one or more of the luma edges 508, the edge-guided colorization system 106 includes the one or more edges in the modified edge map 324.

[0084] like Figure 5, the edge-guided colorization system 106 provides the user-drawn edge element 322 within the colorization graphical user interface 504. The edge-guided colorization system 106 can receive additional edges via the user-drawn edge element 322. In particular, the edge-guided colorization system 106 receives hand-drawn input based on user interaction with the user-drawn edge element 322, and generates the user-drawn edge 510. The edge-guided colorization system 106 includes the user-drawn edge 510 within the set of modified color edges within the modified edge map 324.

[0085] As discussed above, the edge-guided colorization system 106 can utilize the edge map as an input to the edge-guided colorization neural network to incorporate edge guidance. FIG. 6A to FIG. 6B An edge-guided colorization system 106 is illustrated for training and applying an edge-guided colorization neural network to generate a colored image in accordance with one or more embodiments. The edge-guided colorization system 106 can predict the color of a grayscale image by combining a user color point and edge guidance. In CIE Lab space, a grayscale digital image includes an L or lightness channel, and a desired colored image includes ab or color components.

[0086] Fig. 6A The edge-guided colorization system 106 is illustrated for training an edge-guided colorization neural network 606. The edge-guided colorization system 106 uses the ground truth color Y 602, the training grayscale image X 604, the training color point U 614, and the Canny edge V 612 as inputs to the edge-guided colorization neural network 606. The edge-guided colorization neural network 606 generates the training predicted color The edge-guided colorization system 106 compares it to the ground truth color Y 602. The edge-guided colorization system 106 adjusts the parameters of the edge-guided colorization neural network 606 to reduce the training predicted color The loss 610 between the ground truth color Y 602 .

[0087] Relative to Fig. 6A , the edge-guided colorization system 106 generates training inputs for the edge-guided colorization neural network 606 based on the training ground-truth color image. In particular, the edge-guided colorization system 106 isolates the luminance values ​​of the training ground-truth color image to generate the training grayscale image X 604. The edge-guided colorization system 106 also isolates the chrominance values ​​of the training ground-truth color image to generate the ground-truth color Y 602 corresponding to the training grayscale image X 604. Therefore, in one or more embodiments, the ground-truth color Y 602 includes a chrominance map of the training color image. In addition, the edge-guided colorization system 106 extracts the intensity or luminance values ​​from the training color image to generate the training grayscale image X 604.

[0088] The edge-guided colorization neural network 606 incorporates edge guidance when generating a colorized image. During training, the edge-guided colorization system 106 uses a projection function to simulate edges drawn by the user. In at least one embodiment, the edge-guided colorization system 106 projects edges by applying a Canny edge detector 620 to the ground truth color Y 602. The purpose of applying the Canny edge detector 620 is to indicate the area where the chromaticity value actually changes in the chromaticity diagram (i.e., the ground truth color Y 602). Therefore, in at least one embodiment, the edge-guided colorization system 106 utilizes a low-level edge detector. Additionally, in at least one embodiment, the edge-guided colorization system 106 extracts color edges at half resolution rather than full resolution, which often produces overly complex color edges. The edge-guided colorization system 106 uses Canny edges V 612 during application to simulate a color edge map or a modified edge map input.

[0089] During training, the edge-guided colorization system 106 utilizes the training color point U 614 as an input to the edge-guided colorization neural network 606. The edge-guided colorization system 106 simulates user interaction by randomly exposing points (e.g., sampling pixels) in the ground truth color Y 602 to generate the training color point U 614. By sampling from the ground truth color and using the training color point U 614 as input, the edge-guided colorization system 106 trains the edge-guided colorization neural network 606 to depend on the training color point U 614. During application, the edge-guided colorization neural network 606 follows the colors provided by the user even without explicit constraints.

[0090] like Fig. 6A As shown in FIG. 1 , the edge-guided colorization neural network 606 generates the training predicted color Training to predict color Included is a predicted chromaticity map that, when combined with the corresponding grayscale image (ie, training grayscale image X604), creates a complete colorized image.

[0091] The edge-guided colorization system 106 is trained to predict color The edge-guided colorization system 106 trains the parameters of the edge-guided colorization neural network 606 to map the input training grayscale image X 604, the training color point U 614, and the canny edge V 612 as closely as possible to the ground truth color Y 602. In at least one embodiment, the edge-guided colorization system 106 trains the parameters of the edge-guided colorization neural network 606 to map the input training grayscale image X 604, the training color point U 614, and the canny edge V 612 to the ground truth color Y 602. Each pixel of is compared to the corresponding pixel in the ground truth color Y 602 to utilize a regression loss of l0 per pixel. As mentioned previously, the edge-guided colorization system 106 can also utilize alternative loss functions.

[0092] During training, the edge-guided colorization system 106 incorporates the training grayscale image X 604 , the training color point U 614 , and the Canny edge V 612 as inputs to the edge-guided colorization neural network 606 The edge-guided colorization system 106 adjusts the parameters of the edge-guided colorization neural network 606 to minimize the loss 610 using the following objective function:

[0093]

[0094] in Denotes a loss function that maps the output of the edge-guided colorization neural network 606 against the ground truth color Y 602 (i.e., training to predict color ). Function P u (Y) indicates a user interaction simulated by randomly revealing a point in the ground truth color Y 602 (ie, a training color point U 614). Function P u (Y) Simulates a user-drawn edge (ie, Canny edge V612).

[0095] Although Fig. 6A FIG. 1 illustrates how the edge-guided colorization system 106 trains the edge-guided colorization neural network 606. Figure 6B The edge-guided colorization system 106 applying the edge-guided colorization neural network 606 according to one or more embodiments is illustrated. During application, the edge-guided colorization system 106 uses the edge-guided colorization neural network 606 to analyze the grayscale digital image X 622, the user color point U 624, and the user edge map 626. Specifically, the edge-guided colorization system 106 generates a grayscale channel, a color point channel, and an edge map channel, and analyzes these channels via the edge-guided colorization neural network 606. The edge-guided colorization neural network 606 generates a predicted color The edge-guided colorization system 106 combines this with the grayscale digital image X 622 to create a colorized image 630 .

[0096] During application, the edge-guided colorization system 106 receives a grayscale digital image X 622 from a user. The edge-guided colorization system 106 utilizes the trained edge-guided colorization neural network 606 to generate a predicted color for the received grayscale digital image X 622 The desired output includes a colorized version of the grayscale digital image X 622 .

[0097] The edge-guided colorization system 106 includes the user color point U 624 as an input to the edge-guided colorization neural network 606. In at least one embodiment, the edge-guided colorization system 106 represents the user color point U 624 via a set of color point channels as Where the first two channels are the chromaticity values ​​selected by the user, and the third channel is a binary mask indicating whether the user has selected a color at that location. In at least one embodiment, the color point channel starts with all zeros and is sparsely filled as the edge-guided colorization system 106 receives user color points.

[0098] The edge-guided colorization system 106 receives the user color point U 624 via the colorization graphical user interface. For example, in at least one embodiment, the edge-guided colorization system 106 presents the grayscale image and the ab color gamut via the colorization graphical user interface. The edge-guided colorization system 106 receives selections of various points in the grayscale image and selections of chromaticity values ​​corresponding to the various points on the color gamut. Fig. 7A and the accompanying discussion provide additional details regarding how the edge-guided colorization system 106 receives the user color point U 624 .

[0099] Additionally, the edge-guided colorization system 106 incorporates edge guidance from the user by using the user edge map V 626 and the corresponding edge map channel. If the user has not modified the color edge map 434, then the user edge map V 626 includes the color edge map 434. Alternatively, if the edge-guided colorization system 106 receives a user modification to the color edge map 434, then the user edge map V 626 includes the modified edge map 324. In at least one embodiment, the edge-guided colorization system 106 indicates the user edge map V 626 as a sparse binary matrix via the edge map channel. More specifically, the edge-guided colorization system 106 indicates the user edge map V 626 as a sparse binary matrix via the edge map channel. It indicates the presence or absence of an edge.

[0100] As previously mentioned, the edge-guided colorization system 106 trains the edge-guided colorization neural network 606 to map the grayscale digital image X 622, the user color point U 624, and the user edge map V 626 to output a predicted color Additionally, in some embodiments, the edge-guided colorization system 106 connects the user edge map V 626 to the feature vectors at all intermediate layers of the neural network. In at least one embodiment, the edge-guided colorization neural network 606 comprises a U-Net convolutional neural network.

[0101] like Figure 6BAs shown in FIG. 1 , the edge-guided colorization neural network 606 generates a predicted color Predicted Color includes a predicted chromaticity map corresponding to the grayscale digital image X 622. In particular, the predicted color Includes the chrominance value of each pixel in the grayscale digital image X 622. In addition, the predicted color Edge guidance integrated by edge-guided colorization neural network 606 is reflected.

[0102] The edge-guided colorization system 106 can be configured to The edge-guided colorization system 106 can generate a colorized image 630 by combining the chromaticity or color value of the grayscale digital image X 622 with the lightness value of the grayscale digital image X 622. The ab values ​​of are combined to generate a colored image 630 in the CIE Lab space. The edge-guided colorization system 106 presents the colored image 630 to the user via a colorization graphical user interface.

[0103] As mentioned, in at least one embodiment, the edge-guided colorization system 106 feeds the user edge map V 626 into multiple layers of the edge-guided colorization neural network 606. For example, the edge-guided colorization system 106 can concatenate the user edge map V 626 into the feature vector at each layer of the edge-guided colorization neural network 606. In another embodiment, the edge-guided colorization system 106 feeds the user edge map V 626 into multiple layers, but not all layers, of the edge-guided colorization neural network 606.

[0104] In particular, as discussed above, the edge-guided colorization neural network 606 may include various layers that perform operations on feature vectors based on learned weights / parameters. Thus, any particular layer may receive an input feature vector as input and generate an output feature vector. The edge-guided colorization system 106 may add (i.e., concatenate) the user edge graph V 626 to the feature vectors of multiple layers of the edge-guided colorization neural network 606. Thus, the edge-guided colorization system 106 emphasizes the user edge graph V 626 at multiple intermediate layers of the edge-guided colorization neural network 314.

[0105] In addition, the edge-guided colorization system 106 can adjust the size and dimension of the user edge map V 626 for the feature vectors of each layer of the edge-guided colorization neural network 606. In particular, the layers of the edge-guided colorization neural network 606 can be set to different sizes to analyze the grayscale digital image X 622 at different levels of detail. Therefore, the edge-guided colorization system 106 can adjust the size of the user edge map V 626 to match the corresponding layer of the edge-guided colorization neural network 606.

[0106] To illustrate, the edge-guided colorization system 106 may analyze the grayscale channel, the color edge map channel, and / or the color point channel using a first layer (e.g., a first convolutional layer) of the edge-guided colorization neural network 606 to generate a feature vector. The edge-guided colorization system 106 may add (e.g., concatenate or otherwise append) the edge map to the feature vector to generate an edge-map-modified feature vector. The edge-guided colorization system 106 may then analyze the edge-map-modified feature vector using a second layer (e.g., a second convolutional layer) of the edge-guided colorization neural network 606. The edge-guided colorization system may iteratively add the edge map (at different sizes or scales) at different layers of the neural network.

[0107] The edge-guided colorization system 106 receives various user interactions to generate inputs for the edge-guided colorization neural network 606. For example, the edge-guided colorization system 106 receives a user color point U 624 via a colorization graphical user interface. Additionally, the edge-guided colorization system 106 receives inputs including color edge modifications 310 to generate a user edge map V 626. FIG. 7A to FIG. 7B An example colorized graphical user interface is illustrated. More specifically, Fig. 7A 6 illustrates how the edge-guided colorization system 106 receives a user color point U 624 via a colorization graphical user interface in color mode according to one or more embodiments. Figure 7B 3 illustrates how the edge-guided colorization system 106 receives color edge modifications 310 via a colorization graphical user interface in edge mode according to one or more embodiments. Fig. 7A and Figure 7B A series of example colorized graphical user interfaces are illustrated, but it should be appreciated that the edge-guided colorization system 106 may utilize different colorized graphical user interfaces including different configurations or different user interface elements.

[0108] like Fig. 7AAs illustrated in , in color mode, the edge-guided colorization system 106 presents a colorization graphical user interface 706 via a display screen 704 on a user client device 702 (i.e., user client device 108). As shown, the colorization graphical user interface 706 includes a color point input element 709 through which the edge-guided colorization system 106 receives user input of a user color point U 624. The color point input element 709 includes a color selection element 710, a point selection element 712, a suggested color 716, a recently used color 718, and a selected color 720. Additionally, as shown in FIG. Fig. 7A As illustrated in , the colored graphical user interface 706 in the color mode includes a drawing option selection element 722 , a loading element 725 , a loading element 727 , a restart element 729 , and an exit element 730 .

[0109] As illustrated by the drawing options selection element 722, Fig. 7A Based on the user interaction with the drawing option selection element 722, the edge-guided colorization system 106 switches between presenting a color point input element 708 for selecting a user color point U 624 and presenting a color edge modification element for generating a user edge map V 626. Based on the user interaction with the color mode option of the drawing option selection element 722, the edge-guided colorization system 106 presents a color point input element 709 via the colorization graphical user interface 706.

[0110] Based on the interaction with the point selection element 712, the edge-guided colorization system 106 identifies points of the user color point U 624. The point selection element 712 includes an input grayscale digital image. The edge-guided colorization system 106 detects the selection of the region in the point selection element 712 and associates the user-selected color with each of those points. Additionally, as illustrated, the edge-guided colorization system 106 displays via the point selection element 712 which color points have been selected and associated with color values.

[0111] The edge-guided colorization system 106 associates a color value with the selected color point in the point selection element 712. For example, the edge-guided colorization system detects a user selection of the selected color point 724 and highlights the selected color point 724 to indicate a valid color selection for pixels within the selected color point 724. The edge-guided colorization system 106 receives the color selection to associate with the selected color point 724 via the color selection element 710.

[0112] The edge-guided colorization system 106 presents a color selection element 710 as part of the color point input element 709. The color selection element 710 includes a color gamut that displays the entire range of colors available for the selected color point 724. As illustrated, the color selection element 710 includes an ab color gamut. The edge-guided colorization system 106 detects user interaction with the color selection element 710 and updates the selected color 720 to display the color currently selected by the icon in the color selection element 710.

[0113] In addition to presenting the color selection element 710, the edge-guided colorization system 106 also presents a suggested color 716. The edge-guided colorization system 106 predicts possible colors for the selected color point 724 and presents the possible colors in the suggested colors 716. In at least one embodiment, the edge-guided colorization system 106 uses a color prediction neural network to predict possible colors for a given point. The edge-guided colorization system 106 trains the color prediction neural network using training colors and training grayscale images. Based on analyzing the values ​​of the training grayscale images, the edge-guided colorization system 106 generates a prediction of the color. Additionally, the edge-guided colorization system 106 can improve the accuracy of the possible colors presented within the suggested colors 716 by associating the colors of other color points. For example, if the user indicates a color for a color point, the edge-guided colorization system 106 determines a lower probability that the same color will be used at a different color point.

[0114] As the edge-guided colorization system 106 receives the color point U 624 via the color point input element 709, the edge-guided colorization system 106 updates the colorized image 714. The colorized image 714 includes the input grayscale digital image with the predicted colors determined by the edge-guided colorization neural network 606. In at least one embodiment, the edge-guided colorization system 106 generates the colorized image 714 using the predicted colors before receiving any color points. Thus, the edge-guided colorization system 106 can receive corrections to the predicted colors via the color point input element 709. In another embodiment, the edge-guided colorization system 106 presents the grayscale image as the colorized image 714 and updates the colorized image 714 by adding colors in real time based on the received user color points.

[0115] like Fig. 7A, the colorization graphical user interface 706 further includes a load element 727, a restart element 729, and an exit element 730. Based on the interaction with the load element 725, the edge-guided colorization system 106 loads a different grayscale image for colorization. For example, the edge-guided colorization system 106 may access a grayscale image stored at the user client device 702. Based on the user selection of the load element 727, the edge-guided colorization system 106 saves the entered user color point and the latest edge modification. Based on the user selection of the restart element 729, the edge-guided colorization system 106 removes all selected color points and edge modifications. In at least one embodiment, the load element 727 and the restart element 729 are specific color modes or edge modes. For example, based on the user selection of the load element 727 in the color mode, the edge-guided colorization system 106 saves the current input color point. If the edge-guided colorization system 106 detects the selection of the restart element 729 while in the color mode, the edge-guided colorization system 106 only resets the color point and not the edge. Based on the user selection of the exit element 730 , the edge-guided colorization system 106 closes the colorization graphical user interface 706 .

[0116] Based on the user selection of the edge mode of the drawing option selection element 722 , the edge-guided colorization system 106 updates the colorization graphical user interface 706 to the edge mode. Figure 7B The colorized graphical user interface 706 in edge mode is illustrated, through which the edge-guided colorization system 106 receives edge modifications to generate an edge map. The edge-guided colorization system 106 presents the colorized graphical user interface 706 via the display screen 704 on the user client device 702. Figure 7B , the edge-guided colorization system 106 presents a color edge modification element 735 via the colorization graphical user interface 706. The color edge modification element 735 includes a modified edge map element 736, a color edge threshold element 732, and a user-drawn edge element 734 for input of color edge modification. Additionally, the edge-guided colorization system 106 presents a drawing option selection element 722, a selectable gray color element 740, a display edge element 742, an edge suggestion element 744, and an update edge element 738.

[0117] The edge-guided colorization system 106 presents the color edge threshold component 732 as part of the color edge modification component 735. Based on interaction with the color edge threshold component 732, the edge-guided colorization system 106 determines a modified edge threshold to use when thresholding the initial color edge map 422. A higher modified edge threshold is linked to fewer color edges, while a lower modified edge threshold produces more color edges. Figure 7B, the color edge threshold element 732 includes a sliding scale and an indication of a threshold value (e.g., t=0.5). The edge-guided colorization system 106 determines a modified edge threshold value based on user interaction with the sliding scale or the threshold indication. For example, the edge-guided colorization system 106 modifies the edge threshold value based on a user selecting or dragging the sliding scale. Additionally, the edge-guided colorization system 106 enables a user to manually enter a modified threshold value using the threshold indication.

[0118] The edge-guided colorization system 106 also presents a user-drawn edge element 734 as part of a color edge modification element 735. The user-drawn edge element 734 includes an area in which the user can draw other desired edges. Based on user interaction with the user-drawn edge element 734 (e.g., dragging a cursor over the user-drawn edge element or clicking on the user-drawn edge element), the edge-guided colorization system 106 adds the user-drawn edge to the modified edge map. For example, in addition to the color edge 726, the edge-guided colorization system 106 also adds the user-drawn edge 728 to the modified edge map element 736.

[0119] The edge-guided colorization system 106 determines what to display as part of the user-drawn edge element based on the detected user interactions with the grayscale element 740 and the display edge element 742. As illustrated, the edge-guided colorization system 106 detects the selection of the grayscale element 740 and thus presents the uploaded grayscale image within the user-drawn edge element 734. Alternatively, in at least one embodiment, the edge-guided colorization system 106 presents an option to display only the edge as part of the user-drawn edge element. Additionally, based on the selection of the display edge element 742, the edge-guided colorization system 106 displays the color edge 726. Based on the detected deselection of the display edge element 742, the edge-guided colorization system 106 removes the color edge 726 and displays only the user-drawn edge 728.

[0120] The edge-guided colorization system 106 also includes an edge suggestion element 744 as part of the color edge modification element. Based on detecting a user interaction with the edge suggestion element 744, the edge-guided colorization system 106 presents additional luminance edges as suggested edges. For example, in one embodiment, the edge-guided colorization system 106 presents an additional panel that displays luminance edges of the grayscale image. In another embodiment, based on the selection of the edge suggestion element 744, the edge-guided colorization system 106 overlays luminance edges on top of the color edges 726 and the user-drawn edges 728 in the user-drawn edge element 734. For example, the edge-guided colorization system 106 presents the luminance edges as a different color than the color edges 726 and the user-drawn edges 728. Based on the user selection of one or more of the luminance edges, the edge-guided colorization system 106 determines that the one or more edges are to be included in the modified edge map. The edge-guided colorization system 106 highlights the selected one or more of the luminance edges to indicate that the selected one or more edges are to be included in the modified edge map.

[0121] As the edge-guided colorization system 106 receives user interactions through the color edge threshold component 732, the user-drawn edge component, and the edge suggestion component 744, the edge-guided colorization system 106 updates the modified edge map component 736 in real time. For example, the edge-guided colorization system 106 adds or removes edges in the modified edge map component 736 to illustrate the latest changes to the modified edges.

[0122] like Figure 7B , the colorization graphical user interface 706 includes a colorized image 714. The edge-guided colorization system 106 presents the most recently predicted colorized image. In at least one embodiment, the edge-guided colorization system 106 refreshes the colorized image 714 based on a user selection of an update edge element 738. In another embodiment, the edge-guided colorization system 106 automatically updates the colorized image 714 based on a user interaction with a color edge modification element 735.

[0123] Despite Figure 7B 706. Although not shown in FIG. 70, the edge-guided colorization system 106 also removes edges based on user interaction. In at least one embodiment, the edge-guided colorization system 106 presents a remove edge option via the colorization graphical user interface 706. Based on a selection of the remove edge option and an edge in the color edge (e.g., displayed via the modified edge map element 736, the user-drawn edge element 734, or another edge display), the edge-guided colorization system 106 removes the selected edge from the modified edge map.

[0124] Figure 8A schematic diagram of components of an edge-guided colorization system 106 is illustrated in accordance with one or more embodiments. As shown, the edge-guided colorization system 106 may be part of a computing device 800 (e.g., a user client device 108 and / or a server device(s) 102). Additionally, the edge-guided colorization system 106 may be part of an image editing system 104. The edge-guided colorization system 106 may include, but is not limited to, a color edge prediction neural network training engine 802, a color edge prediction neural network application engine 804, a graphical user interface manager 806, an edge modification manager 808, a color point input manager 810, an edge-guided colorization neural network training engine 812, an edge-guided colorization neural network application engine 814, and a storage manager 816. The edge-guided colorization system 106 may be implemented as part of the image editing system 104 in a distributed system of server devices for managing, storing, and editing digital images. Alternatively, the edge-guided colorization system 106 may be implemented on a single computing device such as a Figure 1 on a user client device 108).

[0125] In one or more embodiments, each of the components of edge-guided colorization system 106 communicates with each other using any suitable communication technology. Figure 1 As illustrated in , components of the edge-directed colorization system 106 may communicate with one or more other devices including a user client device 108. Figure 8 Components of the edge-guided colorization system 106 are shown as separated, but any of the subcomponents may be combined into fewer components, such as into a single component or divided into more components as may serve a particular implementation. Figure 8 , but at least some of the components used to perform operations in conjunction with the edge-guided colorization system 106 described herein may be implemented on other devices within the environment.

[0126] The components of the edge-guided colorization system 106 may include software, hardware, or both. For example, the components of the edge-guided colorization system 106 may include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices (e.g., the user client device 108). The computer-executable instructions of the edge-guided colorization system 106, when executed by one or more processors, may cause the computing device to perform the image editing methods described herein. Alternatively, the components of the edge-guided colorization system 106 may include hardware (such as a dedicated processing device) to perform a specific function or group of functions. Additionally or alternatively, the components of the edge-guided colorization system 106 may include a combination of computer-executable instructions and hardware.

[0127] In addition, the components of the edge-guided colorization system 106 that perform the functions described herein can be implemented, for example, as part of a stand-alone application, as a module of an application, as an application plug-in including a content management application, as one or more library functions that can be called by other applications, and / or as a cloud computing model. Thus, the components of the edge-guided colorization system 106 can be implemented as part of a stand-alone application on a personal computing device or a mobile device. Alternatively or additionally, the components of the edge-guided colorization system 106 can be implemented in any application that allows the creation and delivery of marketing content to users, including but not limited to, CREATIVE (such as and ) applications. “ADOBE”, “ADOBE PHOTOSHOP”, “ADOBE LIGHTROOM”, “ADOBE ILLUSTRATOR”, “ADOBECOLOR”, “ADOBE CREATIVE CLOUD” and “ADOBE INDESIGN” are registered trademarks of Adobe Systems Incorporated in the United States and / or other countries.

[0128] like Figure 8 As shown in , the edge-guided colorization system 106 includes a color edge prediction neural network training engine 802. The color edge prediction neural network training engine 802 uses training data to train network parameters of the color edge prediction neural network. In particular, the color edge prediction neural network training engine 802 inputs a ground truth grayscale digital image into the color edge prediction neural network to generate predicted edges. The color edge prediction neural network training engine 802 compares the predicted edges with ground truth data (including ground truth chrominance edges). Additionally, the color edge prediction neural network training engine 802 can determine, evaluate, identify, or generate a measure of the loss or error between the predicted edges and the ground truth data.

[0129] The edge-guided colorization system 106 also includes a color edge prediction neural network application engine 804. The color edge prediction neural network application engine 804 generates a color edge map based on receiving the grayscale digital image. The color edge prediction neural network application engine 804 manages an initial color edge map including initial color edges and corresponding confidence scores generated by the color edge prediction neural network. Additionally, the color edge prediction neural network application engine 804 applies NMS and thresholding to the initial color edge map. Therefore, the color edge prediction neural network application engine 804 accesses, manages, and stores color edge thresholds. Additionally, the color edge prediction neural network application engine 804 applies hysteresis to the thresholded edge map to generate a color edge map including color edges.

[0130] The edge-guided colorization system 106 also includes a graphical user interface manager 806. The graphical user interface manager 806 generates, manages, and receives input from one or more graphical user interfaces. The graphical user interface manager 806 generates a colorized graphical user interface at the user client device 108, which presents a color edge map, a color edge threshold element, an edge suggestion element, and a user-drawn edge element. The graphical user interface manager 806 receives user interactions with one or more of the mentioned elements and communicates the user interactions to the edge modification manager 808. Additionally, the graphical user interface manager 806 presents data related to coloring a grayscale image. For example, the graphical user interface manager 806 can display a grayscale image, a predicted colorized image, and other data via the user client device 108.

[0131] The edge modification manager 808 receives, implements, and manages color edge modifications to generate a modified edge map as an input to the edge-guided colorization neural network. The edge modification manager 808 accesses user responses from the graphical user interface manager 806. In particular, the edge modification manager 808 adds and removes edges by determining modified edge thresholds. Additionally, the edge modification manager 808 adds one or more selected edges of user-drawn edges and illumination edges to the modified edge map. The edge modification manager 808 can also remove selected edges to generate a modified edge map.

[0132] like Figure 8 As illustrated in FIG, the edge-guided colorization system 106 also includes a color point input manager 810. The color point input manager 810 communicates with the graphical user interface manager 806 to generate user color points that are input to the edge-guided colorization neural network. The color point input manager 810 accesses, manages, and stores chromaticity values ​​of selected color points linked to any grayscale image.

[0133] The edge-guided colorization neural network training engine 812 is part of the edge-guided colorization system 106. The edge-guided colorization neural network training engine 812 uses training data to train network parameters of the edge-guided colorization neural network. In particular, the edge-guided colorization neural network training engine 812 accesses, manages, and stores training data including ground-truth color images. In particular, the edge-guided colorization neural network training engine 812 separates the ground-truth color images into training grayscale images and ground-truth colors. The edge-guided colorization neural network training engine 812 extracts training color points and canny edges from the ground-truth colors and uses these training color points and canny edges in addition to using the training grayscale images as inputs to the edge-guided colorization neural network. The edge-guided colorization neural network training engine 812 accesses the training predicted color output of the edge-guided colorization network, compares the training predicted color output with the ground-truth colors, and changes the network parameters of the edge-guided colorization neural network. The edge-guided colorization neural network training engine 812 can determine, evaluate, identify, or generate a measure of the loss or error between the training predicted colors and the ground-truth colors.

[0134] The edge-guided colorization system 106 includes an edge-guided colorization neural network application engine 814. The edge-guided colorization neural network application engine 814 generates predicted colors based on receiving a grayscale digital image, a user color point, and / or a user edge map. The edge-guided colorization neural network application engine 814 communicates with the graphical user interface manager 806, the edge modification manager 808, and the color point input manager 810 to access the grayscale digital image input, the user color point input, and the user edge map input to the edge-guided colorization neural network. The edge-guided colorization neural network application engine 814 also manages the predicted color output of the edge-guided colorization neural network. In at least one embodiment, the edge-guided colorization neural network application engine 814 also combines the predicted colors with the grayscale digital image to generate a colored image.

[0135] The edge-guided colorization system 106 includes a storage manager 816. The storage manager 816 stores (via one or more memory devices) the color edge prediction neural network 304, the edge-guided colorization neural network 606, the training data 822, the digital image 824, and / or the color edges 826. The color edge prediction neural network 304 includes a neural network trained by the color edge prediction neural network training engine 802 to generate an initial color edge map based on receiving and analyzing a grayscale digital image. The storage manager 816 stores the trained network parameters of the color edge prediction neural network 304.

[0136] The storage manager 816 also stores the edge-guided colorization neural network 606. The edge-guided colorization neural network training engine 812 trains the edge-guided colorization neural network 606 to generate predicted colors based on the received grayscale digital image, the user color point, and the user edge map. The storage manager 816 stores the trained network parameters of the edge-guided colorization neural network 606.

[0137] The storage manager 816 also stores training data 822. The training data 822 includes ground truth color images for training the color edge prediction neural network 304 and the edge-guided colorization neural network 606. More specifically, for training the color edge prediction neural network 304, the training data 822 includes chromaticity maps, ground truth grayscale digital images, and ground truth chromaticity edges. For training the edge-guided colorization neural network 606, the training data 822 includes training grayscale images, training color points, canny edges, and ground truth colors.

[0138] The storage manager 816 stores the digital image 824. The digital image 824 includes the grayscale image received from the user and the corresponding colorized image generated by the edge-guided colorization neural network 606.

[0139] The storage manager 816 also stores color edges 826. The color edges 826 include color edges for generating a color edge map for a final colorized image. For example, the color edges 826 include an initial color edge map and corresponding edge confidence scores, a uniform thickness edge map, and a color edge map generated by the color edge prediction neural network 304, as well as a modified edge map. In addition, the color edge 826 stores an illumination edge used as a suggested edge.

[0140] Fig. 9 A flow chart of a method for generating a colorized image is shown. Fig. 9 The actions according to one embodiment are illustrated, but alternative embodiments may omit, add, reorder and / or modify Fig. 9 Any of the actions shown in . Fig. 9 The actions of may be performed as part of a method. Alternatively, a non-transitory computer readable medium may include instructions that, when executed by one or more processes, cause a computing device to perform Fig. 9 In some other embodiments, the system may perform Fig. 9 action.

[0141] like Fig. 9As shown in , series of actions 900 includes action 910 of generating a color edge map using a trained color edge prediction neural network. In particular, action 910 includes: generating a color edge map for a grayscale digital image using a color edge prediction neural network, wherein the color edge map includes color edges. Action 910 also includes the action of generating a color edge map by: generating an initial color edge map including initial edges using a color edge prediction neural network; determining a confidence score for each of the initial edges using a color edge prediction neural network; and including the initial edge in the color edge map as a color edge in the color edge based on determining that the confidence score associated with one of the initial edges satisfies a color edge threshold. Action 910 also includes the action of generating a color edge map by: generating a color edge map from the initial color edge map using non-maximum suppression.

[0142] The series of actions 900 includes providing a color edge map, a color point input element, and a grayscale digital image for display 920. In particular, action 920 includes providing a colorized graphical user interface for display at a client device, the colorized graphical user interface including the color edge map, the color point input element, and the grayscale digital image.

[0143] The series of actions 900 also includes an action 930 of generating a colorized image using the trained edge-guided colorization neural network. In particular, action 930 includes generating a colorized image using the edge-guided colorization neural network based on the grayscale digital image, the color edge map, and the user interaction with the color point input element.

[0144] In at least one embodiment, action 930 also includes additional actions: generating a grayscale channel reflecting a grayscale digital image, a color edge map channel reflecting a color edge map, and a color point channel reflecting a user input; processing the grayscale channel and the color point channel using an edge-guided colorization neural network to generate a first feature vector via a first layer of the edge-guided colorization neural network; connecting the color edge map channel to the first feature vector to generate a feature vector modified by the edge map; and processing the feature vector modified by the edge map using a second layer of the edge-guided colorization neural network.

[0145] The series of actions 900 includes an action of providing the colorized image for display 940. In particular, action 940 includes providing the colorized image for display via a colorized graphical user interface.

[0146] In at least one embodiment, the series of actions 900 further includes additional actions: identifying a user interaction with the color edge modification element via a colored graphical user interface; and generating a modified edge map by modifying the color edge based on the user interaction with the color edge modification element.

[0147] The series of actions 900 may also include additional actions: providing the colorized image for display via a colorized graphical user interface; generating a modified edge map based on a user interaction with a color edge modification element; and generating a modified colorized image using an edge-guided colorization neural network based on the grayscale digital image, the modified edge map, and the user interaction with the color point input element. In addition, in at least one embodiment, the color edge modification element includes a color edge threshold element, and the additional actions further include: determining a modified edge threshold based on a user interaction with the color edge threshold element; and including a set of modified color edges in the modified edge map based on the modified edge threshold.

[0148] Additionally, in at least one embodiment, the color edge modification element includes an edge suggestion element, and the additional actions also include generating a modified edge map by: generating luminance edges of the grayscale digital image by utilizing a canny edge detector based on user interaction with the edge suggestion element; presenting the luminance edges via a colorized graphical user interface; and including one or more edges in the modified edge map based on user selection of one or more of the luminance edges.

[0149] In at least one embodiment, the color edge modification element includes a user-drawn edge element, and the additional actions further include generating a modified edge map by: generating a user-drawn edge based on a user interaction with the user-drawn edge element; and including the user-drawn edge in the modified edge map.

[0150] In addition to (or as an alternative to) the actions described above, in some embodiments, the series of actions 900 also includes steps for generating a colorized image from the grayscale digital image and the color points using color edges. Figure 3 , Figure 4B , Figure 5 and Figure 6B The described actions may include corresponding actions (or structures) for performing steps for generating a colorized image from a grayscale digital image and color points using color edges.

[0151] As discussed in more detail below, embodiments of the present disclosure may include or utilize a special purpose computer or a general purpose computer including computer hardware, such as, for example, one or more processors and system memory. Embodiments within the scope of the present disclosure also include physical computer-readable media and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more of the processes described herein may be implemented at least in part as instructions implemented in a non-transient computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transient computer-readable medium (e.g., a memory, etc.) and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.

[0152] Computer-readable media can be any available media that can be accessed by a general-purpose computer system or a special-purpose computer system. A computer-readable medium that stores computer-executable instructions is a non-transient computer-readable storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present disclosure may include at least two distinct types of computer-readable media: a non-transient computer-readable storage medium (device) and a transmission medium.

[0153] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, (e.g., RAM-based) solid-state drives ("SSD"), flash memory, phase-change memory ("PCM"), other types of memory or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code means in the form of computer-executable instructions or data structures and that can be accessed by a general purpose or special purpose computer.

[0154] A "network" is defined as one or more data links that enable electronic data to be transported between computer systems and / or modules and / or other electronic devices. When information is transferred or provided to a computer via a network or another communication connection (either hardwired, wireless, or a combination of hardwired and wireless), the computer appropriately views the connection as a transmission medium. Transmission media may include networks and / or data links that can be used to carry desired program code means in the form of computer executable instructions or data structures and can be accessed by general-purpose or special-purpose computers. Combinations of the above should also be included within the scope of computer-readable media.

[0155] Further, upon reaching various computer system components, program code means in the form of computer executable instructions or data structures may be automatically transferred from a transmission medium to a non-transitory computer readable storage medium (device) (or vice versa). For example, computer executable instructions or data structures received over a network or data link may be buffered in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to computer system RAM and / or a low-volatility computer storage medium (device) at the computer system. Thus, it should be understood that a non-transitory computer readable storage medium (device) may be included in a computer system component that also (or even primarily) utilizes a transmission medium.

[0156] Computer executable instructions include instructions or data that, for example, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions when executed at a processor. In some embodiments, computer executable instructions are executed on a general-purpose computer to transform a general-purpose computer into a special-purpose computer that implements the elements of the present disclosure. Computer executable instructions can be, for example, binary, intermediate format instructions (such as assembly language) or even source code. Although the subject matter has been described in a language specific to structural features and / or method actions, it should be understood that the subject matter defined in the accompanying claims is not necessarily limited to the described features or the actions described above. On the contrary, the described features and actions are disclosed as example forms for implementing the claims.

[0157] Those skilled in the art will appreciate that the present disclosure can be practiced in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. The present disclosure can also be practiced in a distributed system environment, in which local computer systems and remote computer systems linked by a network (by a hardwired data link, a wireless data link, or a combination of a hardwired data link and a wireless data link) all perform tasks. In a distributed system environment, program modules can be located in both local memory storage devices and remote memory storage devices.

[0158] Embodiments of the present disclosure may also be implemented in a cloud computing environment. In this specification, "cloud computing" is defined as a model for implementing on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be employed in a market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources may be quickly configured via virtualization and published with low management effort or service provider interaction, and then scaled accordingly.

[0159] The cloud computing model may consist of various characteristics, such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured services, etc. The cloud computing model may also expose various service models, such as, for example, software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model may also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc. In this specification and in the claims, a "cloud computing environment" is an environment that employs cloud computing.

[0160] Fig.10 1000 is a block diagram of a computing device 1000 that can be configured to perform one or more of the processes described above. It should be appreciated that one or more computing devices (such as computing device 1000) can implement edge-guided colorization system 106 and image editing system 104. Fig.10 As shown, computing device 1000 may include processor 1002, memory 1004, storage device 1006, I / O interface 1008, and communication interface 1010, which may be communicatively coupled via communication infrastructure 1012. In some embodiments, computing device 1000 may include more than one embodiment of the present invention. Fig.10 The components shown in the figure are fewer or more. Fig.10 Components of computing device 1000 are shown in FIG.

[0161] In one or more embodiments, the processor 1002 includes hardware for executing instructions, such as hardware constituting a computer program. As an example and not limitation, in order to execute instructions for dynamically modifying a workflow, the processor 1002 may fetch (or obtain) instructions from an internal register, an internal cache, a memory 1004, or a storage device 1006, and decode and execute these instructions. The memory 1004 may be a volatile memory or a non-volatile memory for storing data, metadata, and programs executed by (multiple) processors. The storage device 1006 includes a storage device for storing data or instructions for executing the methods described herein, such as a hard disk, a flash drive, or other digital storage device.

[0162] The I / O interface 1008 allows a user to provide input to the computing device 1000, receive output from the computing device 1000, and otherwise transfer data to and receive data from the computing device 1000. The I / O interface 1008 may include a mouse, a keypad or keyboard, a touch screen, a camera, an optical scanner, a network interface, a modem, other known I / O devices, or a combination of such I / O interfaces. The I / O interface 1008 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O interface 1008 is configured to provide graphics data to the display for presentation to the user. The graphics data may represent one or more graphical user interfaces and / or any other graphical content such as may serve a particular implementation.

[0163] The communication interface 1010 may include hardware, software, or both. In any case, the communication interface 1010 may provide one or more interfaces for communication between the computing device 1000 and one or more other computing devices or networks, such as, for example, packet-based communication. By way of example and not limitation, the communication interface 1010 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired-based network; or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as WI-FI.

[0164] Additionally, the communication interface 1010 can support communication with various types of wired or wireless networks. The communication interface 1010 can also support communication using various communication protocols. The communication infrastructure 1012 can also include hardware, software, or both that couple the components of the computing device 1000 to each other. For example, the communication interface 1010 can use one or more networks and / or protocols to enable multiple computing devices connected through a specific infrastructure to communicate with each other to perform one or more aspects of the process described herein. For illustration, the digital content activity management process can allow multiple devices (e.g., client devices and server devices) to exchange information using various communication networks and protocols to share information, such as digital messages, user interaction information, participation metrics, or activity management resources.

[0165] In the foregoing description, the present disclosure has been described with reference to specific exemplary embodiments of the present disclosure. Various embodiments and aspects of the present disclosure are described with reference to the details discussed herein, and the accompanying drawings illustrate various embodiments. The above description and drawings are illustrative of the present disclosure and should not be construed as limiting the present disclosure. Several specific details are described to provide a thorough understanding of the various embodiments of the present disclosure.

[0166] The present disclosure may be implemented in other specific forms without departing from its spirit or essential characteristics. The described embodiments should be considered in all respects to be merely illustrative and not restrictive. For example, the method described herein may be performed with fewer or more steps / actions, or the steps / actions may be performed in different orders. Additionally, the steps / actions described herein may be repeated or performed in parallel with each other, or may be repeated or performed in parallel with different instances of the same or similar steps / actions. Therefore, the scope of the present application is indicated by the appended claims rather than the foregoing description. All changes within the meaning and equivalent scope of the claims should be included within their scope.

Claims

1. A non-transitory computer-readable medium for generating a colorized image, the non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause a computing device to: generating a color edge map for the grayscale digital image using a color edge prediction neural network, wherein the color edge map includes color edges; providing a colorized graphical user interface for display at a client device, the colorized graphical user interface comprising the color edge map, the color point input element, and the grayscale digital image; generating a colored image using an edge-guided colorization neural network based on the grayscale digital image, the color edge map, and a user interaction with the color point input element; as well as The colorized image is provided for display via the colorized graphical user interface.

2. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: providing the color edge map together with color edge modification elements for display as part of the colorized graphical user interface; generating a modified edge map based on user interaction with the color edge modification element; and A modified colorized image is generated using the edge-guided colorization neural network based on the grayscale digital image, the modified edge map, and the user interaction with the color point input element.

3. The non-transitory computer-readable medium of claim 2, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the color edge map by: Using the color edge prediction neural network to generate an initial color edge map including initial edges; determining a confidence score for each of the initial edges using the color edge prediction neural network; and Based on determining that a confidence score associated with one of the initial edges satisfies a color edge threshold, the one initial edge is included in the color edge map as one of the color edges.

4. The non-transitory computer readable medium of claim 3, wherein the color edge modification element comprises a color edge threshold element, and the non-transitory computer readable medium further comprises instructions that, when executed by the at least one processor, cause the computing device to generate the modified edge map by: determining a modified edge threshold based on user interaction with the color edge threshold element; and A modified set of color edges is included in the modified edge map based on the modified edge threshold.

5. The non-transitory computer-readable medium of claim 2, wherein the color edge modification element comprises an edge suggestion element, and the non-transitory computer-readable medium further comprises instructions that, when executed by the at least one processor, cause the computing device to generate the modified edge map by: generating illumination edges of the grayscale digital image by utilizing a canny edge detector based on user interaction with the edge proposal component; presenting the illumination edge via the colorized graphical user interface; and Based on a user selection of one or more of the illumination edges, the one or more edges are included in the modified edge map.

6. The non-transitory computer-readable medium of claim 4, wherein the color edge modification element comprises a user-drawn edge element, and the non-transitory computer-readable medium further comprises instructions that, when executed by the at least one processor, cause the computing device to generate the modified edge map by: generating a user drawn edge based on user interaction with the user drawn edge element; and The user drawn edges are included in the modified edge map.

7. The non-transitory computer-readable medium of claim 3, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the color edge map by utilizing non-maximum suppression to generate the color edge map from the initial color edge map.

8. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to train the edge-guided colorization neural network by: applying a canny edge detector to the ground truth color corresponding to the ground truth color image to generate canny edges; applying the edge-guided colorization neural network to a training grayscale image corresponding to the ground truth color image, training color points, and the canny edges; generating a predicted color using the edge-guided colorization neural network; and Parameters of the edge-guided colorization neural network are modified by comparing the predicted color to the ground truth color.

9. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the colorized image by: generating a grayscale channel reflecting the grayscale digital image, a color edge map channel reflecting the color edge map, and a color point channel reflecting user input; Processing the grayscale channel and the color point channel using the edge-guided colorization neural network to generate a first feature vector via a first layer of the edge-guided colorization neural network; Joining the color edge map channel to the first feature vector to generate an edge map modified feature vector; and The modified feature vector of the edge map is processed using a second layer of the edge-guided colorization neural network.

10. A system for generating a colorized image, comprising: one or more memory devices storing the grayscale digital image, the color edge prediction neural network, and the edge-guided colorization neural network; At least one server is configured to enable the system to: Generate a color edge map including color edges for the grayscale digital image using the color edge prediction neural network; providing a colorized graphical user interface for display at a client device, the colorized graphical user interface comprising the color edge map, a color point input element, and a color edge modification element for modifying the color edge; identifying, via the colored graphical user interface, a user interaction with the color edge modification element; generating a modified edge map by modifying color edges based on the user interaction with the color edge modification element; as well as A colored image is generated using the edge-guided colorization neural network based on the modified edge map and user interaction with the color point input element.

11. The system of claim 10, further comprising instructions that, when executed by the at least one server, cause the system to generate the color edge map by: Using the color edge prediction neural network to generate an initial color edge map including initial edges; determining a confidence score for each of the initial edges using the color edge prediction neural network; and Based on determining that a confidence score associated with one of the initial edges satisfies a color edge threshold, the one initial edge is included in the color edge map as one of the color edges.

12. The system of claim 11, wherein the color edge modification element comprises a color edge threshold element, and the system further comprises instructions that, when executed by the at least one server, cause the system to generate the modified edge map by: determining a modified edge threshold based on user interaction with the color edge threshold element; and A modified set of color edges is included in the modified edge map based on the modified edge threshold.

13. The system of claim 10, wherein the color edge modification element comprises an edge suggestion element, and the system further comprises instructions that, when executed by the at least one server, cause the system to generate the modified edge map by: generating illumination edges of the grayscale digital image by utilizing a canny edge detector based on user interaction with the edge proposal component; presenting the illumination edge via the colorized graphical user interface; and Based on a user selection of one or more of the illumination edges, the one or more edges are included in the modified edge map.

14. The system of claim 12, wherein the color edge modification element comprises a user-drawn edge element, and the system further comprises instructions that, when executed by the at least one server, cause the system to generate the modified edge map by: generating a user drawn edge based on user interaction with the user drawn edge element; and The user drawn edges are included in the modified edge map.

15. The system of claim 11, further comprising instructions that, when executed by the at least one server, cause the system to generate the color edge map by: utilizing non-maximum suppression to generate the color edge map from the initial color edge map.

16. The system of claim 10, further comprising instructions that, when executed by the at least one server, cause the system to train the edge-guided colorization neural network by: applying a canny edge detector to the ground truth color corresponding to the ground truth color image to generate canny edges; applying the edge-guided colorization neural network to a training grayscale image corresponding to the ground truth color image, training color points, and the canny edges; generating a predicted color using the edge-guided colorization neural network; and Parameters of the edge-guided colorization neural network are modified by comparing the predicted color to the ground truth color.

17. The system of claim 10, further comprising instructions that, when executed by the at least one server, cause the system to generate the colored image by: coupling the color edge map into a plurality of layers of the edge-guided colorization neural network.

18. A computer-implemented method for generating an enhanced color digital image in a digital media environment for editing and generating digital images, comprising: generating a color edge map for the grayscale digital image using a color edge prediction neural network, wherein the color edge map includes color edges; providing a colorized graphical user interface for display at a client device, the colorized graphical user interface comprising the color edge map, the color point input element, and the grayscale digital image; generating a colored image using an edge-guided colorization neural network based on the grayscale digital image, the color edge map, and a user interaction with the color point input element; as well as The colorized image is provided for display via the colorized graphical user interface.

19. The method according to claim 18, further comprising: providing the color edge map together with color edge modification elements for display as part of the colorized graphical user interface; generating a modified edge map based on user interaction with the color edge modification element; and A modified colorized image is generated using the edge-guided colorization neural network based on the grayscale digital image, the modified edge map, and the user interaction with the color point input element.

20. The method of claim 19, further comprising generating the color edge map by: Using the color edge prediction neural network to generate an initial color edge map including initial edges; determining a confidence score for each of the initial edges using the color edge prediction neural network; and Based on determining that a confidence score associated with one of the initial edges satisfies a color edge threshold, the one initial edge is included in the color edge map as one of the color edges.

Images