Performing interactive digital image operations with modified machine learning models

Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media for performing interactive digital image editing operations with machine learning models and feature backpropagation refinement layers. For example, the disclosed systems perform interactive digital image editing operations by incorporating a feature backpropagation refinement layer into a non-interactive machine learning model that utilizes a consistency loss to adjust the feature backpropagation refinement layer according to one or more user interactions. In some embodiments, the disclosed systems utilize a feature backpropagation refinement layer that includes a bias sublayer for localizing changes to a digital image and a convolution sublayer for per-channel scaling and feature combination across channels. In some cases, the disclosed systems utilize a consistency loss that facilitates localized modifications to a digital image based on a distance of various pixels or features from a user interaction.

Description

Perform interactive digital image manipulation using a modified machine learning model. Background Technology

[0001] Existing hardware and software platforms provide systems and methods for modifying digital images using machine learning models. For example, some existing image modification systems utilize various types of machine learning models to generate or modify digital images for different tasks such as semantic segmentation, depth estimation, and style transfer. However, despite these advances, many conventional image modification systems still have significant shortcomings, particularly in terms of flexibility, accuracy, and efficiency.

[0002] As just suggested, many conventional image editing systems are inflexible. In particular, existing systems are typically limited to generating or modifying digital images automatically by analyzing input digital images to produce output digital images. However, due to the rigid nature of many neural network architectures, conventional systems are limited to manipulating or modifying digital images using only the learned parameters of their pre-trained layers. In fact, many existing systems utilize machine learning models that are trained as black boxes, where user interaction is not part of the training process. Even for existing systems that attempt to combine user interaction with machine learning models, these systems typically require training the model based on user interaction from the outset and cannot adapt the automatic (e.g., non-interactive) model to interactive use.

[0003] Furthermore, some conventional image editing systems inaccurately generate modified digital images. Specifically, many existing systems generate modified digital images that do not accurately reflect the relevant digital image editing task. For example, if a client attempts to segment a specific object depicted in a digital image, conventional systems often incorrectly include or exclude certain pixels. Using black-box machine learning models, conventional systems cannot leverage learned neural network features to correct the neural network's predictions.

[0004] Therefore, conventional systems typically generate predictive modifications to digital images via neural networks and then rely on separate image editing tools for users to manually correct any errors in the predictions. However, this approach leads to significant inefficiencies. In fact, since image editing tools are not part of the neural network, these separate tools cannot utilize the learned features of the neural network to intelligently correct the generated predictions. Instead, these tools require substantial user interaction and user interfaces, which increases the time and computational resources needed to generate the modified digital images. Furthermore, because these image editing tools rely almost entirely on client-side device interaction to refine the digital images, the resulting modifications are often still inaccurate and imprecise.

[0005] For existing systems that attempt to combine user interaction with machine learning models, these systems still frequently generate inaccurate digital images. In fact, the model architectures used by some existing systems do not accurately (or cannot) account for differences in location across the digital image when modifying it in response to user interaction. Therefore, based on receiving user interaction to modify a digital image generated by a machine learning model, conventional systems typically generate modified digital images by using the machine learning model to change pixels across the entire digital image, even if the user interaction only instructs the selection of a few pixels within the initial digital image to be changed.

[0006] Therefore, conventional image editing systems have several drawbacks. Summary of the Invention

[0007] This disclosure describes one or more embodiments of systems, methods, and non-transitory computer-readable media that utilize a machine learning model architecture including a feature backpropagation refinement layer that performs interactive digital image editing operations to address one or more of the aforementioned or other problems in the art. In practice, in some embodiments, the disclosed system can utilize a non-interactive machine learning model to perform interactive digital image editing based on user input. For example, the disclosed system performs interactive digital image editing operations by incorporating a feature backpropagation refinement layer into a machine learning model that scales and / or biases features generated by learned parameters of the machine learning model based on interactive user input. For example, the disclosed system can refine segmentation maps, image mattes, depth maps, restored digital images, or stylized digital images to align with user input based on user input that intelligently emphasizes features generated by learned parameters of a neural network.

[0008] In some embodiments, the disclosed system utilizes a feature backpropagation refinement layer comprising a bias sublayer for localizing changes to the digital image and a convolutional sublayer for channel-wise scaling and feature emphasis across channels. In some cases, the disclosed system utilizes a consistency loss that facilitates modification of the digital image based on the distance of various pixels or features from the user interaction. By utilizing the described feature backpropagation refinement layer and / or consistency loss, the disclosed system can flexibly, efficiently, and accurately generate enhanced digital images by refining the digital image output locally in response to user interaction, even in machine learning models not trained to adapt to user input.

[0009] Additional features and advantages of one or more embodiments of this disclosure are set forth in the following description and will be apparent in part from the description, or may be learned by practice of these exemplary embodiments. Attached Figure Description

[0010] This disclosure describes one or more embodiments of the present disclosure with additional specificity and detail by referring to the accompanying drawings. The following paragraphs briefly describe these drawings, in which:

[0011] Figure 1 illustrates an example system environment in which an interactive image editing system according to one or more embodiments operates;

[0012] Figure 2 illustrates an overview of generating a refined digital image through backpropagation of parameters of a refinement layer based on modified features in response to user interaction, according to one or more embodiments.

[0013] Figures 3A-3B illustrate example backpropagation for modifying the parameters of the feature backpropagation refinement layer according to one or more embodiments;

[0014] Figures 4A-4B illustrate example architectures of feature backpropagation refinement layers according to one or more embodiments;

[0015] Figures 5A and 5B illustrate examples of determining consistency loss according to one or more embodiments;

[0016] Figures 6A-6C illustrate examples of generating refined digital images in the form of segmented digital images according to one or more embodiments;

[0017] Figures 7A-7B illustrate examples of refined digital images generated in the form of digital image matting according to one or more embodiments;

[0018] Figures 8A-8C illustrate examples of generating refined digital images in the form of digital image depth maps according to one or more embodiments;

[0019] Figures 9A-9C show example diagrams of experimental results from an interactive image editing system according to one or more embodiments;

[0020] Figure 10 illustrates a schematic diagram of an interactive image editing system according to one or more embodiments;

[0021] Figure 11 illustrates a flowchart of a series of actions according to one or more embodiments for generating a refined digital image using a digital image neural network including a feature backpropagation refinement layer and / or consistency loss; and

[0022] Figure 12 shows a block diagram of an example computing device according to one or more embodiments. Detailed Implementation

[0023] This disclosure describes one or more embodiments of an interactive image editing system that utilizes a feature backpropagation refinement layer of a machine learning model to accurately and flexibly modify digital images using interactive user input. Specifically, in some embodiments, the interactive image editing system modifies a non-interactive machine learning model, such as a non-interactive neural network (e.g., a neural network not trained based on user interaction), to adapt to user input in generating refined digital images (e.g., image segmentation, image matting, depth maps, etc.). To convert a non-interactive neural network into an interactive neural network, in some embodiments, the interactive image editing system utilizes a modified neural network architecture that includes a dedicated feature backpropagation refinement layer. For example, the feature backpropagation refinement layer includes additional components, such as bias sublayers and convolutional sublayers, each with parameters that can be modified in response to user interaction. In some cases, the interactive image editing system further (or alternatively) utilizes a consistency loss to inform modifications to the model parameters based on user interaction.

[0024] As mentioned earlier, in one or more embodiments, the interactive image editing system modifies or refines digital images based on user interaction with the digital image. Specifically, in some embodiments, the interactive image editing system incorporates a feature backpropagation refinement layer and utilizes consistency loss to modify the output of a typically non-interactive neural network architecture. Thus, in some embodiments, the interactive image editing system transforms or converts a non-interactive neural network into an interactive neural network in response to user interaction for modifying or refining the digital image.

[0025] In one or more embodiments, an interactive image editing system generates modified digital images from an initial digital image. For example, the interactive image editing system utilizes a neural network including layers trained for a specific task (e.g., semantic segmentation, image matting, depth estimation, hole filling, or style transfer) to generate modified digital images, such as segmentation maps, image matting, depth maps, restored digital images, or stylized digital images. In some cases, the neural network includes (non-interactive) layers trained for one or more of the aforementioned tasks, as well as other (interactive) layers, such as a feature backpropagation refinement layer that includes bias sublayers and convolutional sublayers tunable based on user interaction.

[0026] Furthermore, in some embodiments, the interactive image editing system receives or detects user interaction involving the selection of one or more pixels of a modified digital image. In response to the user interaction, in some embodiments, the interactive image editing system updates or modifies the bias parameters of the bias sublayer and the scaling parameters of the convolution sublayer. For example, in response to the user interaction, the interactive image editing system backpropagates to modify the bias parameters of the bias sublayer and the scaling parameters of the convolution sublayer.

[0027] In some cases, interactive image editing systems backpropagate to reduce loss metrics associated with user interactions. For example, an interactive image editing system may determine cross-entropy loss, L1 loss, or L2 loss based on user interactions (e.g., a user interaction identifies a set of pixels that are mislabeled, resulting in a cross-entropy loss corresponding to the user interaction). Furthermore, an interactive image editing system may determine consistency loss based on the corresponding distance from the user interaction, penalizing or imposing modifications or refinements on pixels (or their corresponding features). In practice, in some cases, the interactive image editing system applies a consistency loss function such that pixels (or features) farther from the user interaction are less likely to change during backpropagation. Additionally, in one or more embodiments, the interactive image editing system backpropagates to reduce loss metrics (e.g., reduce cross-entropy loss and consistency loss) by modifying the bias parameters of the bias sublayers and the scaling parameters of the convolutional sublayers.

[0028] In one or more embodiments, the interactive image editing system further generates a refined digital image. For example, the interactive image editing system utilizes modified parameters of the feature backpropagation refinement layer (e.g., modified bias parameters of the bias sublayer and modified scaling parameters of the convolution sublayer) and learned parameters of other neural network layers to generate the refined digital image. In some cases, the interactive image editing system generates the refined digital image by correcting or modifying one or more pixels from the modified digital image (e.g., a digital image generated prior to the most recent user interaction).

[0029] For example, an interactive image editing system generates refined digital images for one or more tasks, such as semantic segmentation, image matting, depth mapping, hole filling, or style transfer. In practice, in some embodiments, the interactive image editing system utilizes different neural networks for each task, where the neural networks are trained for their specific task, including layers and neurons tuned to generate specific outputs. For example, an interactive image editing system might use a semantic segmentation neural network for semantic segmentation, an image matting neural network for image matting, a depth mapping neural network for depth mapping, a hole filling neural network (e.g., a generative adversarial neural network), or a style transfer neural network for style transfer. As previously mentioned, in some cases, each of the aforementioned neural networks includes a feature backpropagation refinement layer and / or consistency loss for interactive applications, even when other layers of the neural network are not interactive.

[0030] As described above, embodiments of interactive image editing systems can offer several advantages over conventional image modification systems. For example, embodiments of interactive image editing systems can improve flexibility compared to conventional image modification systems. While many existing systems utilize rigid black-box neural networks that generate output (e.g., in the absence of user interaction), embodiments of interactive image editing systems enhance (or utilize) neural networks or other machine learning models to adapt to interactive tasks. For example, unlike conventional systems, embodiments of interactive image editing systems utilize feature backpropagation refinement layers and / or consistency losses to adapt and apply non-interactive neural networks to update or refine digital images in response to user interaction at runtime.

[0031] Even compared to existing systems that train neural networks based on user interaction, implementations of interactive image editing systems still offer improved flexibility. For example, some previous systems have learned parameters trained based on historical user input, but apply these learned parameters to new user input. In contrast, interactive image editing systems can modify the parameters of the feature backpropagation refinement layer at runtime to align the modified digital image with the user input. Therefore, interactive image editing systems can proactively and dynamically modify the internal workings of the neural network as it is applied. Furthermore, unlike previous systems that are strictly fixed to specific use cases such as semantic segmentation (e.g., as a result of model training), interactive image editing systems utilize feature backpropagation refinement layers that can be adapted to many different neural network architectures for many different image editing tasks, such as semantic segmentation, depth estimation, style transfer, hole filling, and image matting.

[0032] Furthermore, implementations of interactive image editing systems offer improved accuracy compared to conventional image editing systems. Specifically, unlike conventional systems that rely on standalone image editing tools that cannot utilize learned neural network features for image editing, interactive image editing systems incorporate feature backpropagation refinement layers and / or consistency losses to leverage learned features to generate refined digital images. In effect, instead of relying on standalone tools with independent features to correct predictions from non-interactive neural networks, interactive image editing systems utilize modified neural networks that include feature backpropagation refinement layers with updatable parameters to modify internal network features in response to user interaction. This approach also offers the benefit of increased efficiency by avoiding the excessive and unnecessary time, user interaction, and user interface required by conventional systems that modify digital images using standalone, non-intelligent tools. In fact, with just a click, interactive image editing systems can refine digital image output to correct errors in segmentation, matting, depth mapping, hole filling, style transfer, or other image editing tasks.

[0033] As another example of improved accuracy, while some prior systems may incorporate user interactions, these systems still inaccurately modify the pixels of the entire digital image based on user interactions (e.g., by modifying pixels farther from the user interaction). On the other hand, some embodiments of interactive image editing systems utilize bias maps (e.g., as part of a feature backpropagation refinement layer) and / or consistency losses to more accurately modify target pixels of the digital image. For example, in response to user interactions indicating one or more pixels of the digital image, interactive image editing systems utilize bias maps and / or consistency losses to accurately update or modify the digital image by penalizing changes to pixels (or corresponding features) based on the corresponding distance from the user interaction.

[0034] As suggested in the foregoing discussion, this disclosure utilizes a variety of terms to describe the features and benefits of interactive image editing systems. Additional details regarding the meaning of these terms as used in this disclosure are provided below. Specifically, the term "neural network" refers to a machine learning model that can be trained and / or tuned based on inputs to determine a classification or approximate an unknown function. For example, a neural network includes interconnected models of artificial neurons (e.g., hierarchically organized) that pass and learn to approximate complex functions and generate outputs (e.g., generated digital images) based on multiple inputs provided to the neural network. In some cases, a neural network refers to an algorithm (or a set of algorithms) that implements deep learning techniques to model high-level abstractions in data. For example, a neural network can include convolutional neural networks, recurrent neural networks (e.g., LSTM), graph neural networks, or generative adversarial neural networks.

[0035] For example, the term "non-interactive neural network" refers to a neural network with parameters that have not been learned or trained based on user interaction (e.g., user interaction input channels). For instance, a non-interactive neural network includes one or more layers that operate independently of user interaction input. Specifically, a non-interactive neural network may include a neural network that has not been trained based on sample user interaction (e.g., no channels or layers in the neural network process or analyze user interaction for prediction). For example, a non-interactive neural network includes one or more neural network layers with learned parameters that were determined during the training process of the neural network without utilizing sample user interaction. A non-interactive neural network may include a feature backpropagation refinement layer that adds interactive components capable of modifying internal network parameters in response to user interaction.

[0036] As just mentioned, in some embodiments, interactive image editing systems utilize modified neural network architectures that include feature backpropagation refinement layers. As used herein, the term "feature backpropagation refinement layer" refers to a layer within a neural network that modifies or updates parameters based on user interaction (e.g., runtime user interaction). For example, a feature backpropagation refinement layer includes bias parameters and scaling parameters for bias and scaling features (e.g., for per-channel bias and / or per-channel scaling) for various digital image tasks. In practice, in some cases, the feature backpropagation refinement layer includes bias sublayers and convolutional sublayers that begin with initialized, unlearned parameters (e.g., zero or one), which are then modified based on user input to emphasize intermediate (e.g., latent) features in the neural network.

[0037] As used herein, the term "bias sublayer" refers to a sublayer within a feature backpropagation refinement layer that includes bias parameters for biasing features based on user interaction. For example, a bias sublayer may refer to a channel-wise bias sublayer that biases or emphasizes feature modifications across corresponding channels of features extracted from a digit image. As another example, a bias sublayer may refer to a bias map that localizes changes to neural network features based on the location of user interaction. To illustrate, a bias sublayer may add bias parameter values ​​to input features to emphasize / de-emphasize them. In the above or other examples, the bias sublayer includes bias parameters that can be modified to reduce or otherwise update the loss metric. For example, "bias parameters" refer to neural network parameters, such as the weights or biases of the bias sublayer, which are modifiable based on user interaction to adjust how extracted (e.g., latent) features within the neural network are biased or localized.

[0038] In practice, as used herein, the term "bias map" refers to a bias sublayer (e.g., a two-dimensional map) that localizes changes to neural network features at specific locations corresponding to a digital image. For example, a bias map can apply different bias parameters to different pixels or feature regions in a digital image. In some cases, the bias map includes dimensions corresponding to the dimensions of the digital image (or features extracted from the digital image). Therefore, in some situations, the bias map applies different bias parameters to different regions of the feature map within the neural network.

[0039] Furthermore, as used herein, the term "convolutional sublayer" refers to a sublayer that is part of a feature backpropagation refinement layer and includes scaling parameters for scaling neural network features. For example, a convolutional sublayer includes scaling parameters for increasing, decreasing, or otherwise scaling feature weights to generate a digital image of neural network features (e.g., by adding or otherwise combining feature weights). To illustrate, a convolutional layer can multiply feature weights by scaling parameters applied to input features to emphasize / de-emphasize that feature within the neural network. The convolutional layer can then add / combine the results to generate output features. In some cases, the convolutional sublayer can be modified on a channel-by-channel basis to adjust the extracted feature channels. In some embodiments, the convolutional sublayer has a first dimension corresponding to the number of extracted feature channels and a second dimension corresponding to the dimension of the digital image from which features are extracted. Relatedly, as used herein, the term "scaling parameter" refers to neural network parameters, such as weights as part of the convolutional sublayer, which can be modified based on user interaction to adjust the scaling of extracted (e.g., latent) features. In some embodiments, convolutional sublayers and / or scaling parameters are applied to localized areas or regions of the digital image, while the bias map is applied at a global level across the entire digital image. In other embodiments, the bias map is applied to localized regions, while the convolutional sublayers and / or scaling parameters are applied globally.

[0040] As described above, in one or more embodiments, the interactive image editing system determines consistency loss based on user interaction. As used herein, the term "consistency loss" refers to a loss metric based on one or more distances of extracted features (or corresponding pixels in a digital image) relative to features (or pixels) indicated by user interaction. For example, consistency loss penalizes changes or modifications to features or pixels based on relative distance from user interaction. In some cases, consistency loss reduces the likelihood of modifying features or pixels farther from user interaction, where features or pixels closer to user interaction are more likely to be modified. In this way, consistency loss improves the consistency between the initial output of the neural network and the modified / refined output regarding regions farther from a particular user input.

[0041] Interactive image editing systems can also utilize other loss measures. As used herein, loss measures can include metrics or values ​​that reflect errors, inconsistencies, or inaccuracies. For example, in addition to consistency loss, interactive image editing systems can also utilize relational losses (determined based on relationships indicated by one or more user interactions), such as cross-entropy loss, L1 loss, L2 loss, mean squared error, or another loss measure. In fact, as mentioned above, interactive image editing systems can combine consistency loss with one or more other losses. For example, an interactive image editing system can determine an L1 loss in response to a negative user interaction indicating incorrect pixel values ​​(e.g., incorrect segmentation) in a user-interactive region, and the interactive image editing system can use the L1 loss and consistency loss to correct pixel values ​​without distorting / modifying other parts of the digital image.

[0042] Additional details regarding the interactive image editing system will now be provided with reference to the accompanying drawings. For example, Figure 1 illustrates a schematic diagram of an example system environment for implementing an interactive image editing system 102 according to one or more embodiments. An overview of the interactive image editing system 102 will be described in conjunction with Figure 1. Subsequently, a more detailed description of the components and processes of the interactive image editing system 102 will be provided in conjunction with the following drawings.

[0043] As shown in the figure, the environment includes (multiple) servers 104, client devices 108, a database 112, and a network 114. Each component of the environment communicates via network 114, and network 114 is any suitable network through which computing devices communicate. An example network is discussed in more detail below with reference to Figure 12.

[0044] As described above, the environment includes client device 108. Client device 108 is one of a variety of computing devices, including smartphones, tablets, smart TVs, desktop computers, laptops, virtual reality devices, augmented reality devices, or another computing device as described with respect to Figure 12. Although Figure 1 shows a single instance of client device 108, in some embodiments, the environment includes multiple different client devices, each associated with a different user (e.g., a digital content editor). Client device 108 communicates with server(s) 104 via network 114. For example, client device 108 provides information to server(s) 104 instructing client device interactions (e.g., digital image selection, user interaction with one or more pixels of a digital image, or other input) and receives information from server(s) 104, such as modified and / or refined digital images. Thus, in some cases, interactive image editing system 102 on server(s) 104 provides and receives information based on client device interactions via client device 108.

[0045] As shown in Figure 1, client device 108 includes client application 110. Specifically, client application 110 is a web application, a native application installed on client device 108 (e.g., a mobile application, desktop application, etc.), or a cloud-based application, wherein all or part of its functionality is executed by server(s)104. Based on instructions from client application 110, client device 108 presents or displays information to the user, including digital images, such as segmented digital images, digital image matting, digital image depth maps, restored digital images, and / or stylized digital images.

[0046] As shown in Figure 1, the environment includes multiple servers 104. The multiple servers 104 generate, track, store, process, receive, and transmit electronic data, such as indications of one or more pixels of a digital image. For example, the multiple servers 104 receive data from a client device 108 in the form of instructions for interaction with the client device to correct or update one or more pixels of the digital image. In response, the multiple servers 104 transmit data to the client device 108 so that the client device 108 can interactively display or present a refined digital image.

[0047] In some embodiments, servers 104 communicate with client devices 108 to transmit and / or receive data via network 114. In some embodiments, servers 104 include distributed servers, comprising multiple server devices distributed across network 114 and located in different physical locations. Servers 104 may include content servers, application servers, communication servers, web hosting servers, multidimensional servers, or machine learning servers. Servers 104 may also access and utilize database 112 to store and retrieve information such as initial digital images, modified digital images generated prior to user interaction, refined digital images generated based on user interaction, and digital image neural networks including feature backpropagation refinement layers and / or consistency loss functions.

[0048] As further shown in Figure 1, the servers (multiple) 104 also include an interactive image editing system 102 as part of a digital content editing system 106. For example, in one or more implementations, the digital content editing system 106 may store, generate, modify, edit, enhance, provide, distribute, and / or share digital content, such as digital images. For example, the digital content editing system 106 provides tools to a client device 108 via a client application 110 to manipulate the pixels of a digital image to modify its appearance by adding, removing, or manipulating objects within the digital image. In some implementations, the digital content editing system 106 provides tools for refining a digital image by using a digital image neural network that includes a feature backpropagation refinement layer to correct one or more pixels based on user interaction.

[0049] In one or more embodiments, the servers(s)104 include all or part of the interactive image editing system 102. For example, the interactive image editing system 102 operates on the servers(s) to generate refined digital images based on user interaction. In some cases, the interactive image editing system 102 utilizes digital image neural network capabilities, including feature backpropagation refinement layers and / or consistency losses, locally on the servers(s)104 or from another network location (e.g., database 112), to generate refined digital images based on user interaction.

[0050] In some cases, client device 108 includes all or part of interactive image editing system 102. For example, client device 108 may generate, acquire (e.g., download), or utilize one or more aspects of interactive image editing system 102, such as a digital image neural network including a feature backpropagation refinement layer and / or consistency loss function from server(s)104. In practice, in some implementations, as shown in Figure 1, interactive image editing system 102 is located in all or part of client device 108. For example, interactive image editing system 102 includes a web-hosted application that allows client device 108 to interact with server(s)104. For illustration, in one or more implementations, client device 108 accesses web pages supported and / or hosted by server(s)104.

[0051] Although Figure 1 illustrates a specific arrangement of the environment, in some embodiments, the environment has a different component arrangement and / or may have a completely different number or different groups of components. For example, as described above, the interactive image editing system 102 is implemented by a client device 108 (e.g., entirely or partially located on the client device 108). Furthermore, in one or more embodiments, the client device 108 bypasses the network 114 and communicates directly with the interactive image editing system 102. Additionally, in some embodiments, the environment includes a digital image neural network comprising feature backpropagation refinement layers and / or consistency loss functions stored in a database 112, maintained by the server(s) 104, the client device 108, or a third-party device.

[0052] As described above, in one or more embodiments, the interactive image editing system 102 generates a refined digital image based on user interaction using a digital image neural network including a feature backpropagation refinement layer and / or a consistency loss function (attached to other loss functions). Specifically, the interactive image editing system 102 generates a refined digital image by modifying or refining one or more pixels of the digital image according to parameters updated within the feature backpropagation refinement layer in response to user interaction. Figure 2 illustrates an overview of generating a refined digital image based on user interaction using a neural network. Additional details regarding the various actions described with reference to Figure 2 are provided below with reference to the following figures.

[0053] As shown in Figure 2, the interactive image editing system 102 performs the action 202 of accessing a digital image. Specifically, the interactive image editing system 102 accesses a digital image from a database (e.g., database 112) or receives a digital image as an upload from a client device (e.g., client device 108). For example, the interactive image editing system 102 receives or detects user interaction from the client device 108 that selects a digital image. As shown, the interactive image editing system 102 accesses or receives a digital image depicting a barn in a field with trees in the background.

[0054] As further shown in Figure 2, the interactive image editing system 102 performs the action 204 of extracting features from a digital image. Specifically, the interactive image editing system 102 utilizes one or more layers of a digital image neural network to extract features. For example, the interactive image editing system 102 utilizes neural network layers with learned parameters to extract features (e.g., latent features). In some cases, as part of the training process, the interactive image editing system 102 utilizes a non-interactive neural network with one or more layers having parameters learned without user interaction. In effect, the interactive image editing system 102 extracts features that represent the digital image and are interpretable by the neural network but not necessarily interpretable by a human observer.

[0055] Furthermore, the interactive image editing system 102 performs action 206 to generate a modified digital image. More specifically, the interactive image editing system 102 generates a modified digital image from an initial digital image accessed via action 202. For example, the interactive image editing system 102 generates a modified digital image in the form of a segmented digital image, a digital image matting, a digital image depth map, a restored digital image, or a stylized digital image. To generate the modified digital image, the interactive image editing system 102 utilizes one or more layers of a neural network specific to the applicable task, such as a segmentation neural network, an image matting neural network, a depth mapping neural network, a restoration neural network, or a style transfer neural network.

[0056] In some cases, neural networks also include a feature backpropagation refinement layer, where unlearned parameters (e.g., unlearned bias parameters of the bias sublayer and unlearned scaling parameters of the convolution sublayer) are initialized with values ​​that do not affect the generation of the modified digital image before user interaction. For example, interactive image editing system 102 utilizes learned parameters from certain layers tailored for generating the modified digital image and passes features through the feature backpropagation refinement layer without affecting the modified digital image. In some cases, to prevent unwanted errors in the modified digital image, interactive image editing system 102 initializes the parameters of the feature backpropagation refinement layer (e.g., bias parameters and scaling parameters) to specific values. For example, interactive image editing system 102 initializes the scaling parameter to have a value of 1 (because multiplying by 1 keeps the feature unchanged) and the bias parameter to have a value of 0 (because adding 0 keeps the feature unchanged).

[0057] As further shown in Figure 2, the interactive image editing system 102 performs the action 208 of receiving user interaction. Specifically, the interactive image editing system 102 receives user interaction from the client device 108 in the form of selection of one or more pixels of a modified digital image. For example, the interactive image editing system 102 receives a click on a single pixel, a click-drag over multiple pixels, a box selection of a group of pixels, a first click on a first group of pixels and a second click on a second group of pixels, or some other form of user interaction with the modified digital image. In some embodiments, the interactive image editing system 102 receives user interaction to correct one or more pixels of a modified digital image that the user notices are incorrect or defective in some way. As shown, the interactive image editing system 102 receives user interaction indicating that the barn pixels are incorrectly segmented.

[0058] Furthermore, the interactive image editing system 102 performs an action 210 to determine consistency loss (or another loss metric) based on user interaction. Specifically, the interactive image editing system 102 determines the consistency loss associated with the user interaction. In practice, in response to receiving a user interaction, the interactive image editing system 102 determines the distances between features extracted from the digital image, and / or the distances between pixels corresponding to the extracted features. For example, the interactive image editing system 102 determines the distance from one or more pixels indicated by the user interaction to other pixels within the digital image (e.g., the initial digital image or a modified digital image). As another example, the interactive image editing system 102 determines the distances to features associated with the pixels indicated by the user interaction within a feature map (e.g., those extracted from the initial digital image), and features associated with other pixels in the digital image within the feature map.

[0059] In one or more embodiments, action 210 involves determining a loss other than consistency loss. For example, interactive image editing system 102 determines L1 loss, L2 loss, cross-entropy loss, or other relational losses to encourage changes in pixel values ​​associated with user interaction. In some cases, interactive image editing system 102 uses L1 loss (or some other loss) in conjunction with negative user interaction to correct pixel values ​​of a selected pixel (and its surrounding area). For example, in response to a negative user interaction indicating incorrect pixel values, interactive image editing system 102 uses L1 loss and consistency loss to encourage modification of certain pixel values. In practice, L1 loss can encourage changes to pixel values ​​in the digital image that are the same as (or similar to) the pixel value of the selected pixel. By also using consistency loss in conjunction with L1 loss, interactive image editing system 102 can prevent changes to pixel values ​​of pixels farther from the selected pixel, even if those pixel values ​​are very similar to the pixel value of the selected pixel. In other cases, the interactive image editing system 102 uses different losses and / or different types of user interactions (e.g., positive user interactions indicating the correct pixel value and / or user interactions regarding the relationship between pixels in successive user interactions).

[0060] Based on the distance to a feature or pixel from user interaction, the interactive image editing system 102 determines a consistency loss to penalize changes or modifications based on distance. For example, the interactive image editing system 102 uses consistency loss to reduce the likelihood of altering features or pixels farther from user interaction. In some cases, the interactive image editing system 102 penalizes the refinement of a pixel or feature proportional to its distance from user interaction. In these or other cases, the interactive image editing system 102 utilizes different relationships, such as a threshold distance. For example, the interactive image editing system 102 penalizes the refinement of features or pixels beyond a threshold distance from user interaction. In these or other embodiments, the interactive image editing system 102 utilizes consistency loss for features generated by intermediate layers of a digital image neural network, and in some cases, utilizes multiple consistency loss networks for features generated at multiple different locations in the digital image neural network.

[0061] As further shown in Figure 2, the interactive image editing system 102 performs an action 212 to modify the parameters of the feature backpropagation refinement layer. Specifically, the interactive image editing system 102 modifies the parameters to reduce the total loss metric (e.g., consistency loss and / or other losses). For example, the interactive image editing system 102 backpropagates to adjust or update the bias parameters of the bias sublayer and / or the scaling parameters of the convolutional sublayer. In some cases, the interactive image editing system 102 modifies the bias parameters and convolutional layers to reduce the loss metric determined in response to user interaction.

[0062] Additionally, the interactive image editing system 102 performs the action 214 of generating a refined digital image. More specifically, the interactive image editing system 102 utilizes a digital image neural network to generate a refined digital image from the modified digital image. In fact, in addition to the parameters of one or more other layers of the digital image neural network, the interactive image editing system 102 also utilizes modified bias parameters and modified scaling parameters to generate the refined digital image. For example, the interactive image editing system 102 generates a refined digital image with corrected pixels. As shown, the refined digital image correctly segments the barn pixels that were initially missegmented in the modified digital image.

[0063] As described above, in some embodiments, the interactive image editing system 102 utilizes modified parameters of a feature backpropagation refinement layer within a neural network to generate a refined digital image. Specifically, the interactive image editing system 102 updates or modifies the parameters of the feature backpropagation refinement layer based on user interaction to correct one or more pixels of the modified digital image. Figures 3A-3B illustrate backpropagation according to one or more embodiments to update the parameters of the feature backpropagation refinement layer to generate a refined digital image.

[0064] As shown in Figure 3A, the interactive image editing system 102 utilizes a digital image neural network 304 to analyze or process the digital image 302. Specifically, the interactive image editing system 102 accesses or receives the digital image 302 and inputs it into the digital image neural network 304. For example, the interactive image editing system 102 utilizes one or more layers 306 and 314 with learned parameters determined during training. Specifically, the interactive image editing system 102 utilizes a first set of layers 306 to extract features from the digital image 302. In some other embodiments, the interactive image editing system 102 utilizes digital image neural networks with different architectures (e.g., different structures for feature backpropagation refinement layers).

[0065] Furthermore, the digital image neural network 304 transmits information or data between layers to generate a modified digital image 316. More specifically, the digital image neural network 304 passes the extracted features from the first set of layers 306 to the feature backpropagation refinement layer 308. The feature backpropagation refinement layer 308 then analyzes the extracted features using a bias sublayer 310 and a convolutional sublayer 312. As described above, the interactive image editing system 102 initializes the bias parameters of the bias sublayer 310 and further initializes the scaling parameters of the convolutional sublayer 312. For example, the interactive image editing system 102 initializes the bias parameters (e.g., to 0) and the scaling parameters (e.g., to 1) to preserve or avoid modifying the values ​​of the extracted features. In some embodiments, the bias sublayer 310 processes the extracted features using unlearned bias parameters initialized to 0 (e.g., adding 0 to the extracted features), keeping the extracted features unchanged. Furthermore, the convolutional sublayer 312 processes the extracted features using an unlearned scaling parameter initialized to 1 (e.g., multiplying the extracted features by 1) to keep the extracted features unchanged.

[0066] As further shown in Figure 3A, the digital image neural network 304 further passes the extracted features from the feature backpropagation refinement layer 308 to a set of layers 314. The additional set of layers 314 then processes the extracted features to generate a modified digital image 316. As shown, the interactive image editing system 102 uses the digital image neural network 304 to generate the modified digital image 316 in the form of a segmented digital image. Also as shown, after the first pass through the digital image neural network 304, some pixels in the modified digital image 316 of the barn are incorrectly segmented (e.g., the light gray pixels on the left side of the barn).

[0067] As also shown in Figure 3A, the interactive image editing system 102 receives user interaction 318. More specifically, the interactive image editing system 102 receives user interaction that selects or indicates one or more pixels of a modified digital image. For example, the interactive image editing system 102 receives user interaction in the form of a click or tap, which selects one or more pixels within an incorrectly segmented portion of a barn. As another example, the interactive image editing system 102 receives user interaction that specifically outlines a set of pixels within an incorrectly segmented portion of a barn.

[0068] As shown in Figure 3B, the interactive image editing system 102 performs backpropagation 320 in response to user interaction 318. Specifically, the interactive image editing system 102 receives user interaction 318 and performs backpropagation in response to user interaction 318 to update or modify the parameters of the digital image neural network 304. Specifically, the interactive image editing system 102 performs backpropagation to modify internal parameters such as weights and biases within the feature backpropagation refinement layer 308. In some cases, the interactive image editing system 102 modifies the bias parameters of the bias sublayer 310 and / or modifies the scaling parameters of the convolution sublayer 312. By modifying the bias parameters and / or scaling parameters, the interactive image editing system 102 adjusts or recalibrates how the feature backpropagation refinement layer 308 processes and transmits data, thereby modifying how the feature backpropagation refinement layer 308 generates output features.

[0069] In one or more embodiments, the interactive image editing system 102 performs backpropagation to reduce the loss metric associated with the user interaction 318. More specifically, the interactive image editing system 102 determines the loss metric associated with the user interaction 318 by comparing the pixel indicated by the user interaction 318 (or its corresponding feature extracted from the digital image 302) with other pixels (or their corresponding features extracted from the digital image 302) of the modified digital image 316 using a consistency loss function (and / or other loss functions described above). For example, the interactive image editing system 102 determines the location of one or more pixels indicated or selected by the user interaction 318 within the modified digital image 316. Furthermore, the interactive image editing system 102 determines the consistency loss based on the distance from the one or more selected pixels to other pixels of the modified digital image 316.

[0070] In some cases, the interactive image editing system 102 determines the distances between features within a feature map. For example, the interactive image editing system 102 uses a bias sublayer 310 to generate a feature map (whose dimensions correspond to the dimensions of the digital image 302) to determine the location of features extracted from the digital image 302. Furthermore, the interactive image editing system 102 determines the features indicated by the user interaction 318 and further determines the distances from those features to other features within the feature map. The interactive image editing system 102 also determines a consistency loss metric based on the distance of features (or pixels) from the user interaction, where features (or pixels) with larger distances are weighted to receive a greater refinement penalty than features (or pixels) with smaller distances. Therefore, the interactive image editing system 102 performs backpropagation 320 to reduce the loss metric, focusing the refinement of the parameters (e.g., bias parameters and scaling parameters) of the feature backpropagation refinement layer 308 onto features or pixels closer to the user interaction 318.

[0071] Following backpropagation 320, the feature backpropagation refinement layer 308 generates output features based on the features received from the first set of layers 306, which are then passed to the second set of layers 314. In effect, the interactive image editing system 102 utilizes the second set of layers 314 (including parameters learned through training) to generate a refined digital image 322 from the features passed by the feature backpropagation refinement layer 308. As shown, the interactive image editing system 102 generates a refined digital image 322 depicting correctly segmented pixels. Specifically, in response to user interaction 318, the interactive image editing system 102 backpropagates and utilizes the interactive nature of the feature backpropagation refinement layer 308 to generate the refined digital image 322 to correct the segmentation of pixels on the left side of the barn (e.g., closer to the user interaction).

[0072] As described above, in some of the described embodiments, the interactive image editing system 102 utilizes a feature backpropagation refinement layer (e.g., feature backpropagation refinement layer 308) to generate a set of output features from a set of input features. Specifically, in some embodiments, the interactive image editing system 102 utilizes multiple feature backpropagation refinement layers throughout the neural network architecture, wherein each feature backpropagation refinement layer analyzes or processes a set of input features from a previous layer to generate a set of output features to be passed to subsequent layers. Figures 4A-4B illustrate example arrangements of feature backpropagation refinement layers according to one or more embodiments.

[0073] As shown in Figure 4A, the interactive image editing system 102 generates a set of input features 402. Specifically, the interactive image editing system 102 utilizes previous layers of a digital image neural network (e.g., digital image neural network 304) with parameters learned through training to generate the input features 402. For example, the interactive image editing system 102 extracts the input features 402 by utilizing one or more trained neural network layers to analyze the digital image and extract features from the digital image. As shown, the input features 402 have dimensions corresponding to the dimensions of the input digital image, where the width (W) and height (H) match (or are proportional to) the width and height of the input digital image. Furthermore, the interactive image editing system 102 extracts the input features 402 across multiple feature channels (C). For example, the interactive image editing system 102 extracts features at the channel level, where individual channels represent or include different (types) of features. In some cases, the interactive image editing system 102 extracts different feature channels corresponding to different attributes (visible or other) of the input digital image.

[0074] As also shown in Figure 4A, the interactive image editing system 102 initializes the parameters of the feature backpropagation refinement layer 404 (e.g., before receiving user interaction). Specifically, the interactive image editing system 102 initializes the parameters of the convolutional sublayer 406 and the bias sublayer 408 such that the output features 410 match the input features 402 before backpropagation (in response to user interaction) to update the parameters. For example, for each channel, the interactive image editing system 102 initializes the scaling parameters to all have a value of 1 and the bias parameters to all have a value of 0.

[0075] In one or more embodiments, the interactive image editing system 102 trains (or utilizes) a pre-trained neural network that does not include the feature backpropagation refinement layer 404. In fact, because initialization for scaling and bias does not change the output of the feature backpropagation refinement layer 404, the interactive image editing system 102 can train other layers of the neural network and then subsequently add or insert one or more instances of the feature backpropagation refinement layer 404 into the neural network. In some cases, the interactive image editing system 102 trains a neural network that includes the feature backpropagation refinement layer 404 while fixing (or not modifying) the values ​​of the feature backpropagation refinement layer 404.

[0076] In response to user interaction indicating one or more pixels of a digital image, the interactive image editing system 102 utilizes a feature backpropagation refinement layer 404 (e.g., feature backpropagation refinement layer 308) to process or analyze input features 402. More specifically, the interactive image editing system 102 utilizes the feature backpropagation refinement layer 404 to generate a set of output features 410 from the input features 402. For example, the interactive image editing system 102 utilizes the feature backpropagation refinement layer 404, which includes a convolutional sublayer 406 and a bias sublayer 408.

[0077] As shown in the figure, in some embodiments, the convolutional sublayer 406 and the bias sublayer 408 do not have width and height dimensions corresponding to those dimensions of the input feature 402 (or the input digital image). Instead, the convolutional sublayer 406 and the bias sublayer 408 process the features on a channel-by-channel basis. For example, the convolutional sublayer 406 scales the features within each channel of the input feature 402 according to channel-specific scaling parameters. Furthermore, the bias sublayer 408 biases the input feature 402 on a channel-by-channel basis to emphasize or modify the features using channel-specific bias parameters.

[0078] Based on the analysis or processing of input features 402 using a feature backpropagation refinement layer 404, the interactive image editing system 102 generates output features 410. Specifically, the interactive image editing system 102 generates output features 410 to pass to an additional layer of a digital image neural network (e.g., digital image neural network 304). As shown, the interactive image editing system 102 generates output features whose dimensions match the dimensions of the input features 402.

[0079] As described above, the interactive image editing system 102 can modify the parameters within the feature backpropagation refinement layer 404 based on user input. Specifically, the interactive image editing system 102 can modify the parameters of the convolutional sublayer 406 and the bias sublayer 408 based on a consistency loss (and / or other losses described above). For example, the interactive image editing system 102 can determine a consistency loss that encourages consistency (e.g., penalizes changes) in pixels / regions farther from user interaction. This avoids the efficiency and accuracy problems of conventional systems that modify portions of the digital image that are not adjacent to or close to the user input. In practice, conventional systems often modify pixels far from user interaction, which requires additional, unnecessary user interaction to correct for initially unwanted / undesirable changes. By utilizing a consistency loss, the interactive image editing system 102 can improve upon the architecture utilized by conventional systems.

[0080] As shown in Figure 4B, in some embodiments, the interactive image editing system 102 utilizes feature backpropagation refinement layers with different structures or architectures. In fact, instead of the channel-by-channel bias sublayer 408 of Figure 4A, Figure 4B illustrates a feature backpropagation refinement layer 414 comprising a bias sublayer in the form of a two-dimensional bias map 416. For example, the feature backpropagation refinement layer 414 includes a two-dimensional bias map 416, which includes bias features for localizing changes or refinements to features or pixels. In effect, utilizing the two-dimensional bias map 416, the interactive image editing system 102 can apply bias parameters to isolated pixels / regions and more accurately account for local refinement of the digital image.

[0081] As shown in the figure, the interactive image editing system 102 initializes the bias map 416 (initialized to all 0s) and the convolutional sub-layer 418 (e.g., initialized to all 1s) so that the input features 412 are unaffected before user interaction. In fact, before user interaction, the output features 420 match the input features 412. In response to user interaction, the interactive image editing system 102 updates the parameters of the feature backpropagation refinement layer 414 and modifies the input features 412 to generate the output features 420 via the updated parameters (e.g., updated bias parameters and updated scaling parameters).

[0082] Specifically, the interactive image editing system 102 passes the set of input features 412 to the feature backpropagation refinement layer 414, whereby the bias map 416 localizes the input features 412 according to bias parameters. In effect, the interactive image editing system 102 applies the bias parameters of the bias map 416 to generate a (two-dimensional) biased feature map. Specifically, a biased feature map refers to one or more features that have been modified based on the bias sublayer. For example, the bias map 416 generates a biased feature map that emphasizes / de-emphasizes features at different locations within the input features 412 according to its bias parameters. As shown, the bias map 416 has one or more dimensions (e.g., width and height) corresponding to the dimensions of the input features 412 (or the input digital image). In effect, the bias map 416 is two-dimensional to match the two-dimensional input features 412 (or the two-dimensional input digital image). In some embodiments, the bias map 416 has dimensions N, H, and W, where N represents the batch size (e.g., N = 1), H represents the input feature height, and W represents the input feature width. Therefore, the bias map 416 generates feature maps that map the features to specific coordinate locations (e.g., based on bias parameters) across each channel of the input feature 412.

[0083] In one or more embodiments, the feature backpropagation refinement layer 414 includes multiple bias maps that are layered or stacked together. For example, instead of including only a single bias map 416 for generating feature maps across all feature channels, the feature backpropagation refinement layer 414 includes two or three (or more) bias maps for localizing across different groups of feature channels. For example, the feature backpropagation refinement layer 414 may include a first bias map that localizes feature variations of a first group of feature channels of the input feature 412, a second bias map that localizes feature variations of a second group of feature channels, and a third bias map that localizes feature variations of a third group of feature channels.

[0084] As further shown in Figure 4B, the feature backpropagation refinement layer 414 includes a convolutional sublayer 418. Specifically, the convolutional sublayer 418 includes at least one dimension corresponding to the dimensions (e.g., width or height) of the input features 412 (or the input digital image). Furthermore, the convolutional sublayer 418 includes another dimension corresponding to the number of channels within the set of input features 412. Therefore, the convolutional sublayer 418 uses 1×1 convolutions across various channels to scale features (e.g., features of the feature map generated via the bias map 416). For example, the convolutional sublayer 418 includes multiple 1×1 convolutions corresponding to the dimensions (e.g., width or height) of the input features 412, where each 1×1 convolution includes the same number of channels as the input features 412. In this case, the convolutional sublayer 418 includes convolutions of different dimensions (e.g., 2×2 or 3×3). As shown, the interactive image editing system 102 thus utilizes the feature backpropagation refinement layer 414 to generate output features 420 to be passed to an additional neural network layer. Specifically, the interactive image editing system 102 generates a scaled feature map from the biased feature map generated by the bias map 416. The scaled feature map refers to one or more features modified based on the scaling parameters of the convolutional sub-layer. By utilizing the convolutional sub-layer 418 together with the bias map 416, the interactive image editing system 102 enhances the digital image neural network's ability to refine both global and local predictions (e.g., not only for channel-wise scaling and biasing, but also for feature combinations across different channels).

[0085] In some cases, digital image neural networks (e.g., digital image neural network 304) include multiple feature backpropagation refinement layers, such as feature backpropagation refinement layers 404 or 414, in different parts of the architecture. For example, a digital image neural network includes a first feature backpropagation refinement layer in an earlier part of the architecture to scale and bias more global, general features. In some cases, a digital image neural network also (or alternatively) includes a second feature backpropagation refinement layer in a later part of the architecture to scale and bias more local, detailed features.

[0086] Furthermore, although Figures 4A-4B show a specific arrangement of the bias sublayer and the convolutional sublayer, the interactive image editing system 102 can utilize different arrangements or orders of these sublayers. For example, in some implementations, the interactive image editing system 102 first uses the convolutional sublayer and then applies the bias map.

[0087] As described above, in one or more embodiments, the interactive image editing system 102 determines a consistency loss associated with user interaction. Specifically, the interactive image editing system 102 determines a consistency loss that is used to penalize or punish changes to pixels based on the corresponding distance of the pixel from a pixel or feature selected directly from the user interaction (e.g., changes to more distant pixels or features are penalized more). Figures 5A-5B illustrate example descriptions of determining a consistency loss according to one or more embodiments.

[0088] As shown in Figure 5A, the interactive image editing system 102 determines consistency loss based on the distance between pixels. More specifically, the interactive image editing system 102 identifies pixel 504 (or a group of pixels) indicated or selected by user interaction with digital image 502 (e.g., modified digital image 316). Furthermore, the interactive image editing system 102 determines the distance from the selected pixel 504 to other pixels within digital image 502. For example, the interactive image editing system 102 determines the distance from pixel 504 to pixel 506a, the distance from pixel 504 to pixel 506b, and yet another distance from pixel 504 to pixel 506c.

[0089] Furthermore, the interactive image editing system applies a consistency loss function to penalize refinements made to pixels 506a-c based on their respective distances from pixel 504 (e.g., from user interaction). For example, the interactive image editing system 102 weights pixels 506a-c to penalize refinements made to pixels, where pixel 506c is penalized the least (because it is closest to pixel 504) and pixel 506b is penalized the most (because it is farthest from pixel 504). In some cases, the interactive image editing system 102 only applies penalties to refinements made to pixels that are more than a threshold distance from pixel 504 (or from user interaction) and avoids penalizing pixels within the threshold distance. In these or other cases, the interactive image editing system 102 increases the magnitude or severity of the penalty (e.g., proportionally) as the distance from pixel 504 (or from user interaction) increases.

[0090] As also shown in Figure 5A, the interactive image editing system 102 determines the consistency loss based on the distance between features. Specifically, the interactive image editing system 102 determines the location of feature 510 (or a set of features) indicated by user interaction. For example, the interactive image editing system 102 identifies feature 510 as corresponding to or representing a pixel (e.g., pixel 504) selected by user interaction. In some cases, the interactive image editing system 102 identifies feature 510 within a two-dimensional feature map 508, which shows the location of features extracted or generated via a bias map (e.g., bias map 416) through a feature backpropagation refinement layer (e.g., feature backpropagation refinement layer 414).

[0091] As shown in the figure, the interactive image editing system 102 further determines the distances from feature 510 to other features within feature map 508. For example, the interactive image editing system 102 determines the distance from feature 510 to feature 512a (e.g., representing a specific pixel, such as pixel 506a), another distance from feature 510 to feature 512b (e.g., representing pixel 506b), and yet another distance from feature 510 to feature 512c (e.g., representing pixel 506c) within feature map 508.

[0092] Furthermore, the interactive image editing system 102 applies a consistency loss function to penalize the refinement of features 512a-c based on their respective distances from feature 510 (e.g., from user interaction). For example, the interactive image editing system 102 weights features 512-c to penalize the refinement made to the features, where feature 512 is penalized the least (because it is closest to feature 510) and feature 512b is penalized the most (because it is farthest from feature 510). In some cases, the interactive image editing system 102 only applies penalties to the refinement of features that are more than a threshold distance from feature 510 (or from user interaction) and avoids penalizing features within the threshold distance. In some cases, the interactive image editing system 102 increases the magnitude or severity of the penalty as the distance from feature 510 (or from user interaction) increases (e.g., proportionally).

[0093] In addition to consistency loss, the interactive image editing system 102 can determine different losses specific to different image editing tasks. For example, the interactive image editing system 102 can determine the loss based on segmentation classification. To illustrate, a negative segmentation user interaction can identify incorrectly segmented pixels / regions. In response, the interactive image editing system 102 can determine losses that penalize the segmentation of selected pixels / regions and / or encourage changes to different segmentations. More specifically, the interactive image editing system 102 can determine losses that encourage changes to the segmentation of selected pixels / regions, and can also determine consistency losses that penalize segmentation changes that are further away from the selected pixels / regions (i.e., encourage segmentation consistency).

[0094] The interactive image editing system 102 can adapt to various segmentation user interactions with different losses (e.g., boundary selection, positive segmentation selection, etc.). For illustration, the interactive image editing system 102 can receive two choices: a first interaction indicating the segmentation classification to be copied, and a second choice indicating the segmentation to be changed to match. Similarly, the interactive image editing system 102 can receive two negative segmentation user interactions and determine the loss based on these two negative segmentation user interactions.

[0095] As described above, in addition to consistency loss, the interactive image editing system 102 can also determine a variety of other losses. For example, regarding depth maps, the interactive image editing system 102 can receive depth user interactions indicating that pixels / regions should have greater or less depth. The interactive image editing system 102 can apply losses to encourage increasing or decreasing the depth of pixels / regions, and can apply consistency loss to penalize changes to pixels / regions further away from depth user interactions.

[0096] Similarly, the interactive image editing system 102 can identify style transfer or hole-filling user interactions, which indicate a specific style to be matched or digital content to be filled in a digital image. For example, the interactive image editing system 102 can receive a first user interaction identifying an initial style to be copied and a second user interaction identifying pixels / regions to be changed. The interactive image editing system 102 can determine a loss metric to encourage changing pixels / regions from the initial style to the copied style. The interactive image editing system 102 can utilize consistency loss to penalize changes to pixels / regions further away from the user interaction. Similarly, the interactive image editing system 102 can receive a first user interaction identifying digital content to be copied (from the same digital image or a different digital image). The interactive image editing system 102 can receive a second user interaction identifying digital content to be changed. The interactive image editing system 102 can determine a loss metric based on the difference between the digital content to be changed and the digital content to be copied (while also determining consistency loss to penalize changes to other pixels / regions based on distance from the user interaction).

[0097] In some cases, as shown in Figure 5B, the interactive image editing system 102 utilizes relational loss based on multiple user interactions. For example, in addition to the distance of a pixel (or feature) to a user interaction, the interactive image editing system 102 also determines relational loss based on the relationship between two or more user interactions (e.g., clicks). As shown, the interactive image editing system 102 receives a first user interaction with the digital image 514 that selects a first pixel 516a (or a first group of pixels), and further receives a second user interaction with the digital image 514 that selects a second pixel 516b (or a second group of pixels).

[0098] As also shown in the figure, the interactive image editing system 102 determines a first feature 522a associated with a first pixel 516a and a second feature 522b associated with a second pixel 516b within the feature map 520. In some embodiments, the extracted features do not necessarily come from the same digital image. In fact, as mentioned above regarding digital content hole filling, the interactive image editing system 102 can extract digital content or features from other sources.

[0099] Furthermore, the interactive image editing system 102 also determines the relationship between a first user interaction (or pixel 516a or feature 522a) and a second user interaction (or pixel 516b or feature 522b). For example, the interactive image editing system 102 receives an indication of the relationship between user interactions (e.g., user settings). In some cases, the interactive image editing system 102 determines or defines a relationship such that the pixel value (or feature value) of the pixel indicated by the first user interaction (e.g., pixel 516a or feature 522a) will be replaced with the pixel value (or feature value) of the pixel indicated by the second user interaction (e.g., pixel 516b or feature 522b). In other cases, the interactive image editing system 102 determines a relationship that changes the pixel values ​​(feature values) associated with pixel 516a and pixel 516b (or feature 522a and feature 522b) to be closer (more similar) or farther apart (less similar).

[0100] In response to a first user interaction and / or a second user interaction, the interactive image editing system 102 determines a loss metric to encourage changes to pixels at or near the selected pixels 516a and 516b. Furthermore, the interactive image editing system 102 determines a consistency loss to discourage changes further away from the selected pixels 516a and 516b. In practice, the interactive image editing system 102 determines the consistency loss based on distances relative to other pixels in the digital image 514 and / or relative to other features in the feature map 520 (e.g., feature maps extracted from the digital image 514). For example, the interactive image editing system 102 determines the distance of pixel 518a from pixel 516a (or from the first user interaction), and determines the distance of pixel 518b from pixel 516a (or from the first user interaction).

[0101] Furthermore, the interactive image editing system 102 determines the distance of pixel 518a from pixel 516b (or from the second user interaction), and determines the distance of pixel 518b from pixel 516b (or from the second user interaction). The interactive image editing system 102 thus determines a relational loss based on the difference between the pixel / region selected by the user and the desired change (e.g., incorrect segmentation, induced depth, different style, new digital content to be filled), and then localizes the change by using a consistency loss to penalize the refinement of the digital image further away from the (multiple) user interactions.

[0102] In some embodiments, as part of determining the consistency loss, the interactive image editing system 102 determines the distance of feature 524a from feature 522a (or from the first user interaction) within the feature map 520, and further determines the distance of feature 524b from feature 522a (or from the first user interaction). Additionally, the interactive image editing system 102 determines the distance of feature 524a from feature 522b (or from the second user interaction), and further determines the distance of feature 524b from feature 522b (or from the second user interaction).

[0103] As described above, by using consistency loss, the interactive image editing system 102 penalizes changes or refinements to pixels or features based on their distance relative to the selected pixels or features. For example, the interactive image editing system 102 weights pixels or features to penalize refinements made relative to pixels or features farther away from pixels 516a and 516b or features 522a and 522b. Specifically, based on a specific relationship between pixels 516a and 516b (or features 522a and 522b), the interactive image editing system 102 applies a penalty to reduce the likelihood of selecting a pixel value (or feature value) farther away from pixel 516b (or feature 522b) to replace a pixel value (or feature value) associated with pixel 516a (or feature 522a). Similarly, the interactive image editing system 102 applies weights or penalties to reduce the likelihood of replacing pixels (or features) farther away from pixel 516a (or feature 522a) with other pixels (or features). Therefore, based on the relational loss indicating the relationship indicated by one or more user interactions, and the consistency loss based on the distance to other pixels (or features), the interactive image editing system 102 replaces the pixel value (or feature value near feature 522a) near pixel 516a with the pixel value (or feature value near feature 522b) near pixel 516b.

[0104] As described above, based on another relationship, the interactive image editing system 102 penalizes changes in other ways to reduce the likelihood of pixel modification. For example, the interactive image editing system 102 determines a relationship loss to move the pixel value associated with pixel 516a (or the feature value associated with feature 522a) and the pixel value associated with pixel 516b (or the feature value associated with feature 522b) closer (or farther away). Furthermore, the interactive image editing system 102 applies a consistency loss penalty to ensure that no changes are made to more distant pixels (or features), and to ensure that only pixels (or features) close to (e.g., within a threshold distance) pixel 516a and / or pixel 516b (or feature 522a and / or feature 522b) are refined or modified.

[0105] As previously described, in some embodiments, the interactive image editing system 102 determines other types of loss. For example, the interactive image editing system 102 receives additional user interactions (e.g., three or more) and determines relationships between the user interactions. For example, the interactive image editing system 102 determines that a pixel (or feature) corresponding to a first user interaction is less like a pixel (or feature) corresponding to a second user interaction and more like a pixel (or feature) corresponding to a third user interaction. In other embodiments, the interactive image editing system 102 determines relationship loss based on a single user interaction to indicate only incorrect pixels (or features) without indicating how to correct them. In still other embodiments, the interactive image editing system 102 determines loss based on a first user interaction to indicate one or more incorrect pixels and further indicate how to increase, decrease, or change pixel values ​​(or feature values) in some other specified manner.

[0106] In one or more embodiments, the interactive image editing system 102 combines losses together to form a combined loss or total loss. For example, as part of modifying pixels in a digital image, the interactive image editing system 102 determines relational loss, consistency loss, and / or other losses. As described herein, the interactive image editing system 102 utilizes some losses (e.g., L1 loss, L2 loss, etc.) to encourage changes to pixels near user interaction and uses consistency loss to prevent changes further away from user interaction. In some cases, the interactive image editing system 102 combines losses together (e.g., adds the losses together) to form an overall loss.

[0107] In some embodiments, the interactive image editing system 102 performs the step of generating modified parameters for a feature backpropagation refinement layer. The above description of actions 208-212 of FIG2 (including a more detailed description of supporting actions 208-212 provided in FIG3A-FIG5B) provides various embodiments and supporting actions and algorithms for performing the step of generating modified parameters for a feature backpropagation refinement layer.

[0108] For example, in some embodiments, performing the step of generating modified parameters for a feature backpropagation refinement layer includes: receiving user interaction and backpropagation to modify the parameters of the feature backpropagation refinement layer in response to user interaction (e.g., as described with respect to Figures 3A-3B). In some embodiments, performing the step of generating modified parameters for a feature backpropagation refinement layer further includes: determining a consistency loss (e.g., as described with respect to Figures 5A-5B) and backpropagation to reduce the consistency loss by modifying the parameters of the feature backpropagation refinement layer.

[0109] As described above, in some embodiments, the interactive image editing system 102 utilizes a feature backpropagation refinement layer to generate various types of refined digital images. Specifically, the interactive image editing system 102 utilizes the feature backpropagation refinement layer as part of a digital image neural network to generate refined digital images based on modified parameters of bias sublayers and convolutional sublayers. Figures 6A-6C illustrate the generation of refined digital images in the form of segmented digital images according to one or more embodiments. Subsequently, Figures 7A-7B illustrate the generation of refined digital images in the form of digital image matting according to one or more embodiments. Furthermore, Figures 8A-8C illustrate the generation of refined digital images in the form of digital image depth maps according to one or more embodiments.

[0110] As shown in Figure 6A, the interactive image editing system 102 receives or accesses an initial digital image 602. As shown, the initial digital image 602 depicts an urban scene from the perspective of a car or tram traveling along a street. The digital image 602 depicts various types of objects belonging to different semantic classes or categories, including cars, buildings, roads and people, and lampposts.

[0111] As shown in Figure 6B, the interactive image editing system 102 generates a modified digital image 604. Specifically, the interactive image editing system 102 uses a digital image neural network to generate the modified digital image 604 from the digital image 602. For example, the interactive image editing system 102 utilizes a specific digital image neural network that includes one or more feature backpropagation refinement layers at certain locations in its architecture. In some embodiments, the interactive image editing system 102 uses a digital image neural network to generate segmented digital images, as described by Andrew Tao, Karan Sapra, and Bryan Catanzaro in "Hierarchical Multi-Scale Attention for Semantic Segmentation (arXiv:2005.1082(2020))". In other embodiments, the interactive image editing system 102 uses a digital image neural network to segment digital images with different architectures.

[0112] For example, the interactive image editing system 102 utilizes a digital image neural network that includes three feature backpropagation refinement layers at specified scales at the ends of the neural network backbone before multi-scale features are scaled and cascaded. Specifically, the interactive image editing system 102 utilizes feature backpropagation refinement layers with the following dimensions: number of channels = [96, 192, 384], H = [img_h / 8, img_h / 16, img_h / 32], W = [img_w / 8, img_w / 16, img_w / 32], where H represents height, W represents width, img_h represents image height, and img_w represents image width.

[0113] As shown in the figure, the modified digital image 604 is a panoramic segmentation digital image that depicts initial predictions of different semantic segments of objects depicted within digital image 602. In practice, the interactive image editing system 102 generates the modified digital image 604 (before the user interacts with it) using learned parameters from various neural network layers and unlearned (e.g., initialized) parameters from feature backpropagation refinement layers. Furthermore, the interactive image editing system 102 receives or detects user interactions with the modified digital image 604. As shown, the interactive image editing system 102 receives selections of pixels incorrectly labeled as people. In some cases, the interactive image editing system 102 receives a second user interaction with the modified digital image 604, which selects pixels above a truck that are incorrectly segmented as non-lamppost pixels (where the pixels actually depict lampposts). The interactive image editing system 102 can identify: user interactions that identify incorrect segmentation, user interactions that identify alternative segmentation classes, and / or two interactions that identify matching segmentation classes.

[0114] As shown in Figure 6C, in response to receiving multiple user interactions, the interactive image editing system 102 generates a refined digital image 606. Specifically, in response to user interaction with the modified digital image 604, the interactive image editing system 102 generates the refined digital image 606 based on features extracted from the digital image 602. For example, the interactive image editing system 102 generates the refined digital image 606 by: determining the consistency loss associated with the user interaction in response to the user interaction; backpropagating to modify the parameters of the feature backpropagation refinement layer (e.g., to reduce the consistency loss); and using the modified parameters to refine the features extracted from the digital image 602. As shown, the refined digital image 606 depicts corrected pixels of lampposts and people. In fact, compared to the modified digital image 604, the refined digital image 606 reclassifies pixels previously identified as people (e.g., as building pixels) and reclassifies pixels previously identified as building pixels as lamppost pixels.

[0115] As described above, in some embodiments, the interactive image editing system 102 generates a refined digital image in the form of digital image matting. Figure 7A illustrates a modified digital image 702 in the form of digital image matting according to one or more embodiments. As shown, the modified digital image 702 depicts a glass of water, with some pixels around the glass incorrectly matted out. To generate the modified digital image 702, the interactive image editing system 102 utilizes a digital image neural network, such as IndexNet (or some other neural network architecture), which is modified to include feature backpropagation refinement layers with specific dimensions (e.g., number of channels = 64, H = img_h / 8, W = image_w / 8) in the skip connections. In effect, the interactive image editing system 102 uses learned parameters from one or more neural network layers and unlearned parameters from the feature backpropagation refinement layers to generate the modified digital image 702.

[0116] As further illustrated, the interactive image editing system 102 receives user interactions that indicate one or more pixels of the modified digital image 702. For example, the interactive image editing system 102 may identify: user interactions that identify incorrectly matted pixels / regions, user interactions that identify replacement matting classifications (e.g., replacing the current pixel matting prediction with foreground, background, or alpha value), and / or two interactions that identify matching pixels. In response to the user interactions, the interactive image editing system 102 determines the consistency loss and backpropagates to modify the parameters of the feature backpropagation refinement layer to reduce the consistency loss as described herein. As shown in Figure 7B, using the modified parameters of the feature backpropagation refinement layer, the interactive image editing system 102 generates a refined digital image 704. In effect, the interactive image editing system 102 generates the refined digital image 704 to correct or repair pixels indicated by the user interactions. As shown, the refined digital image 704 is a digital image matting depicting the corrected pixels around a water glass.

[0117] As described above, in some embodiments, the interactive image editing system 102 generates a refined digital image in the form of a digital image depth map. Figures 8A-8C illustrate the generation of a digital image depth map according to one or more embodiments. As shown in Figure 8A, the interactive image editing system 102 receives or accesses an initial digital image 802. Specifically, the initial digital image 802 depicts a bathroom with a shower, toilet, and bathtub, all at different depths relative to the camera that captured the image.

[0118] As shown in Figure 8B, the interactive image editing system 102 generates a modified digital image 804 from a digital image 802. Specifically, the interactive image editing system 102 utilizes a digital image neural network modified to include a feature backpropagation refinement layer with unlearned parameters to generate the modified digital image 804. For example, the interactive image editing system 102 utilizes a digital image neural network, as described by Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh in “From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation (arXiv:1907.10326v5(2020))”, wherein the digital image neural network is modified to include one or more feature backpropagation refinement layers.

[0119] For example, a digital image neural network includes a single feature backpropagation refinement layer with specific dimensions (e.g., number of channels = 256, H = img_h / 32, W = img_w / 32) at the starting features of the decoder. In some cases, the interactive image editing system 102 selects the top 256 channels with the highest absolute activation from the total 1024 channels of the neural network. In some embodiments, the interactive image editing system 102 utilizes neural networks with different architectures. As shown, the modified digital image 804 depicts a depth map indicating different depths of objects depicted within the digital image 802. As further shown, the interactive image editing system 102 receives user interaction indicating one or more pixels of the modified digital image 804.

[0120] As shown in Figure 8C, the interactive image editing system 102 generates a refined digital image 806 from features extracted from the digital image 802 in response to user interaction with the modified digital image 804. For example, the interactive image editing system 102 identifies user interactions indicating two matching depths, user interactions indicating edges (or both sides of an edge), user interactions indicating the need to increase depth, or user interactions indicating the need to decrease depth. The interactive image editing system 102 generates the refined digital image 806 by backpropagating to modify the parameters of the feature backpropagation refinement layer and updating the features extracted from the digital image 802 according to the modified parameters. In practice, the interactive image editing system 102 utilizes a digital image neural network to generate the refined digital image 806, which depicts a correction of the pixels of the modified digital image 804. As shown, compared to the modified digital image 804, the refined digital image 806 shows sharper edges depicting objects with varying depths.

[0121] In some of the described embodiments, the interactive image editing system 102 improves digital image prediction based on user interactions that indicate pixels initially mispredicted. Experimenters have demonstrated improvements to the interactive image editing system 102 in several use cases, including semantic segmentation, image matting, and depth mapping. For example, Figures 9A-9C show graphs depicting the accuracy improvements of the interactive image editing system 102 in response to user interactions (e.g., clicks).

[0122] As shown in Figure 9A, Figures 902 and 904 each depict the results of semantic segmentation. In essence, Figures 902 and 904 show a comparison between the number of clicks received from the client device and the mean Intersection over Union (mIoU) metric representing the pixel accuracy in the generated prediction (e.g., a refined digital image). As shown, Figure 902 illustrates the results measured on the Cityscapes validation dataset, and Figure 904 illustrates the results measured on the Mapillary validation dataset. In each case, the mIoU metric increases with the number of clicks (e.g., indicating one or more pixels within the modified digital image to be corrected), suggesting that the improved accuracy of the neural network is due to the parameters of the backpropagation refinement layer modified in response to user interaction.

[0123] As shown in Figure 9B, Graph 906 depicts the experimental results for image matting. As shown, the interactive image editing system 102 improves the accuracy of digital images generated based on user interaction for image matting. Graph 906 depicts a comparison between the number of clicks and the mean squared error (MSE), where the MSE decreases as the number of clicks increases. In fact, when the interactive image editing system 102 receives user interaction indicating pixels to be corrected, the digital image neural network utilizes modified parameters of the feature backpropagation refinement layer to generate a more accurate digital image.

[0124] As shown in Figure 9C, Graph 908 depicts the experimental results of depth mapping. As illustrated, the interactive image editing system 102 improves the accuracy of the neural network in generating digital image depth maps based on user interaction. Graph 908 depicts this accuracy improvement, where the accuracy metric along the Y-axis increases with the number of clicks along the X-axis. In fact, when the interactive image editing system 102 receives user interaction indicating pixels to be corrected, the digital image neural network utilizes modified parameters of the feature backpropagation refinement layer to generate a more accurate digital image.

[0125] Referring now to Figure 10, additional details regarding the components and capabilities of the interactive image editing system 102 will be provided. Specifically, Figure 10 illustrates an example schematic diagram of the interactive image editing system 102 on an example computing device 1000 (e.g., one or more client devices and / or servers among client device 108 and / or (multiple) servers 104). As shown in Figure 10, the interactive image editing system 102 includes a bias sublayer manager 1002, a convolution sublayer manager 1004, a consistency loss manager 1006, a digital image manager 1008, and a storage manager 1010.

[0126] As just mentioned, the interactive image editing system 102 includes a bias sublayer manager 1002. Specifically, the bias sublayer manager 1002 manages, maintains, modifies, trains, implements, updates, utilizes, or applies bias sublayers as part of a feature backpropagation refinement layer in a digital image neural network. For example, the bias sublayer manager 1002 updates the bias parameters of the bias sublayers in response to user interaction to reduce consistency loss metrics in some cases. In one or more embodiments, the bias sublayer manager 1002 generates feature maps from features extracted from an initial digital image using parameters updated via backpropagation in response to user interaction.

[0127] Furthermore, the interactive image editing system 102 includes a convolutional sublayer manager 1004. Specifically, the convolutional sublayer manager 1004 manages, maintains, modifies, trains, implements, updates, utilizes, or applies convolutional sublayers as part of the feature backpropagation layer of a digital image neural network. For example, the convolutional sublayer manager 1004 modifies or updates the scaling parameters of the convolutional sublayer in response to user interaction involving the selection of one or more pixels. In some cases, the convolutional sublayer manager 1004 processes features extracted from the digital image and / or from the feature map of the bias sublayer to generate output features to be passed to additional layers of the digital image neural network (e.g., for generating a refined digital image).

[0128] As further shown in Figure 10, the interactive image editing system 102 includes a consistency loss manager 1006. Specifically, the consistency loss manager 1006 manages, maintains, determines, generates, identifies, reduces, modifies, or acquires consistency loss metrics. For example, the consistency loss manager 1006 determines consistency loss based on the distance from selected pixels and / or features to other pixels and / or features associated with the digital image. In some cases, the consistency loss manager 1006 determines consistency loss to penalize changes that are farther away from the selected pixels and / or selected features, such that only pixels and / or features closer to the user interaction are modified in response to the user interaction. The consistency loss manager 1006 can determine different types of consistency loss based on different numbers of user interactions and / or the relationships between user interactions. The consistency loss manager 1006 also communicates with the bias sub-layer manager 1002 and / or the convolution sub-layer manager 1004 to modify parameters to reduce consistency loss and generate new features.

[0129] Furthermore, the interactive image editing system 102 includes a digital image manager 1008. Specifically, the digital image manager 1008 receives, accesses, generates, determines, modifies, updates, refines, or identifies digital images. For example, the digital image manager 1008 utilizes one or more layers of a digital image neural network to extract features from the digital image and pass them to a feature backpropagation refinement layer that includes bias sublayers and convolutional sublayers. The digital image manager 1008 also passes the modified features from the feature backpropagation refinement layer to additional neural network layers to generate a modified or refined digital image. In response to modifying the parameters of the feature backpropagation refinement layer, the digital image manager 1008 can generate specific types of refined digital images, such as segmented digital images, digital image depth maps, digital image matting, restored digital images, or stylized digital images.

[0130] The interactive image editing system 102 also includes a storage manager 1010. The storage manager 1010 incorporates or includes one or more memory devices, such as a database 1012 (e.g., database 112), which stores various types of data, such as one or more digital images, modified digital images, refined digital images, and / or digital image neural networks including feature backpropagation refinement layers.

[0131] In one or more embodiments, each component of the interactive image editing system 102 communicates with each other using any suitable communication technology. Furthermore, the components of the interactive image editing system 102 communicate with one or more other devices, including the aforementioned client devices. It will be appreciated that although the components of the interactive image editing system 102 are shown separately in FIG. 10, any sub-components may be combined into fewer components, such as a single component, or divided into more components to serve a particular implementation. Moreover, although the components of FIG. 10 are described in conjunction with the interactive image editing system 102, at least some of the components used to perform operations in conjunction with the interactive image editing system 102 described herein may be implemented on other devices within the environment.

[0132] Components of the interactive image editing system 102 may include software, hardware, or both. For example, components of the interactive image editing system 102 may include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices (e.g., computing device 1000). When executed by one or more processors, the computer-executable instructions of the interactive image editing system 102 may cause the computing device 1000 to perform the methods described herein. Alternatively, components of the interactive image editing system 102 may include hardware, such as dedicated processing equipment for performing specific functions or groups of functions. Additionally or alternatively, components of the interactive image editing system 102 may include a combination of computer-executable instructions and hardware.

[0133] Furthermore, components of the interactive image editing system 102 that perform the functions described herein can be implemented, for example, as part of a standalone application, a module of an application, a plugin for an application (including a content management application), one or more library functions that can be called by other applications, and / or a cloud computing model. Therefore, components of the interactive image editing system 102 can be implemented as part of a standalone application on a personal computing device or mobile device. Alternatively or additionally, components of the interactive image editing system 102 can be implemented in any application that allows the creation of marketing content and its delivery to users, including but not limited to… Experience Manager and Creative Applications such as Adobe Inc.'s registered trademarks or trademarks in the U.S. and / or other countries. and "ADOBE", "ADOBE EXPERIENCE MANAGER", "CREATIVE CLOUD", "PHOTOSHOP", "LIGHTROOM" and "INDESIGN".

[0134] The corresponding text and examples in Figures 1-10 provide a variety of different systems, methods, and non-transitory computer-readable media for generating refined digital images using digital image neural networks that include feature backpropagation refinement layers and / or consistency losses. In addition to the foregoing, embodiments may also be described according to flowcharts including actions for achieving specific results. For example, Figure 11 shows a flowchart of an example sequence or series of actions according to one or more embodiments.

[0135] While Figure 11 illustrates actions according to a particular embodiment, alternative embodiments may omit, add, reorder, and / or modify any actions shown in Figure 11. The actions of Figure 11 may be performed as part of a method. Alternatively, a non-transitory computer-readable medium may include instructions that, when executed by one or more processors, cause a computing device to perform the actions of Figure 1. In another embodiment, a system may perform the actions of Figure 11. Furthermore, the actions described herein may be repeated or performed in parallel with each other, or performed in parallel with different instances of the same or other similar actions.

[0136] Figure 11 illustrates an example series of actions 1100 for generating a refined digital image using a digital image neural network including a feature backpropagation refinement layer and / or consistency loss. Specifically, the series of actions 1100 includes an action 1102 for generating a modified digital image using a neural network with a feature backpropagation refinement layer. For example, action 1102 involves generating a modified digital image from a digital image using a digital image neural network including neural network layers and feature backpropagation refinement layers. The neural network layers have learned parameters, and the feature backpropagation refinement layers include a two-dimensional bias map with unlearned bias parameters and convolutional sub-layers with unlearned scaling parameters. In some cases, action 1102 involves utilizing a digital image neural network including neural network layers with learned parameters determined during training of the digital image neural network without utilizing sample user interaction.

[0137] In some embodiments, action 1102 involves generating a modified digital image from a digital image using learned parameters from multiple neural network layers. In some cases, action 1102 involves utilizing a feature backpropagation refinement layer comprising convolutional sublayers having a first dimension corresponding to the number of feature channels extracted by the multiple neural network layers and a second dimension corresponding to the dimensions of the digital image. In some embodiments, action 1102 involves utilizing the learned parameters from the multiple neural network layers together with unlearned bias parameters from bias sublayers within the feature backpropagation refinement layer and unlearned scaling parameters from convolutional sublayers within the feature backpropagation refinement layer. In one or more embodiments, action 1102 involves generating a modified digital image using multiple neural network layers and a feature backpropagation refinement layer comprising multiple two-dimensional bias maps. Action 1102 may include utilizing learned parameters from the multiple neural network layers learned without utilizing sample user interaction.

[0138] As shown in the figure, action series 1100 also includes action 1104 for determining a loss metric based on user interaction. Specifically, action 1104 involves determining a loss metric based on user interaction with one or more pixels of the modified digital image. For example, action 1104 involves determining the distance from the one or more pixels of the user interaction to other pixels within the modified digital image, and penalizing refinement of pixels of the modified digital image further away from the user interaction based on the distance from the one or more pixels of the user interaction. In some cases, action 1104 involves determining a consistency loss based on the distance from the user interaction in response to user interaction with one or more pixels of the modified digital image.

[0139] In some embodiments, action 1106 involves determining the distance between one or more features corresponding to one or more pixels in the feature map that are interacting with the user and features corresponding to other pixels in the modified digital image. Additionally, action 1106 involves penalizing the refinement of a first feature in the feature map using a first penalty based on a first feature distance to the one or more features corresponding to the one or more pixels. Furthermore, action 1106 involves penalizing the refinement of a second feature in the feature map using a second penalty based on a second feature distance to the one or more features corresponding to the one or more pixels, the second penalty being greater than the first penalty.

[0140] As also shown in the figure, action series 1100 includes action 1106 for generating modified parameters, such as modified bias parameters and modified scaling parameters. Specifically, action 1106 involves using a loss metric to generate modified bias parameters for a two-dimensional bias map and modified scaling parameters for a convolutional sublayer. For example, action 1106 involves backpropagating within a digital image neural network to modify unlearned bias parameters and unlearned scaling parameters to reduce the loss metric determined based on user interaction. In some cases, action 1106 involves using a consistency loss to generate modified parameters for a feature backpropagation refinement layer. In one or more embodiments, action 1106 involves generating modified parameters for a feature backpropagation refinement layer including bias sublayers and convolutional sublayers by generating modified bias parameters for the bias sublayers and modified scaling parameters for the convolutional sublayers based on a consistency loss. Action 1106 may also involve generating modified bias parameters for a bias sublayer including a bias map having dimensions corresponding to the dimensions of the digital image or the dimensions of a feature map representing the digital image.

[0141] Furthermore, the action series 1100 includes an action 1108 that generates a refined digital image from modified bias parameters and modified scaling parameters. Specifically, action 1108 involves backpropagating one or more of the modified parameters of the refinement layer (e.g., modified bias parameters of a two-dimensional bias map and modified scaling parameters of the convolutional sublayer) from the digital image using learned parameters and features of the neural network layers. For example, action 1108 involves: updating one or more pixels of a user interaction, and updating one or more additional pixels of the modified digital image in response to the user interaction based on the modified bias parameters and modified scaling parameters.

[0142] In some cases, action 1108 involves extracting a set of features from a digital image using a first set of learned parameters from the learned parameters of the neural network layers, generating a biased feature map from the set of features using modified bias parameters of a two-dimensional bias map, and generating a scaled feature map from the biased feature map using modified scaling parameters of the convolutional sublayers. In these or other cases, action 1108 involves generating a refined digital image from the scaled feature map using a second set of learned parameters from the learned parameters of the neural network layers. In some embodiments, action 1108 involves...

[0143] In some embodiments, the series of actions 1100 includes the actions of generating additional modified bias parameters for an additional two-dimensional bias map and additional modified scaling parameters for an additional convolutional sublayer. Furthermore, the series of actions 1100 includes the actions of generating a refined digital image from a digital image using the additional modified bias parameters for the additional two-dimensional bias map and the additional modified scaling parameters for the additional convolutional sublayer.

[0144] In one or more embodiments, the action series 1100 includes receiving a user interaction that identifies one or more pixels of a modified digital image. Receiving the user interaction may include: receiving a selection of a region to be cut out from the digital image; receiving a selection to modify an image fragment within a segmented digital image; receiving a selection to modify the depth of a region in a depth map of the digital image; receiving a selection to fill pixels within the restored digital image; or receiving a selection to indicate style transfer within a stylized digital image. Receiving the user interaction may involve receiving a multi-pixel user interaction indicating multiple pixels from the modified digital image.

[0145] Embodiments of this disclosure may include or utilize a dedicated or general-purpose computer including computer hardware (e.g., one or more processors and system memory), as discussed in more detail below. Embodiments within the scope of this disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any media content access device described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., memory, etc.) and executes those instructions to perform one or more processes, including one or more processes described herein.

[0146] Computer-readable media can be any available medium that can be accessed by a general-purpose or special-purpose computer system. A computer-readable medium storing computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of this disclosure may include at least two distinctly different types of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0147] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drive (“SSD”) (e.g., RAM-based), flash memory, phase-change memory (“PCM”), other types of memory, other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0148] A “network” is defined as one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or other communication connection (hardwired, wireless, or a combination of hardwired and wireless), the computer correctly regards that connection as a transmission medium. A transmission medium may include networks and / or data links that can be used to carry desired program code in the form of computer-executable instructions or data structures and that are accessible by a general-purpose or special-purpose computer. Combinations of the foregoing should also be included within the scope of computer-readable media.

[0149] Furthermore, upon arrival at various computer system components, program code in the form of computer-executable instructions or data structures can be automatically transferred from the transmission medium to a non-transitory computer-readable storage medium (device) (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be buffered in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or to a less volatile computer storage medium (device) at the computer system. Therefore, it should be understood that a non-transitory computer-readable storage medium (device) can be included in computer system components that also (or even primarily) utilize the transmission medium.

[0150] Computer-executable instructions include, for example, instructions and data that, when executed at a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or group of functions. In some embodiments, the computer-executable instructions execute on a general-purpose computer to turn the general-purpose computer into a special-purpose computer that implements the elements of this disclosure. The computer-executable instructions may be, for example, binary, intermediate format instructions (such as assembly language), or even source code. Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as exemplary forms for implementing the claims.

[0151] Those skilled in the art will understand that this disclosure can be practiced in networked computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, mobile phones, PDAs, tablets, pagers, routers, switches, etc. This disclosure can also be practiced in distributed system environments, where local and remote computer systems linked via a network (via hardwired data links, wireless data links, or a combination of hardwired and wireless data links) each perform tasks. In a distributed system environment, program modules can reside in local and remote memory storage devices.

[0152] The embodiments of this disclosure can also be implemented in a cloud computing environment. In this specification, "cloud computing" is defined as a model that enables on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be used in the market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources can be rapidly configured via virtualization and published with minimal management effort or service provider interaction, and then scaled accordingly.

[0153] Cloud computing models can be composed of various features, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, and measurable services. Cloud computing models can also expose various service models, such as Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). Different deployment models (such as private cloud, community cloud, public cloud, hybrid cloud, etc.) can also be used to deploy cloud computing models. In this specification and claims, “cloud computing environment” means an environment employing cloud computing.

[0154] Figure 12 illustrates an example computing device 1200 (e.g., computing device 1000, client device 108, and / or (multiple) servers 104) in block diagram form. The computing device 1600 can be configured to perform one or more of the processes described above. It will be understood that the interactive image editing system 102 may include an implementation of the computing device 1200. As shown in Figure 12, the computing device may include a processor 1202, a memory 1204, a storage device 1206, an I / O interface 1208, and a communication interface 1210. Furthermore, the computing device 1200 may include input devices such as a touchscreen, mouse, keyboard, etc. In some embodiments, the computing device 1200 may include fewer or more components than those shown in Figure 12. The components of the computing device 1200 shown in Figure 12 will now be described in more detail.

[0155] In a particular embodiment, processor(s) 1202 includes hardware for executing instructions, such as instructions constituting a computer program. By way of example and not limitation, in order to execute instructions, processor(s) 1202 may fetch (or fetch instructions) from internal registers, internal caches, memory 1204, or storage device 1206, and decode and execute the instructions.

[0156] Computing device 1200 includes memory 1204 coupled to processor(s) 1202. Memory 1204 can be used to store data, metadata, and programs executed by processor(s). Memory 1204 can include one or more of volatile and non-volatile memory, such as random access memory (“RAM”), read-only memory (“ROM”), solid-state drive (“SSD”), flash memory, phase-change memory (“PCM”), or other types of data storage devices. Memory 1204 can be internal or distributed memory.

[0157] Computing device 1200 includes storage device 1206, which includes a storage means for storing data or instructions. By way of example and not limitation, storage device 1206 may include the aforementioned non-transitory storage medium. Storage device 1206 may include a hard disk drive (HDD), flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices.

[0158] The computing device 1200 also includes one or more input or output (“I / O”) devices / interfaces 1208 provided to allow a user to provide input to the computing device 1200 (e.g., user strokes), receive output from the computing device 1200, and otherwise transmit data to and from the computing device 1200. These I / O devices / interfaces 1208 may include a mouse, keypad or keyboard, touchscreen, camera, optical scanner, network interface, modem, other known I / O devices, or combinations of such I / O devices / interfaces 1208. The touchscreen can be activated using a writing device or a finger.

[0159] I / O device / interface 1208 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O device / interface 1208 is configured to provide graphics data to a display for presentation to a user. The graphics data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0160] The computing device 1200 may also include a communication interface 1210. The communication interface 1210 may include hardware, software, or both. The communication interface 1210 may provide one or more interfaces for communication (e.g., packet-based communication) between the computing device and one or more other computing devices 1200 or one or more networks. By way of example and not limitation, the communication interface 1210 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks such as Wi-Fi. The computing device 1200 may also include a bus 1212. The bus 1212 may include hardware, software, or both for coupling components of the computing device 1200 to each other.

[0161] In the foregoing description, specific exemplary embodiments of this disclosure have been described with reference to them. Various embodiments and aspects of this disclosure have been described with reference to the details discussed herein, and various embodiments are illustrated in the accompanying drawings. The above description and drawings are illustrative of this disclosure and should not be construed as limiting it. Numerous specific details have been described to provide a thorough understanding of various embodiments of this disclosure.

[0162] This disclosure may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The described embodiments are to be considered illustrative rather than restrictive in all respects. For example, the methods described herein may be performed with fewer or more steps / actions, or the steps / actions may be performed in a different order. Furthermore, the steps / actions described herein may be repeated, or performed in parallel with each other or in parallel with different instances of the same or similar steps / actions. Therefore, the scope of this disclosure is indicated by the appended claims rather than the foregoing description. All modifications falling within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A non-transitory computer-readable medium comprising instructions, when executed by at least one processor, to cause a computing device to perform the following operations: generating a modified digital image from a digital image using a digital image neural network, the digital image neural network including a neural network layer and a feature backpropagation refinement layer, the neural network layer having learned parameters, the feature backpropagation refinement layer including a two-dimensional bias map having unlearned bias parameters and a convolutional sublayer having unlearned scaling parameters; determining a loss metric based on user interaction with one or more pixels of the modified digital image; generating modified bias parameters of the two-dimensional bias map and modified scaling parameters of the convolutional sublayer using the loss metric; and generating the refined digital image from the digital image using the learned parameters of the neural network layer, the modified bias parameters of the two-dimensional bias map, and the modified scaling parameters of the convolutional sublayer.

2. The non-transitory computer-readable medium of claim 1, further comprising, when executed by the at least one processor, instructions for the computing device to determine the loss metric by: determining the distance from the one or more pixels of the user interaction to other pixels within the modified digital image; and penalizing refinement of pixels in the modified digital image further from the user interaction based on the distance from the one or more pixels of the user interaction.

3. The non-transitory computer-readable medium of claim 1, further comprising, when executed by the at least one processor, instructions for the computing device to generate the modified bias parameters and the modified scaling parameters by backpropagating within the digital image neural network to modify the unlearned bias parameters and the unlearned scaling parameters to reduce the loss metric determined based on the user interaction.

4. The non-transitory computer-readable medium of claim 1, further comprising, when executed by the at least one processor, instructions for the computing device to generate the refined digital image by: updating the one or more pixels of the user interaction, and updating one or more additional pixels of the modified digital image in response to the user interaction, based on the modified bias parameter and the modified scaling parameter.

5. The non-transitory computer-readable medium of claim 1, further comprising, when executed by the at least one processor, instructions for the computing device to generate the refined digital image by: extracting a set of features from the digital image using a first set of learned parameters from the learned parameters of the neural network layer; The modified bias parameters of the two-dimensional bias map are used to generate a biased feature map from the set of features; And using the modified scaling parameters of the convolutional sublayer to generate a scaled feature map from the biased feature map.

6. The non-transitory computer-readable medium of claim 5, further comprising, when executed by the at least one processor, instructions for the computing device to generate the refined digital image by: generating the refined digital image from the scaled feature map using a second set of learned parameters from the learned parameters of the neural network layers.

7. The non-transitory computer-readable medium of claim 1, further comprising, when executed by the at least one processor, instructions for the computing device to generate the modified digital image using the digital image neural network including the neural network layer, the neural network layer having the learned parameters determined during training of the digital image neural network without utilizing sample user interaction.

8. The non-transitory computer-readable medium of claim 1, further comprising, when executed by the at least one processor, instructions for the computing device to perform the following operations: generating additional modified bias parameters for an additional two-dimensional bias map and additional modified scaling parameters for an additional convolutional sublayer; and generating the refined digital image from the digital image using the additional modified bias parameters of the additional two-dimensional bias map and the additional modified scaling parameters for the additional convolutional sublayer.

9. A system comprising: One or more memory devices include: a digital image; a neural network including multiple neural network layers having learned parameters and further including a feature backpropagation refinement layer; and one or more processors configured to cause the system to: generate a modified digital image from the digital image using the learned parameters of the multiple neural network layers; determine a consistency loss based on the distance from the user interaction in response to a user interaction with one or more pixels of the modified digital image; generate modified parameters of the feature backpropagation refinement layer using the consistency loss; and generate a refined digital image from the digital image using the learned parameters of the multiple neural network layers and the modified parameters of the feature backpropagation refinement layer.

10. The system of claim 9, wherein the one or more processors are further configured to cause the system to determine the consistency loss by: determining a distance between one or more features corresponding to the one or more pixels interacting with the user within a feature map and features corresponding to other pixels of the modified digital image; penalizing the refinement of a first feature in the feature map using a first penalty, the first penalty being based on a first distance between the first feature distance and the one or more features corresponding to the one or more pixels; and penalizing the refinement of a second feature in the feature map using a second penalty, the second penalty being based on a second distance between the second feature distance and the one or more features corresponding to the one or more pixels, the second distance being greater than the first distance, and the second penalty being greater than the first penalty.

11. The system of claim 9, wherein the one or more processors are further configured to cause the system to: generate the modified digital image using the feature backpropagation refinement layer, the feature backpropagation refinement layer comprising a convolutional sublayer having a first dimension corresponding to the number of feature channels extracted via the plurality of neural network layers and a second dimension corresponding to the dimension of the digital image.

12. The system of claim 9, wherein the one or more processors are further configured to cause the system to generate the modified digital image using the learned parameters of the plurality of neural network layers, together with the unlearned bias parameters of the bias sublayers within the feature backpropagation refinement layer and the unlearned scaling parameters of the convolutional sublayers within the feature backpropagation refinement layer.

13. The system of claim 9, wherein the one or more processors are further configured to cause the system to generate the modified parameters of the feature backpropagation refinement layer, which includes a bias sublayer and a convolutional sublayer, by generating modified bias parameters for the bias sublayer and modified scaling parameters for the convolutional sublayer based on the consistency loss.

14. The system of claim 13, wherein the one or more processors are further configured to cause the system to: generate the modified bias parameters for the bias sublayer including a bias map having dimensions corresponding to the dimensions of the digital image or the dimensions of a feature map representing the digital image.

15. The system of claim 9, wherein the one or more processors are further configured to: cause the system to generate the modified digital image using the plurality of neural network layers and the feature backpropagation refinement layer comprising a plurality of two-dimensional bias maps.

16. The system of claim 9, wherein the one or more processors are further configured to cause the system to: determine a relationship loss between the one or more pixels indicated by the user interaction and the one or more additional pixels indicated by the additional user interaction in response to additional user interaction; and generate the modified parameters of the feature backpropagation refinement layer from the consistency loss and the relationship loss.

17. A computer-implemented method, comprising: A modified digital image is generated from a digital image using a non-interactive digital image neural network, the non-interactive digital image neural network including a neural network layer with learned parameters and a feature backpropagation refinement layer with unlearned initialization parameters; receiving a user interaction identifying one or more pixels for refining the modified digital image; and performing a step for generating modified parameters for the feature backpropagation refinement layer. And by using the modified parameters of the refinement layer to backpropagate the features, the non-interactive digital image neural network is used to generate a refined digital image based on user interaction.

18. The computer-implemented method of claim 17, wherein receiving the user interaction comprises one of: receiving a selection of a region for matting a digital image, receiving a selection for modifying an image fragment within a segmented digital image, receiving a selection for modifying the depth of a region in a depth map of a digital image, receiving a selection for filling pixels within a restored digital image, or receiving a selection for indicating style transfer within a stylized digital image.

19. The computer-implemented method of claim 17, wherein generating the modified digital image comprises: The non-interactive digital image neural network utilizes the feature backpropagation refinement layer, which includes a bias sub-layer with unlearned bias parameters and a convolution sub-layer with unlearned scaling parameters.

20. The computer-implemented method of claim 17, wherein receiving the user interaction includes receiving a multi-pixel user interaction indicating a plurality of pixels from the modified digital image.

Images