Generate synthetic digital images using a neural network with a two-stream encoder architecture

Abstract

Embodiments of the present disclosure relate to generating synthetic digital images using a neural network with a dual-stream encoder architecture. The present disclosure relates to accurately generating synthetic digital images using a neural network with a dual-stream encoder architecture that realistically depicts a foreground object from one digital image for a scene from another digital image. For example, the disclosed system may identify features from a foreground image using a foreground encoder of a neural network, and further identify features from a background encoder using a background encoder. The disclosed system may then utilize a decoder to fuse the features together and generate a synthetic digital image. The disclosed system may train the neural network using an easy-to-difficult data augmentation scheme implemented via self-teaching. The disclosed system may further include the neural network within an end-to-end framework for automation of the image synthesis process.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of digital images, and more particularly to generating composite digital images using neural networks. Background Art

[0002] In recent years, significant improvements have been made in the implementation of various hardware and software platforms for editing digital images. For example, many conventional systems perform image synthesis, thereby combining regions from different digital images to generate a new composite digital image. To illustrate, some conventional systems can generate a composite digital image by superimposing an image of a foreground object (e.g., a person) depicted in one digital image onto a scene (e.g., a background) depicted in another digital image. As a result, the resulting composite digital image can have the appearance that the image of the foreground object was actually captured against the background.

[0003] However, despite these advances, conventional image synthesis systems are still limited by several technical deficiencies that result in inaccurate and inflexible operation. For example, conventional image synthesis systems are generally unable to accurately generate synthetic digital images that realistically depict foreground objects from a first digital image against a background from a second digital image. To illustrate, many conventional systems generate synthetic digital images by extracting a region of a digital image containing a foreground object and then pasting the region directly onto a desired background. However, since the pixels along the boundary are equivalent to a linear combination of both the foreground object and the background, this cut-and-paste approach often results in unnatural artifacts along the boundary of the foreground object. Some conventional image synthesis systems attempt to address boundary artifacts by implementing low-level image blending methods (e.g., Poisson blending, Laplacian pyramid blending, feathering, guided filtering, etc.). However, these methods often introduce other undesirable effects, such as color distortion or non-smooth halo artifacts, etc. Therefore, such conventional systems can introduce additional inaccuracies in the final synthesized digital image.

[0004] In addition to the concern about accuracy, conventional image synthesis systems are also inflexible. For example, conventional image synthesis systems are generally inflexible because they cannot generate synthetic digital images without guidance provided via manual user input. To illustrate, as an alternative to low-level image blending methods, some conventional systems utilize image matting methods to suppress boundary artifacts. However, image matting typically requires manual input (e.g., ternary graphs) to identify the foreground, background, and uncertain regions to be processed. Indeed, many conventional systems cannot complete many other basic editing steps of the image synthesis process (e.g., segmentation, matting, foreground color decontamination) without a large amount of user input. In addition, many conventional systems cannot adapt to the limited availability of training data, resulting in minimal training of models for generating synthetic digital images.

[0005] These and additional problems exist with conventional image synthesis systems. Summary of the invention

[0006] One or more embodiments described herein provide benefits and / or utilize systems, methods, and non-transient computer-readable media to solve one or more of the foregoing or other problems in the art, which utilize a multi-level fusion neural network with a dual-stream encoder architecture to accurately generate realistic synthetic digital images. For example, in one or more embodiments, the system provides a foreground image and a background image to the multi-level fusion neural network. The multi-level fusion neural network can use two separate encoders to extract multi-scale features from each of the foreground image and the background image, respectively. The multi-level fusion neural network can then use a decoder to generate a synthetic digital image based on the multi-scale features, thereby providing a natural blend between the object extracted from the foreground image and the scene depicted in the background image. In one or more embodiments, the system trains the multi-level fusion neural network to generate a synthetic digital image by a data enhancement scheme from easy to difficult implemented via self-teaching. In some embodiments, the system includes the multi-level fusion neural network in a deep learning image synthesis framework that provides automatic end-to-end digital image synthesis. In this way, the system can flexibly generate a synthetic digital image that accurately depicts a foreground object against a background.

[0007] To illustrate, in one or more embodiments, the system identifies a foreground image, a background image, and a segmentation mask corresponding to the foreground image. The system can then generate a synthetic digital image based on the foreground image, the background image, and the segmentation mask. In particular, the system can utilize a foreground encoder of a multi-level fusion neural network to generate a foreground feature map based on the foreground image and the segmentation mask. The system can further utilize a background encoder of a multi-level fusion neural network to generate a background feature map based on the background image and the segmentation mask. Subsequently, the system can utilize a decoder of a multi-level fusion neural network to generate a synthetic digital image based on the foreground feature map and the background feature map.

[0008] Additional features and advantages of one or more embodiments of the present disclosure are summarized in the description which follows, and in part will be apparent from the description, or may be learned by practicing such example embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present disclosure will describe one or more embodiments of the present invention with additional specificity and detail by referring to the accompanying drawings. The following paragraphs briefly describe these drawings, in which:

[0010] Figure 1 illustrates an example environment in which an image composition system may operate in accordance with one or more embodiments;

[0011] Figure 2 illustrates a block diagram of an image synthesis system for generating a composite digital image in accordance with one or more embodiments;

[0012] Figure 3 A schematic diagram of a multi-level fusion neural network according to one or more embodiments is illustrated;

[0013] Figure 4 illustrates a plurality of tables including neural network layers included in a foreground encoder, a background encoder, and a decoder of a multi-stage fusion neural network according to one or more embodiments;

[0014] Figure 5 illustrates a block diagram of generating a segmentation mask according to one or more embodiments;

[0015] FIG. 6A to FIG. 6C A block diagram illustrating training a multi-level fusion neural network to generate a synthetic digital image via self-teaching using an easy-to-difficult data augmentation scheme according to one or more embodiments;

[0016] Figure 7 illustrates a table reflecting experimental results regarding the effectiveness of an image synthesis system according to one or more embodiments;

[0017] Figure 8illustrates another table reflecting additional experimental results regarding the effectiveness of the image synthesis system according to one or more embodiments;

[0018] Fig. 9 Additional tables illustrating other experimental results regarding the effectiveness of image synthesis systems according to one or more embodiments are illustrated;

[0019] Fig.10 An example schematic diagram of an image synthesis system according to one or more embodiments is illustrated;

[0020] Fig.11 A block diagram illustrating a series of actions for generating a composite digital image according to one or more embodiments; and

[0021] Fig.12 A block diagram of an exemplary computing device is illustrated in accordance with one or more embodiments. DETAILED DESCRIPTION

[0022] One or more embodiments described herein include an image synthesis system that utilizes a multi-level fusion neural network with a dual-stream encoder architecture to accurately generate a digital composite image that realistically mixes a foreground object with a background. For example, the image synthesis system may provide a foreground image and a background image to a multi-level fusion neural network to generate a composite digital image. The multi-level fusion neural network may implement a dual-stream encoder pipeline to extract feature maps of different levels for each image, and then use a decoder to fuse the feature maps step by step to generate a composite digital image. The composite digital image may, for example, depict one or more objects extracted from the foreground image for a scene from a background image. In one or more embodiments, the image synthesis system implements a self-teaching strategy to gradually train the multi-level fusion neural network via a data enhancement scheme from easy to difficult. In some embodiments, the image synthesis system implements the multi-level fusion neural network as part of a deep learning image synthesis framework that provides end-to-end automation of the image synthesis process.

[0023] To provide an example, in one or more embodiments, an image synthesis system identifies a foreground image, a background image, and a segmentation mask corresponding to the foreground image. The image synthesis system can then generate a synthesized digital image. In particular, the image synthesis system can generate a foreground feature map based on the foreground image and the segmentation mask using a foreground encoder of a multi-stage fusion neural network. The image synthesis system can further generate a background feature map based on the background image and the segmentation mask using a background encoder of the multi-stage fusion neural network. Subsequently, the image synthesis system can generate a synthesized digital image based on the foreground feature map and the background feature map using a decoder of the multi-stage fusion neural network.

[0024] As just mentioned, in one or more embodiments, the image synthesis system utilizes a multi-stage fusion neural network with two encoders to generate a composite digital image combining a foreground image and a background image. In fact, in one or more embodiments, the multi-stage fusion neural network includes a foreground encoder and a background encoder to generate separate feature maps corresponding to the foreground image and the background image, respectively. The multi-stage fusion neural network can then use a decoder to generate a final composite digital image based on the feature map.

[0025] In one or more embodiments, the multi-stage fusion neural network further utilizes skip links to transfer layer-specific feature maps between the encoder and the decoder. For example, the multi-stage fusion neural network can provide at least one layer-specific feature map from each of the foreground encoder and the background encoder (e.g., generated at the same encoder level) to the decoder (e.g., to a specific layer of the decoder). Therefore, in addition to the feature maps generated by the foreground encoder and the background encoder, the decoder can also generate a synthetic digital image based on the layer-specific feature map.

[0026] As mentioned above, in one or more embodiments, the image synthesis system uses an easy-to-difficult data enhancement scheme implemented via self-teaching to train a multi-level fusion neural network to generate a synthetic digital image. For example, the image synthesis system may first train the multi-level fusion neural network to generate a synthetic digital image based on an easy digital image (e.g., a synthetic digital image including a foreground image depicting a foreground object against a solid color background). The image synthesis system may then use a semi-trained multi-level fusion neural network to generate a more difficult training image (e.g., a more difficult training image including a foreground image depicting a foreground object against a background that varies in detail (in other words, a background that is not a solid color)). The image synthesis system may then use the difficult (hard) training image to further train the multi-level fusion neural network to generate a synthetic digital image based on the difficult digital image. Therefore, the image synthesis system may use the multi-level fusion neural network to generate a training image, which is then further used to train the multi-level fusion neural network to process a more difficult (i.e., more complex) image.

[0027] As further mentioned above, in one or more embodiments, the image synthesis system implements a multi-level fusion neural network within a deep learning image synthesis framework that provides end-to-end functionality for the image synthesis process. In fact, as previously mentioned, in addition to the foreground image and the background image, the multi-level fusion neural network can also utilize a segmentation mask in order to generate a synthetic digital image. Therefore, in some embodiments, the image synthesis system generates a segmentation mask, which is then provided to the multi-level fusion neural network. For example, in one or more embodiments, the image synthesis system uses a foreground segmentation neural network to generate a segmentation mask corresponding to the foreground image. In some embodiments, the image synthesis system may further use a mask refinement neural network to modify the segmentation mask (e.g., refine details along the boundaries of the foreground object depicted in the segmentation mask). The image synthesis system may then provide the modified segmentation mask to the multi-level fusion neural network together with the foreground image and the background image for generating a synthetic digital image.

[0028] The image synthesis system provides several advantages over conventional systems. For example, the image synthesis system improves accuracy. In particular, by using a multi-level fusion neural network with a dual-stream encoder architecture, the image synthesis system can extract features from foreground images and background images at different scales and fuse the features. In fact, the image synthesis system optimizes the use of contextual features and color features from those images, thereby providing a natural blend between foreground objects and the background. In particular, by using a multi-level fusion neural network, the image synthesis system can remove undesirable artifacts caused by color contamination, aliasing, and inaccuracies of segmentation masks.

[0029] In addition, the image synthesis system has increased flexibility. For example, because the multi-level fusion neural network generates a more accurate synthesized digital image, the image synthesis system requires less guidance via manual user input. In particular, the image synthesis system can flexibly perform basic editing steps, such as color decontamination and automatic removal of boundary artifacts. In addition, by incorporating the multi-level fusion neural network into the end-to-end framework, the image synthesis system can perform other basic editing steps, such as segmentation, without the need for user input. In fact, the image synthesis system can generate a synthesized digital image based solely on the provision of a foreground image and a background image. Additionally, the image synthesis system flexibly adapts to the limited availability of training data, thereby generating additional and more complex training data using an easy-to-difficult data enhancement scheme implemented via self-teaching.

[0030] As the foregoing discussion illustrates, the present disclosure utilizes various terms to describe the features and benefits of an image synthesis system. Additional details are now provided regarding the meaning of these terms. For example, as used herein, the term "synthesized digital image" refers to a digital image. In particular, a synthesized digital image may refer to a digital image generated from portions of two or more digital images. For example, a synthesized digital image may include a digital image that includes a foreground object (e.g., a person) from a first digital image and a scene from a second digital image.

[0031] Relatedly, as used herein, the term "foreground image" refers to a digital image that depicts a foreground object. In particular, a foreground image may refer to a digital image that can be used to provide a foreground object for a composite digital image. For example, a foreground image may include a digital image that depicts a person or other object, which is used to generate a composite digital image having the same depiction of the person or object.

[0032] Similarly, as used herein, the term "background image" refers to a digital image depicting a scene. In particular, a background image may refer to a digital image depicting a scene that can be used to provide a background for a composite digital image. For example, a background image may include a digital image depicting a scene that is then used to generate a composite digital image depicting the same scene as the background.

[0033] As used herein, the term "segmentation mask" refers to the identification of pixels in an image that represent an object. In particular, a segmentation mask may refer to an image filter that helps to divide a digital image into separate parts. For example, a segmentation mask may include a filter corresponding to a digital image (e.g., a foreground image) that identifies a portion of the digital image that belongs to a foreground object (i.e., a pixel of the digital image) and a portion of the digital image that belongs to a background. For example, a segmentation map may refer to a map of a digital image that has an indication of whether the pixel is part of an object (e.g., a foreground object) for each pixel. In this implementation, the indication may include a binary indication (1 for pixels that belong to an object and 0 for pixels that do not belong to an object). In an alternative implementation, the indication may include a probability (e.g., a number between 1 and 0) indicating the likelihood that the pixel belongs to an object. In this implementation, the closer the value is to 1, the more likely the pixel is to belong to the foreground or object, and vice versa.

[0034] In addition, as used herein, the term "multi-level fusion neural network" refers to a computer algorithm or model that generates a synthetic digital image. In particular, a multi-level fusion neural network may refer to a computer algorithm that analyzes a foreground image and a background image to generate a synthetic digital image. For example, a multi-level fusion neural network may refer to a neural network. In one or more embodiments, the multi-level fusion neural network includes a dual-stream encoder architecture. For example, the multi-level fusion neural network may include a "foreground encoder" for processing a foreground image and a "background encoder" for processing a background image.

[0035] Additionally, as used herein, the term "foreground segmentation neural network" refers to a computer algorithm or model that generates a segmentation mask. In particular, a foreground segmentation neural network may refer to a computer algorithm that analyzes a foreground image to generate a segmentation mask corresponding to the foreground image. For example, a foreground segmentation neural network may generally include a neural network, or more specifically, include a significant object segmentation model, such as a portrait segmentation model.

[0036] Additionally, as used herein, the term "mask refinement neural network" refers to a computer algorithm or model that modifies a segmentation mask. In particular, a mask refinement neural network may refer to a computer algorithm that analyzes a segmentation mask to generate a modified (e.g., refined) segmentation mask. For example, a mask refinement neural network may include a neural network.

[0037] As used herein, the term "neural network" refers to a machine learning model that can be adjusted (e.g., trained) based on inputs to approximate unknown functions. In particular, the term "neural network" may include a model of interconnected artificial neurons (e.g., organized in layers) that communicate and learn to approximate complex functions and generate outputs based on multiple inputs provided to the model. For example, the term "neural network" includes one or more machine learning algorithms. Additionally, a neural network may refer to an algorithm (or set of algorithms) that implements a deep learning technique that utilizes a set of algorithms to model high-level abstractions in data.

[0038] Additionally, as used herein, the term "feature map" refers to a collection of one or more eigenvalues. In particular, a feature map may refer to a grid, matrix, vector, or different multidimensional collection of eigenvalues. In addition, a feature map may include the locations or positions of the individual eigenvalues ​​that make up the feature map. For example, a feature map may include eigenvalues ​​corresponding to pixels of a digital image, so that the feature map represents the digital image in the form of eigenvalues. Specifically, a "foreground feature map" may refer to a feature map corresponding to a foreground image, and a "background feature map" may refer to a feature map corresponding to a background image. Similarly, a "layer-specific feature map" may refer to a feature map generated by a specific neural network layer of an encoder, rather than a feature map output by the encoder as a whole.

[0039] In addition, as used herein, the term "easy to difficult data enhancement scheme" refers to the process of training a model. In particular, the easy to difficult data enhancement scheme may refer to a method of training a model using progressively more complex training data. For example, the easy to difficult data enhancement scheme may refer to a process of training a model (e.g., a multi-level fusion neural network) to generate a synthetic digital image by training the model using an easy digital image (e.g., an easy foreground image), and then further training the model using a difficult digital image (e.g., a difficult foreground image). More details about the easy to difficult data enhancement scheme will be provided below.

[0040] As used herein, the term "easy foreground image" refers to a digital image with a relatively simple background (e.g., compared to a difficult foreground image). In particular, an easy foreground image may refer to a digital image that depicts a foreground object against a relatively simple background, and which is used to generate a synthetic digital image that depicts the foreground object against another background. For example, an easy foreground image may include a digital image that depicts a foreground object against a solid color background. An "easy training foreground image" may refer to an easy foreground image that is used to train a model to generate a synthetic digital image based on a foreground image with a relatively simple background (e.g., as part of an easy-to-difficult data augmentation scheme).

[0041] Similarly, as used herein, the term "difficult foreground image" may refer to a digital image having a relatively complex background (e.g., compared to an easy foreground image). In particular, a difficult foreground image may refer to a digital image that depicts a foreground object against a relatively complex background, and the digital image is used to generate a synthetic digital image that depicts the foreground object against another background. For example, a difficult foreground image may include a digital image that depicts a foreground object against a background that varies in detail (i.e., a background that is not a solid color). A "difficult training foreground image" may refer to a difficult foreground image that is used to train a model to generate a synthetic digital image based on a foreground image having a relatively complex background (e.g., as part of an easy-to-difficult data augmentation scheme).

[0042] Additional details about the image synthesis system will now be provided with reference to the accompanying drawings. Figure 1 A schematic diagram of an exemplary system environment ("environment") 100 is illustrated in which an image composition system 106 may be implemented. Figure 1 As illustrated in FIG. 1 , environment 100 may include server(s) 102 , network 108 , client devices 110 a - 110 n , and users 114 a - 114 n .

[0043] although Figure 1Although environment 100 is depicted as having a particular number of components, environment 100 may have any number of additional or alternative components (e.g., any number of servers, client devices, or other components that communicate with image composition system 106 via network 108). Similarly, although Figure 1 A particular arrangement of server(s) 102 , network 108 , client devices 110 a - 110 n , and users 114 a - 114 n is illustrated, but various additional arrangements are possible.

[0044] The server(s) 102, network 108, and client devices 110a to 110n may be directly or indirectly (e.g., via the following description with respect to Fig.12 In addition, the server(s) 102 and the client devices 110a to 110n may include computing devices (including, for example, Fig.12 One or more computing devices discussed in more detail).

[0045] As mentioned above, the environment 100 includes (multiple) servers 102. (Multiple) servers 102 can generate, store, receive and / or transmit data, including synthesized digital images. For example, (multiple) servers 102 can receive a foreground image and a background image (and possibly a segmentation mask corresponding to the foreground image) from a client device (e.g., one of the client devices 110a to 110n) and transmit the synthesized digital image back to the client device. In one or more embodiments, (multiple) servers 102 include data servers. (Multiple) servers 102 can also include communication servers or web hosting servers.

[0046] like Figure 1 As shown in , the server(s) 102 may include an image editing system 104. In particular, the image editing system 104 may provide functionality by which a user (e.g., any one of the users 114a to 114n) may generate, edit, manage, and / or store digital images. For example, the user 114a may use the client device 110a to generate or otherwise access one or more digital images. Subsequently, the user 114a may use the client device 110a to send the digital image(s) to the image editing system 104 hosted on the server(s) 102 via the network 108. The image editing system 104 then provides a number of options by which the user 114a may edit the digital image(s), store the digital image(s), and subsequently search, access, and view the digital image(s).

[0047] Additionally, the server(s) 102 include an image synthesis system 106. In particular, in one or more embodiments, the image synthesis system 106 generates a synthesized digital image using the server(s) 102. For example, the image synthesis system 106 can use the server(s) 102 to generate a synthesized digital image based on a foreground image, a background image, and a segmentation mask corresponding to the foreground image using a multi-level fusion neural network.

[0048] For example, in one or more embodiments, the server(s) 102 may identify a foreground image, a background image, and a segmentation mask. The server(s) 102 may then use a foreground encoder of a multi-stage fusion neural network to generate a foreground feature map based on the foreground image and the segmentation mask. The server(s) 102 may further use a background encoder of a multi-stage fusion neural network to generate a background feature map based on the background image and the segmentation mask. Based on the foreground feature map and the background feature map, the server(s) 102 may use a decoder of a multi-stage fusion neural network to generate a synthesized digital image.

[0049] In one or more embodiments, the client devices 110a to 110n include computer devices that allow users of the devices (e.g., users 114a to 114n) to access, edit, and view digital images including synthesized digital images. For example, the client devices 110a to 110n may include smart phones, tablet computers, desktop computers, laptop computers, or other electronic devices. The client devices 110a to 110n may include one or more applications (e.g., image synthesis application 112) that allow users 114a to 114n to access, edit, and view digital images including synthesized digital images. For example, the image synthesis application 112 may include a software application installed on the client devices 110a to 110n. Additionally or alternatively, the image synthesis application 112 may include a software application hosted on (multiple) servers 102, which may be accessed by the client devices 110a to 110n through another application such as a web browser.

[0050] Image synthesis system 106 may be implemented in whole or in part by separate elements of environment 100. In practice, although Figure 1 The image composition system 106 is illustrated as being implemented with respect to (multiple) servers 102, but the various components of the image composition system 106 may be implemented in any component of the environment 100. In particular, a portion or all of the image composition system 106 may be implemented by a client device. Fig.10 The components of image composition system 106 are discussed in more detail.

[0051] As mentioned above, the image synthesis system 106 may generate a composite digital image. Figure 22 is a block diagram of an image synthesis system 106 for generating a composite digital image 212 according to one or more embodiments. Figure 2 , the image synthesis system 106 identifies the foreground image 202 and the background image 206. In one or more embodiments, the image synthesis system 106 identifies the foreground image 202 and the background image 206 by receiving the foreground image 202 and the background image 206 from an external source, such as a third-party system or a client device. In some embodiments, the image synthesis system 106 identifies the foreground image 202 and the background image 206 from a database storing digital images.

[0052] like Figure 2 As shown in , foreground image 202 depicts foreground object 204 (i.e., a person) against a background. Although foreground image 202 depicts one foreground object, in one or more embodiments, image synthesis system 106 may identify a foreground image that depicts multiple foreground objects against a background. Figure 2 As shown in , background image 206 depicts scene 208 (ie, a scene that may be used as a background).

[0053] like Figure 2 As shown in FIG. 1 , the image synthesis system 106 utilizes a multi-stage fusion neural network 210 to analyze the foreground image 202 and the background image 206. In one or more embodiments, the multi-stage fusion neural network 210 includes a dual-stream encoder neural network architecture for separately analyzing the foreground image 202 and the background image 206. Figure 3 to Figure 4 The architecture of the multi-stage fusion neural network 210 is discussed in more detail.

[0054] like Figure 2 As shown in , based on the analysis of the foreground image 202 and the background image 206 by the multi-level fusion neural network 210, the image synthesis system 106 generates a synthesized digital image 212. In particular, the synthesized digital image 212 depicts the foreground object 204 from the foreground image 202 positioned above the scene 208 from the background image 206.

[0055] It should be noted that although the scene 208 of the background image 206 does not depict significant (i.e., discernible) foreground objects, in one or more embodiments, the image synthesis system 106 can identify the background image as having a scene that depicts one or more significant foreground objects. Indeed, in some embodiments, the image synthesis system 106 can identify various digital images as background images. As a result of identifying a particular digital image as a background digital image, regardless of whether the scene already depicts one or more foreground objects, the resulting synthesized digital image can depict foreground objects for the scene depicted by the background image. Additionally, although the image synthesis system 106 is discussed below as performing image synthesis in the context of a portrait of a person depicted as a foreground object, it should be noted that one or more embodiments of the image synthesis system 106 can generate synthesized digital images that depict various objects as foreground objects.

[0056] As mentioned above, the image synthesis system 106 may generate a synthesized digital image from a foreground image and a background image using a multi-stage fusion neural network. Figure 3 A schematic diagram of a multi-level fusion neural network 300 is illustrated according to one or more embodiments.

[0057] like Figure 3 As shown in FIG. 1 , the multi-stage fusion neural network 300 includes a dual-stream encoder architecture. Using the dual-stream encoder architecture, the multi-stage fusion neural network 300 can analyze the foreground image and the background image separately. In particular, as Figure 3 As shown in FIG. 1 , the multi-stage fusion neural network 300 includes a foreground encoder 302 for analyzing a foreground image and a background encoder 304 for analyzing a background image.

[0058] like Figure 3As illustrated, the multi-stage fusion neural network 300 also includes a decoder 306 and skip links 308a to 308f. The decoder 306 can generate a synthetic digital image based on the analysis of both the foreground encoder 302 and the background encoder 304 (i.e., based on the feature maps generated by the foreground encoder and the background encoder 302, 304). In one or more embodiments, each of the skip links 308a to 308f provides a feature map generated by a specific layer of either the foreground encoder 302 or the background encoder 304 (i.e., a layer-specific feature map) to a specific layer of the decoder 306. In one or more embodiments, one or more of the skip links 308a to 308c provides a layer-specific feature map from a neural network layer of the foreground encoder 302 that is at the same encoder level as a neural network layer of the background encoder 304 from which one of the skip links 308d to 308f provides a layer-specific feature map. However, in some embodiments, the skip links 308a to 308c provide a layer-specific feature map generated by a neural network layer of the foreground encoder 302 that is at a different encoder level. Additionally, although Figure 3 The multi-stage fusion neural network 300 is illustrated as having a specific number of skip links, but in some embodiments, the multi-stage fusion neural network 300 may include more or fewer skip links.

[0059] like Figure 3 As shown in FIG. 1 , the multi-level fusion neural network 300 receives a foreground image 310 and a background image 312 as input. In addition, the multi-level fusion neural network 300 receives or optionally generates a segmentation mask 314 and a reverse segmentation mask 316 corresponding to the foreground image 310. In one or more embodiments, the multi-level fusion neural network 300 receives input as a concatenation between a digital image and a corresponding mask. For example, Figure 3 As shown in FIG, the multi-stage fusion neural network 300 receives a concatenation of a foreground image 310 and a segmentation mask 314 at a foreground encoder 302. Similarly, the multi-stage fusion neural network 300 receives a concatenation of a background image 312 and a reverse segmentation mask 316 at a background encoder 304.

[0060] The multi-level fusion neural network 300 may use the foreground encoder 302 to generate a foreground feature map based on the foreground image 310 and the segmentation mask 314. The multi-level fusion neural network 300 may further use the background encoder 304 to generate a background feature map (not shown) based on the background image 312 and the inverse segmentation mask 316.

[0061] In one or more embodiments, the multi-level fusion neural network 300 provides the foreground feature map and the background feature map generated by the foreground encoder 302 and the background encoder 304, respectively, to the decoder 306. For example, the multi-level fusion neural network 300 may combine the foreground feature map and the background feature map (e.g., via concatenation) and provide the combined feature map to the decoder 306. The multi-level fusion neural network 300 may then utilize the decoder 306 to generate a synthesized digital image 318 based on the foreground feature map and the background feature map. Figure 3 As shown in , in some embodiments, the multi-stage fusion neural network 300 further provides the layer-specific feature maps from the foreground encoder 302 and the background encoder 304 to the decoder 306 (i.e., to a specific layer of the decoder 306) using skip links 308a to 308f. Therefore, the decoder 306 can further generate a synthetic digital image 318 based on the layer-specific feature maps.

[0062] By using a multi-level fusion neural network, the image synthesis system 106 operates more accurately than conventional systems. In fact, because the multi-level fusion neural network provides more natural blending at the boundaries of foreground objects, the image synthesis system 106 improves the realism with which the foreground objects are depicted against the background, thereby removing artifacts caused by color contamination, aliasing, and inaccuracies of the segmentation mask. As described below with respect to Figures 7 to 9 As will be discussed in greater detail, the image synthesis system 106 improves the quality of the resulting synthesized digital image in several metrics.

[0063] In one or more embodiments, the foreground encoder and the background encoder of the multi-stage fusion neural network each include multiple neural network layers. In addition, in some embodiments, the decoder of the multi-stage fusion neural network includes multiple neural network layers. Figure 4 Illustrated are multiple tables including neural network layers included in a foreground encoder, a background encoder, and a decoder of a multi-stage fusion neural network according to one or more embodiments.

[0064] like Figure 4 As shown in Tables 402 and 404 in , in one or more embodiments, the foreground encoder (referred to as "FGE") and the background encoder (referred to as "BGE") of the multi-level fusion neural network both include the same encoder architecture. In particular, the foreground encoder and the background encoder can include the same number of neural network layers and the same type of neural network layers at each encoder level. However, in one or more embodiments, the foreground encoder and the background encoder include different encoder architectures (i.e., different numbers of neural network layers and / or different neural network layers at one or more of the encoder levels).

[0065] like Figure 4As shown in Table 406 in FIG. 4 , in one or more embodiments, the decoder of the multi-stage fusion neural network includes multiple neural network layers, including multiple neural network layers that perform concatenation operations (i.e., “D-Concat1”, “D-Concat2”, and “D-Concat3”). Each of these neural network layers combines the output of the previous neural network layer (or the foreground feature map and the background feature map output by the foreground encoder and the background encoder, respectively) with the output of the foreground encoder and the background encoder via a skip link (as described above with reference to FIG. 4 ). Figure 3 For example, the neural network layer of the decoder labeled “D-Concat1” concatenates the foreground feature map and the background feature map with the layer-specific feature maps generated by the neural network layers labeled “FGE-TD2” and “BGE-TD2”. Similarly, the neural network layer of the decoder labeled “D-Concat2” concatenates the output of the “D-Non-local” layer with the layer-specific feature maps generated by the “FGE-TD1” and “BGE-TD1” neural network layers of the foreground encoder and the background encoder, respectively. In addition, the “D-Concat 3” layer of the decoder concatenates the output of the “D-Dense2(12)” layer with the layer-specific feature maps generated by the “FGE-Conv1-BN-Relu” and “BGE-Conv1-BN-Reul” neural network layers of the foreground encoder and the background encoder, respectively. As mentioned above, however, the multi-stage fusion neural network may include more than one neural network layer. Figure 3 to Figure 4 The skip links discussed may be more or less skip links. In addition, in one or more embodiments, the multi-stage fusion neural network may utilize skip links to provide layer-specific feature maps from different neural network layers of the foreground encoder and the background encoder and / or to different neural network layers of the decoder.

[0066] Therefore, in one or more embodiments, the image synthesis system 106 identifies a foreground image, a background image, and a segmentation mask corresponding to the foreground image. The image synthesis system 106 can then generate a synthesized digital image based on the foreground image, the background image, and the segmentation mask using a multi-level fusion neural network. Figure 3 to Figure 4 The described algorithms and actions may include corresponding structures to perform steps for generating a synthetic digital image using a multi-level fusion neural network based on a foreground image, a background image, and a segmentation mask. Figure 3 to Figure 4 The described multi-level fusion neural network architecture may include corresponding structures to perform steps for generating a synthetic digital image using a multi-level fusion neural network based on a foreground image, a background image, and a segmentation mask.

[0067] As mentioned above, in one or more embodiments, the image synthesis system 106 implements a multi-level fusion neural network within a framework that provides end-to-end image synthesis. In fact, as mentioned above, in one or more embodiments, the image synthesis system 106 identifies a segmentation mask corresponding to the foreground image and uses the segmentation mask in addition to the foreground image and the background image when generating a synthesized digital image. Therefore, in one or more embodiments, the image synthesis system generates a segmentation mask corresponding to the foreground image. Figure 5 A block diagram of generating a segmentation mask in accordance with one or more embodiments is illustrated.

[0068] like Figure 5 As shown in , the image synthesis system provides a foreground image 502 (i.e., a digital image to be used as a foreground image by the multi-stage fusion neural network to generate a synthesized digital image) to a foreground segmentation neural network 504. In one or more embodiments, the foreground segmentation neural network 504 includes a model (e.g., a neural network) that has been previously trained to generate a segmentation mask. In one or more embodiments, the image synthesis system 106 trains the foreground segmentation neural network 504 to generate a segmentation mask. However, in some embodiments, a third-party system trains the foreground segmentation neural network 504 to generate a segmentation mask.

[0069] The image synthesis system 106 utilizes the foreground segmentation neural network 504 to generate a raw segmentation mask 506 based on the foreground image 502. Figure 5 As can be seen in FIG, the original segmentation mask 506 includes two separate parts corresponding to the two parts identified by the foreground segmentation neural network 504 from the foreground image 502 - the foreground object 508 and the background 510. Figure 5 As can be further seen in FIG. 5 , the boundary between the foreground object 508 and the background 510 is not clearly defined (i.e., the boundary has a certain degree of fuzziness). In fact, in one or more embodiments, the foreground segmentation neural network 504 generates a segmentation mask that does not clearly delineate the boundary between the foreground object and the background. For example, in some embodiments, the foreground segmentation neural network 504 processes the foreground image at a low resolution, thereby generating a segmentation mask that is affected by upsampling artifacts (such as jagged boundaries). However, in some embodiments, the foreground segmentation neural network 504 can generate a segmentation mask with a clearly defined boundary between the foreground object and the background.

[0070] like Figure 5, the image synthesis system 106 then provides the original segmentation mask 506 to the mask refinement neural network 512. In one or more embodiments, the mask refinement neural network 512 includes a model (e.g., a neural network) that has been previously trained to modify (e.g., refine) the segmentation mask. In one or more embodiments, the image synthesis system 106 trains the mask refinement neural network 512 to modify the segmentation mask. For example, the image synthesis system 106 can train the mask refinement neural network 512 by using sample image patches of various sizes so that the mask refinement neural network 512 learns to focus on different levels of local details. In practice, the image synthesis system 106 can provide the sampled patches (e.g., generated by the foreground segmentation neural network 504) and the pre-computed segmentation mask to the mask refinement neural network 512, and then modify its parameters using the determined loss. However, in some embodiments, a third-party system trains the mask refinement neural network 512 to modify the segmentation mask.

[0071] In one or more embodiments, the mask refinement neural network 512 has a similar architecture to the foreground segmentation neural network 504, except that the mask refinement neural network 512 takes a four-channel RGB-A digital image as input, where the fourth channel contains the original segmentation mask (e.g., the original segmentation mask 506). In addition, in some embodiments, the image synthesis system 106 trains the mask refinement neural network 512 using the same training data and the same loss function used to train the foreground segmentation neural network 504. For example, in one or more embodiments, the image synthesis system 106 uses the same cross entropy loss function to train the foreground segmentation neural network 504 and the mask refinement neural network 512.

[0072] like Figure 5 As shown in , the image synthesis system 106 utilizes the mask refinement neural network 512 to modify the original segmentation mask 506. In practice, the mask refinement neural network 512 generates a modified segmentation mask 514 based on the original segmentation mask 506. Figure 5 As can be seen in , the modified segmentation mask 514 includes more clearly defined boundaries between the foreground objects 516 and the background 518 when compared to the original segmentation mask 506 .

[0073] The image synthesis system 106 may provide the modified segmentation mask 514 together with the corresponding foreground image and background image to the multi-stage fusion neural network to generate a synthetic digital image. In one or more embodiments, the image synthesis system 106 provides the original segmentation mask 506 to the multi-stage fusion neural network to generate a synthetic digital image. In some embodiments, the image synthesis system 106 receives or otherwise accesses (rather than generates) a segmentation mask (i.e., the original segmentation mask or the modified segmentation mask) for generating a synthetic digital image. For example, the image synthesis system 106 receives the segmentation mask from a user device or a third-party system. The image synthesis system 106 may also access the segmentation mask from a database or other storage device.

[0074] In one or more embodiments, the image synthesis system 106 further generates an inverse segmentation mask based on either the original segmentation mask 506 or the modified segmentation mask 514 (provided to any of the multi-stage fusion neural networks). The image synthesis system 106 can then provide the inverse segmentation mask to the multi-stage fusion neural network for generating a synthesized digital image.

[0075] Thus, in one or more embodiments, the image synthesis system 106 implements a multi-stage fusion neural network within the framework that provides end-to-end image synthesis by including a foreground segmentation neural network within the framework. Additionally, the image synthesis system 106 can include a mask refinement neural network within the framework. Indeed, in one or more embodiments, a higher quality segmentation mask produces a higher quality synthesized digital image. Thus, the image synthesis system 106 can use the framework to identify a foreground image and a background image, generate a segmentation mask corresponding to the foreground image, and then generate a synthesized digital image based on the foreground image, the background image, and the segmentation mask.

[0076] In one or more embodiments, the image synthesis system 106 trains the components of the end-to-end framework (i.e., the foreground segmentation neural network, the mask refinement neural network, and the multi-level fusion neural network) together. However, in some embodiments, the image synthesis system 106 trains each component separately. As mentioned above, the image synthesis system 106 may also rely on a third-party system to train the foreground segmentation neural network and / or the mask refinement neural network.

[0077] By employing a framework that provides end-to-end image synthesis, the image synthesis system 106 operates more flexibly than conventional systems. In fact, the image synthesis system 106 improves the automation of the image synthesis process. Thus, the image synthesis system 106 can generate a composite digital image without guidance provided via manual user input, which many conventional systems rely on.

[0078] As mentioned above, the image synthesis system 106 can utilize the data enhancement scheme from easy to difficult to train the multi-level fusion neural network to generate the synthetic digital image. In addition, the image synthesis system 106 can use the self-teaching method to achieve training. FIG. 6A to FIG. 6C Illustrated is a block diagram of training a multi-level fusion neural network to generate a synthetic digital image via self-teaching using an easy-to-hard data augmentation scheme in accordance with one or more embodiments.

[0079] In particular, Fig. 6A A block diagram of an image synthesis system 106 for training a multi-level fusion neural network 616 to generate a synthetic digital image based on an easy foreground image according to one or more embodiments is illustrated. Fig. 6A As shown in , the image synthesis system 106 implements training by providing an easy training foreground image 604 to a multi-stage fusion neural network 616. The easy training foreground image 604 depicts a foreground object 610 against a solid color background 612. In one or more embodiments, the image synthesis system 106 generates the easy training foreground image 604 by synthesizing a matted image (i.e., containing the foreground object 610) from a matted dataset with a solid color background 612 using an alpha channel of the matted image.

[0080] like Fig. 6A As shown in , the image synthesis system 106 further provides the training background image 606 to the multi-level fusion neural network 616. The training background image 606 depicts a scene 614 that is used as a background for the resulting predicted synthesized digital image. In one or more embodiments, the image synthesis system 106 retrieves the training background image 606 from a database of digital images. For example, the image synthesis system 106 can retrieve digital images from the Internet and use the retrieved digital images as the training background image 606.

[0081] like Fig. 6A As shown in , the image synthesis system 106 further provides the training segmentation mask 602 and the reverse training segmentation mask 608 corresponding to the easy training foreground image 604 to the multi-layer fusion neural network 616. In one or more embodiments, the image synthesis system 106 generates the training segmentation mask 602 and the reverse training segmentation mask 608 based on the easy training foreground image 604. However, in some embodiments, the image synthesis system 106 uses a third-party system to retrieve or otherwise access the training segmentation mask 602 and / or the reverse training segmentation mask 608.

[0082] The image synthesis system 106 utilizes a multi-level fusion neural network 616 to generate a predicted synthesized digital image 618 based on the easy training foreground image 604, the training background image 606, the training segmentation mask 602, and the reverse training segmentation mask 608. In one or more embodiments, the multi-level fusion neural network 616 generates the predicted synthesized digital image 618 as described above with respect to Figure 3 to Figure 4 The predicted synthetic digital image 618 discussed. The image synthesis system 106 then uses a loss function 620 to determine the loss (i.e., error) caused by the multi-stage fusion neural network 616 based on a comparison between the predicted synthetic digital image 618 and the target synthetic digital image 622. In one or more embodiments, the image synthesis system 106 generates the target synthetic digital image 622 using the same alpha channel of the matted image used to generate the easy training foreground image 604 (e.g., the alpha channel used to synthesize the foreground object 610 with the solid color background 612).

[0083] In one or more embodiments, the loss function 620 includes a plurality of different losses. In practice, the image synthesis system 106 can determine the loss caused by the multi-stage fusion neural network 616 by determining a plurality of different losses and then combining the losses. For example, in one or more embodiments, the image synthesis system 106 determines the loss caused by the multi-stage fusion neural network 616 by comparing the predicted synthesized digital image 618 with the target synthesized digital image 622 to determine the L1 loss and the perceptual loss. The image synthesis system 106 can then combine the L1 loss and the perceptual loss to determine the combined loss as follows:

[0084] L all =L1+λ p L p (1)

[0085] In Equation 1, λ P Denotes the perceptual loss L P In one or more embodiments, the image synthesis system 106 weights the L1 loss and the perceptual loss equally. However, in some embodiments, the image synthesis system 106 applies a relatively higher or lower weight to the perceptual loss.

[0086] In one or more embodiments, the image synthesis system 106 determines the perceptual loss by using a pre-trained visual geometry group (VGG) neural network to extract feature vectors for both the predicted synthetic digital image 618 and the target synthetic digital image 622. The image synthesis system 106 then compares the feature vectors to determine the perceptual loss. In one or more embodiments, the image synthesis system 106 determines the perceptual loss as described by J. Johnson et al. in the document “Perceptual Losses for Real-Time Style Transfer and Super-Resolution” (https: / / arxiv.org / abs / 1603.08155), which is incorporated herein by reference in its entirety. In one or more embodiments, the image synthesis system evaluates the perceptual loss of the relu 1-1 and relu 2-1 layers of the VGG neural network.

[0087] like Fig. 6A As shown in , the image synthesis system 106 then back-propagates the determined loss to the multi-level fusion neural network 616 (as indicated by dashed line 624) to modify its parameters. Thus, with each iteration of training, the image synthesis system 106 gradually improves the accuracy with which the multi-level fusion neural network 616 can generate a synthetic digital image based on the easy foreground image. As shown, the image synthesis system 106 can thus generate a semi-trained multi-level fusion neural network 626.

[0088] Figure 6B The diagram illustrates a method for utilizing a multi-level fusion neural network 640 (ie, Fig. 6A 644 and 646. The block diagram of the image synthesis system 106 is shown in FIG. 646 , which uses a semi-trained multi-level fusion neural network 626 to generate a difficult training triplet 648 to further train the multi-level fusion neural network 640. As shown, the difficult training triplet 648 includes a target synthetic digital image 642, a difficult training foreground image 644, and a training background image 646.

[0089] like Figure 6BAs shown in , the image synthesis system 106 generates a difficult training triplet 648 based on an easy training foreground image 630, a first training background image 632, a second training background image 634, a segmentation mask 636 corresponding to the easy training foreground image 630, and an inverse segmentation mask 638 corresponding to the easy training foreground image 630. In particular, the image synthesis system 106 utilizes a multi-level fusion neural network 640 to generate a target synthetic digital image 642 based on the easy training foreground image 630, the first training background image 632, the segmentation mask 636, and the inverse segmentation mask 638. The image synthesis system 106 further utilizes the multi-level fusion neural network 640 to generate a difficult training foreground image 640 based on the easy training foreground image 644, the second training background image 634, the segmentation mask 636, and the inverse segmentation mask 638. In one or more embodiments, the multi-level fusion neural network 640 generates the target synthetic digital image 642 as described above with respect to Figure 3 to Figure 4 The target synthetic digital image 642 and the difficult training foreground image 644 are discussed. Additionally, the image synthesis system 106 includes the first training background image 632 in the difficult training triplet 648 as the training background image 646.

[0090] Figure 6C FIG. 1 is a block diagram of an image synthesis system 106 that further trains a multi-level fusion neural network 640 (ie, a multi-level fusion neural network 640) using a difficult training triplet 648 according to one or more embodiments. Fig. 6A A semi-trained multi-level fusion neural network 626) is used to generate a synthetic digital image based on a difficult foreground image. Figure 6C As shown in FIG. 1 , the image synthesis system 106 may utilize a multi-level fusion neural network 640 to generate a predicted synthesized digital image 650 based on a difficult training foreground image 644, a training background image 646, a segmentation mask 636, and a reverse segmentation mask 638. In one or more embodiments, the multi-level fusion neural network 640 generates the predicted synthesized digital image 650 as described above with respect to FIG. Figure 3 to Figure 4 The predicted composite digital image 650 is discussed.

[0091] The image synthesis system 106 then uses the loss function 652 to determine the loss incurred by the multi-stage fusion neural network 640 based on a comparison between the predicted synthesized digital image 650 and the target synthesized digital image 642. In one or more embodiments, the image synthesis system 106 utilizes the same loss function as described above with respect to Fig. 6A The same loss function as discussed and expressed in Eq. 1.

[0092] like Figure 6CAs shown in , the image synthesis system 106 then back-propagates the determined loss to the multi-level fusion neural network 640 (as indicated by the dashed line 654) to modify its parameters. Therefore, with each iteration of training, the image synthesis system 106 gradually improves the accuracy that the multi-level fusion neural network 640 can use to generate a synthetic digital image based on the difficult foreground image. As shown, the image synthesis system 106 can thus generate a trained multi-level fusion neural network 656.

[0093] Therefore, the image synthesis system 106 can implement a self-teaching training method by utilizing a multi-level fusion neural network to generate additional training images, and then using those training images to further train the multi-level fusion neural network. Additionally, the image synthesis system 106 can implement an easy-to-difficult data enhancement scheme by increasing the complexity of the training images during the training process. By using self-teaching and easy-to-difficult data enhancement, the image synthesis system 106 operates more flexibly than conventional systems. In particular, even in the case of limited availability of training images, the image synthesis system 106 can flexibly provide extensive training to the multi-level fusion neural network. Therefore, due to the lack of training images, the image synthesis system 106 can circumvent the limitations on training performed on conventional systems.

[0094] As mentioned above, utilizing a multi-level fusion neural network allows the image synthesis system 106 to more accurately generate a synthesized digital image that realistically depicts a foreground object from one digital image against a scene (i.e., background) provided by another digital image. Researchers have conducted studies to determine the accuracy of one or more embodiments of a multi-level fusion neural network used by the image synthesis system 106. Figures 7 to 9 Each illustrates a table reflecting experimental results regarding the effectiveness of a multi-stage fusion neural network used by an image synthesis system according to one or more embodiments.

[0095] Figure 7 A table reflecting experimental results based on the performance of one or more embodiments of the image synthesis system 106 is illustrated. The researchers used the above reference FIG. 6A to FIG. 6C The discussed self-teaching data augmentation schemes from easy to difficult were combined with a synthetic dataset based on matting to train a multi-level fusion neural network. The researchers performed the optimization using the Adam optimization algorithm.

[0096] Figure 7The table shown in compares the performance of the image synthesis system 106 to the performance of one or more other methods commonly used by conventional systems. For example, the results also measure the performance of various traditional blending-based synthesis methods, such as feathering and Laplacian pyramid blending. The results further show the performance of the cutout-based synthesis method using prior art cutout methods (such as Closed, KNN, and Info-flow). The results include the performance of the copy-paste method as a baseline.

[0097] The researchers provided the same refined segmentation mask for each measurement method. The researchers trained the foreground segmentation neural network and the mask refinement neural network using the DUTS, MSRA-10K, and portrait segmentation datasets, and optimized the models using the Adam optimization algorithm. For the copy-and-paste method, the refined segmentation mask was used as a soft alpha matte for synthesis. For the feathering method, the researchers applied a Gaussian blur with σ=2 to soften the mask. For Laplacian pyramid blending, the researchers used the OpenCV implementation. For the cutout-based method, the researchers binarized the refined segmentation mask and then generated a pseudo-ternary map by marking a narrow band of width 16 as unknown.

[0098] like Figure 7 The results are shown in , using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) measured in dB to compare the performance of each method. Using both metrics, higher scores correspond to higher quality image synthesis (i.e., better accuracy). As shown in Figure 7 As can be seen, the image synthesis system 106 performs significantly better than the other models, scoring 26.06 PSNR and 0.9291 SSIM.

[0099] Figure 8 Another table reflecting additional experimental results is illustrated, which compares the performance of the image synthesis system 106 with the performance of one or more conventional systems using conventional image synthesis methods. In particular, Figure 8 The results shown in illustrate the perceived quality provided by each method as collected via a user study. The user study involved forty-four participants, including image editing experts. Each participant was shown twenty image sets, each consisting of a foreground image and the synthetic results of the compared methods. In each image set, the researchers asked the participants to rank the three most favorite results. Figure 8 The results shown in provide the average ranking of the compared methods, where lower averages correspond to better perceptual quality.

[0100] and Figure 7The results compare the performance of the image synthesis system 106 to the performance of Laplacian pyramid blending (Lap-Pyramid), closed form (Closed), information flow (Info-flow), and copy-paste as a baseline. The results further compare the performance of the image synthesis system 106 to the performance of a neural network with a single-stream encoder architecture (Single-Enc).

[0101] like Figure 8 As shown in , the image synthesis system 106 provides improved perceptual quality over the other compared methods, resulting in an average ranking of 2.84. In the twenty image sets, the synthesized digital images generated by the image synthesis system 106 ranked first in eleven images. By comparison, six of the synthesized digital images generated by the second-place method (copy-paste baseline) ranked first.

[0102] Fig. 9 Yet another table reflecting additional experimental results comparing the performance of various embodiments of the image synthesis system 106 is illustrated. In particular, Fig. 9 The results reflect several cleanup studies in which one or more features of the image synthesis system 106 have been removed, thereby comparing the resulting performance to one or more embodiments of the image synthesis system 106 implementing a multi-stage fusion neural network within an end-to-end framework that was trained using the easy-to-hard data augmentation scheme discussed above. In particular, the results reflect the performance of various embodiments: an embodiment of the image synthesis system 106 that does not use the easy-to-hard data augmentation scheme (wo-DataAug) to train the multi-stage fusion neural network, an embodiment in which the multi-stage fusion neural network includes a single-stream encoder architecture (Single-Enc), and an embodiment in which the image synthesis system 106 does not use a mask refinement neural network (wo-RefNet) to refine the segmentation mask. As Fig. 9 As shown in , an embodiment of the image synthesis system 106 that includes the aforementioned features performs significantly better than an embodiment in which at least one of the features is removed, achieving a PSNR score of 26.06 and a SSIM score of 0.9291.

[0103] Now refer to Fig.10 , additional details will now be provided regarding the various components and capabilities of the image composition system 106. In particular, Fig.10The image synthesis system 106 implemented by the server(s) 102 and the image editing system 104 is illustrated. As shown, the image synthesis system 106 may include, but is not limited to, a foreground segmentation neural network application manager 1002, a mask refinement neural network application manager 1004, an inverse segmentation mask generator 1006, a multi-level fusion neural network training engine 1008, a multi-level fusion neural network application manager 1010, and a data storage device 1012 (which includes a training digital image 1014, a foreground segmentation neural network 1016, a mask refinement neural network 1018, and a multi-level fusion neural network 1020).

[0104] As just mentioned and as Fig.10 , the image synthesis system 106 includes a foreground segmentation neural network application manager 1002. In particular, the foreground segmentation neural network application manager 1002 can receive a foreground image and utilize a foreground segmentation neural network to generate a corresponding segmentation mask. In one or more embodiments, when generating the segmentation mask, the foreground segmentation neural network application manager 1002 can divide the foreground image into two separate parts corresponding to the two parts identified by the foreground segmentation neural network from the foreground image - a foreground object and a background.

[0105] like Fig.10 As shown in , the image synthesis system 106 may also include a mask refinement neural network application manager 1004. In particular, the mask refinement neural network application manager 1004 may utilize a mask refinement neural network to modify a previously generated segmentation mask. For example, in one or more embodiments, the mask refinement neural network application manager 1004 receives a segmentation mask generated by the foreground segmentation neural network application manager 1002 (i.e., an original segmentation mask), and uses a mask refinement neural network to generate a segmentation mask to modify the segmentation mask (e.g., by refining the boundary between the foreground object and the background).

[0106] Additionally, if Fig.10 As shown in , the image synthesis system 106 includes a reverse segmentation mask generator 1006. In particular, the reverse segmentation mask generator 1006 can generate a reverse segmentation mask corresponding to a previously generated segmentation mask. For example, in one or more embodiments, the reverse segmentation mask generator 1006 generates a reverse segmentation mask based on a segmentation mask generated by the foreground segmentation neural network application manager 1002. In some embodiments, the reverse segmentation mask generator 1006 generates a reverse segmentation mask based on a segmentation mask generated by the mask refinement neural network application manager 1004.

[0107] In addition, if Fig.10As shown in , the image synthesis system 106 includes a multi-level fusion neural network training engine 1008. In particular, the multi-level fusion neural network training engine 1008 can train the multi-level fusion neural network to generate a synthetic digital image. In one or more embodiments, the multi-level fusion neural network training engine 1008 uses a data enhancement scheme from easy to difficult to train the multi-level fusion neural network. In some embodiments, the multi-level fusion neural network training engine 1008 further uses a self-teaching method to train the multi-level fusion neural network, thereby using the multi-level fusion neural network to generate additional training data for further training.

[0108] like Fig.10 As shown in , the image synthesis system 106 also includes a multi-level fusion neural network application manager 1010. In particular, the multi-level fusion neural network application manager 1010 can utilize the multi-level fusion neural network trained by the multi-level fusion neural network training engine 1008. For example, the multi-level fusion neural network application manager 1010 can identify a foreground image, a background image, and a segmentation mask corresponding to the foreground image, and then use the trained multi-level fusion neural network to generate a synthesized digital image that depicts a foreground object from the foreground image for a scene from the background image.

[0109] In addition, if Fig.10 As shown in , the image synthesis system 106 includes a data storage device 1012. In particular, the data storage device 1012 may include a training digital image 1014, a foreground segmentation neural network 1016, a mask refinement neural network 1018, and a multi-level fusion neural network 1020. The training digital image 1014 may include a digital image for training the multi-level fusion neural network (e.g., an easy training foreground image and a difficult training foreground image). The training digital image 1014 may also include a target digital image for training. When training the multi-level fusion neural network, the multi-level fusion neural network training engine 1008 may obtain the training digital image from the training digital image 1014. The foreground segmentation neural network 1016 may store the foreground segmentation neural network used by the foreground segmentation neural network application manager 1002 to generate a segmentation mask. Similarly, the mask refinement neural network 1018 may store the mask refinement neural network used by the mask refinement neural network application manager 1004 to modify the segmentation mask. Additionally, the multi-level fusion neural network 1020 may store a multi-level fusion neural network that is trained by the multi-level fusion neural network training engine 1008 and used by the multi-level fusion neural network application manager 1010 to generate a synthetic digital image.

[0110] Each of the components 1002 to 1020 of the image synthesis system 106 may include software, hardware, or both. For example, the components 1002 to 1020 may include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices (such as a client device or a server device). The computer-executable instructions of the image synthesis system 106, when executed by one or more processors, may cause the (multiple) computing devices to perform the methods described herein. Alternatively, the components 1002 to 1020 may include hardware (such as a dedicated processing device) to perform a specific function or group of functions. Alternatively, the components 1002 to 1020 of the image synthesis system 106 may include a combination of computer-executable instructions and hardware.

[0111] In addition, the components 1002 to 1020 of the image synthesis system 106 can be implemented, for example, as one or more operating systems, one or more independent applications, one or more modules of an application, one or more plug-ins, one or more library functions or functions that can be called by other applications, and / or a cloud computing model. Therefore, the components 1002 to 1020 of the image synthesis system 106 can be implemented as independent applications, such as desktop or mobile applications. In addition, the components 1002 to 1020 of the image synthesis system 106 can be implemented as one or more web-based applications hosted on a remote server. Alternatively or additionally, the components 1002 to 1020 of the image synthesis system 106 can be implemented in a series of mobile device applications or "apps". For example, in one or more embodiments, the image synthesis system 106 may include a digital software application (such as CREATIVE or ) or in combination with digital software applications such as CREATIVE or ) operation. “ADOBE”, “CREATIVE CLOUD” and “PHOTOSHOP” are registered trademarks or trademarks of Adobe Systems Incorporated in the United States and / or other countries.

[0112] Figures 1 to 10 , corresponding text, and examples provide a number of different methods, systems, devices, and non-transitory computer-readable media for the image synthesis system 106. In addition to the foregoing, Fig.11 As shown in FIG. 1 , one or more embodiments may also be described in terms of flow diagrams that include acts for achieving certain results. Fig.11 Can be performed with more or fewer actions. In addition, actions can be performed in different orders. Additionally, the actions described herein can be repeated or performed in parallel with each other or with different instances of the same or similar actions.

[0113] As mentioned, Fig.11 1100 for generating a composite digital image according to one or more embodiments. Fig.11 The actions according to one embodiment are illustrated, but alternative embodiments may omit, add, reorder and / or modify Fig.11 Any action shown in . Fig.11 The actions of can be performed as part of a method. For example, in some embodiments, Fig.11 The actions may be performed as part of a computer-implemented method in a digital media environment for editing digital images. Alternatively, a non-transitory computer-readable medium may store instructions that, when executed by at least one processor, cause a computing device to perform Fig.11 In some embodiments, the system may perform Fig.11 For example, in one or more embodiments, the system includes at least one memory device including a multi-level fusion neural network trained to generate a synthetic digital image, the multi-level fusion neural network including a foreground encoder, a background encoder, and a decoder. The system may also include causing the system to execute Fig.11 at least one server device of the action.

[0114] The series of actions 1100 includes an action 1102 of identifying an image and a segmentation mask. For example, action 1102 includes identifying a foreground image, a background image, and a segmentation mask corresponding to the foreground image. In one or more embodiments, identifying the segmentation mask corresponding to the foreground image includes: generating the segmentation mask based on the foreground image using a foreground segmentation neural network. In fact, in one or more embodiments, the image synthesis system 106 identifies the foreground image and the background image, and generates the segmentation mask based on the foreground image using a foreground segmentation neural network. In one or more embodiments, the image synthesis system 106 further uses a mask refinement neural network to modify the segmentation mask corresponding to the foreground image. For example, the image synthesis system 106 can modify the boundaries of the foreground object depicted in the segmentation mask based on the foreground image and the segmentation mask using the mask refinement neural network.

[0115] The series of actions 1100 also includes an action of generating a foreground feature map 1104. For example, action 1104 includes generating a synthetic digital image based on the foreground image, the background image, and the segmentation mask by the following operations: using a foreground encoder of a multi-stage fusion neural network to generate a foreground feature map based on the foreground image and the segmentation mask.

[0116] The series of actions 1100 also includes an action 1106 of generating a background feature map. For example, action 1106 includes generating a synthetic digital image based on a foreground image, a background image, and a segmentation mask by: using a background encoder of a multi-level fusion neural network to further generate a background feature map based on the background image and the segmentation mask. In one or more embodiments, the image synthesis system 106 additionally generates a reverse segmentation mask based on the segmentation mask corresponding to the foreground image. Therefore, the image synthesis system 106 can generate a background feature map based on the background image and the segmentation mask by: generating a background feature map based on the background image and the reverse segmentation mask.

[0117] Additionally, the series of actions 1100 includes an action 1108 of generating a synthesized digital image. For example, action 1108 includes generating a synthesized digital image based on the foreground image, the background image, and the segmentation mask by: using a decoder of a multi-stage fusion neural network to further generate a synthesized digital image based on the foreground feature map and the background feature map. For example, the image synthesis system 106 can combine the foreground feature map and the background feature map to generate a combined feature map, and then use the decoder of the multi-stage fusion neural network to generate a synthesized digital image based on the combined feature map. In one or more embodiments, the synthesized digital image includes a foreground object from the foreground image depicted for a scene from the background image.

[0118] In one or more embodiments, the image synthesis system 106 further utilizes skip links to provide layer-specific feature maps to the decoder of the multi-level fusion neural network layer. For example, the image synthesis system 106 may identify a first layer-specific feature map generated by a foreground encoder and a second layer-specific feature map generated by a background encoder. The image synthesis system 106 may then provide the first layer-specific feature map and the second layer-specific feature map to the layers of the decoder of the multi-level fusion neural network via the skip links. Therefore, generating a synthesized digital image based on the foreground feature map and the background feature map using the decoder of the multi-level fusion neural network may include: using the decoder to further generate the synthesized digital image based on the first layer-specific feature map and the second layer-specific feature map. In one or more embodiments, the layer of the foreground encoder corresponding to the first layer-specific feature map is at the same encoder level as the layer of the background encoder corresponding to the second layer-specific feature map.

[0119] In one or more embodiments, the foreground image includes a training foreground image, and the background image includes a training background image. Therefore, the series of actions 1100 may also include actions for training a multi-level fusion neural network. In practice, the image synthesis system 106 may train the multi-level fusion neural network to generate a synthetic digital image by: comparing the synthetic digital image with the target synthetic digital image to determine the loss; and modifying the parameters of the multi-level fusion neural network based on the determined loss. In one or more such embodiments, the image synthesis system 106 may identify the foreground image by: generating the foreground image using the multi-level fusion neural network. In addition, in one or more such embodiments, the image synthesis system 106 may generate the target synthetic digital image using the multi-level fusion neural network.

[0120] As mentioned, in one or more embodiments, the series of actions 1100 also includes an action for training a multi-level fusion neural network to generate a synthetic digital image. For example, in one or more embodiments, the image synthesis system 106 trains the multi-level fusion neural network to generate a synthetic digital image by: generating a target synthetic digital image based on an easy training foreground image and a first training background image using the multi-level fusion neural network; generating a difficult training foreground image based on the easy training foreground image and the second training background image using the multi-level fusion neural network; generating a predicted synthetic digital image based on the difficult training foreground image and the first training background image using the multi-level fusion neural network; and modifying parameters of the multi-level fusion neural network based on a comparison of the predicted synthetic digital image with the target synthetic digital image. In one or more embodiments, the easy training foreground image includes a foreground object depicted against a solid color background. In some embodiments, the difficult training foreground image includes a foreground object depicted against a background from a second training background image, wherein the background varies in detail.

[0121] In one or more embodiments, the action for training the multi-level fusion neural network to generate a synthesized digital image further includes: comparing the predicted synthesized digital image with the target synthesized digital image to determine an L1 loss and a perceptual loss; and determining a combined loss based on the L1 loss and the perceptual loss. Therefore, the image synthesis system 106 can modify the parameters of the multi-level fusion neural network based on the comparison of the predicted synthesized digital image with the target synthesized digital image by the following operations: modifying the parameters based on the combined loss. In one or more embodiments, determining the combined loss includes: applying a weight to the perceptual loss to generate a weighted perceptual loss; and combining the L1 loss with the weighted perceptual loss to generate a combined loss.

[0122] Embodiments of the present disclosure may include or utilize a special-purpose computer or a general-purpose computer that includes computer hardware, such as, for example, one or more processors and system memory, as discussed in more detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more of the processes described herein may be implemented at least in part as instructions implemented in a non-transient computer-readable medium and executable by one or more computing devices (e.g., any media content access device described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transient computer-readable medium (e.g., a memory, etc.) and executes those instructions, thereby executing one or more processes, including one or more of the processes described herein.

[0123] Computer-readable media can be any available media that can be accessed by a general-purpose computer system or a special-purpose computer system. A computer-readable medium that stores computer-executable instructions is a non-transient computer-readable storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present disclosure may include at least two distinct types of computer-readable media: a non-transient computer-readable storage medium (device) and a transmission medium.

[0124] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives (“SSD”) (e.g., RAM-based), flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired program code devices in the form of computer-executable instructions or data structures and that can be accessed by a general purpose or special purpose computer.

[0125] "Network" is defined as one or more data links that support the transport of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or another communication connection (hardwired, wireless, or a combination of hardwired or wireless), the computer appropriately regards the connection as a transmission medium. Transmission media may include networks and / or data links, which may be used to carry desired program code devices in the form of computer executable instructions or data structures and may be accessed by general-purpose computers or special-purpose computers. The above combinations should also be included within the scope of computer-readable media.

[0126] In addition, upon reaching various computer system components, program code means in the form of computer executable instructions or data structures may be automatically transferred from a transmission medium to a non-transient computer readable storage medium (device) (or vice versa). For example, a computer executable instruction or data structure received over a network or data link may be buffered in RAM within a network interface module (e.g., "NIC") and then ultimately transferred to the computer system RAM and / or to a less volatile computer storage medium (device) at the computer system. Thus, it should be understood that a non-transient computer readable storage medium (device) may be included in a computer system component that also (or even primarily) utilizes a transmission medium.

[0127] Computer executable instructions include, for example, instructions and data, which, when executed by a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or function group. In some embodiments, computer executable instructions are executed on a general-purpose computer to turn a general-purpose computer into a special-purpose computer that implements an element of the present disclosure. Computer executable instructions can be, for example, binary files, intermediate format instructions (such as assembly language), or even source code. Although the subject matter has been described in a language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or actions described above. On the contrary, the described features and actions are disclosed as an example form of implementing the claims.

[0128] Those skilled in the art will appreciate that the present disclosure can be practiced in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. The present disclosure can also be practiced in a distributed system environment, in which local computer systems and remote computer systems linked by a network (linked by a hardwired data link, a wireless data link, or a combination of a hardwired data link and a wireless data link) both perform tasks. In a distributed system environment, program modules can be located in local and remote memory storage devices.

[0129] Embodiments of the present disclosure may also be implemented in a cloud computing environment. In this description, "cloud computing" is defined as a model for supporting on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be adopted in the market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources may be quickly configured via virtualization and released with less management effort or service provider interaction, and then scaled accordingly.

[0130] The cloud computing model may consist of various characteristics, such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured services, etc. The cloud computing model may also reveal various service models, such as, for example, software as a service (“SaaS”), platform as a service (“PaaS”), and infrastructure as a service (“IaaS”). The cloud computing model may also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc. In this description and in the claims, a “cloud computing environment” is an environment that employs cloud computing.

[0131] Fig.12 A block diagram of an example computing device 1200 that can be configured to perform one or more of the above-described processes is illustrated. It should be appreciated that one or more computing devices (such as computing device 1200) can represent the computing devices described above (e.g., server(s) 102 and client devices 110a to 110n). In one or more embodiments, computing device 1200 can be a mobile device (e.g., a mobile phone, a smart phone, a PDA, a tablet computer, a laptop, a camera, a tracker, a watch, a wearable device, etc.). In some embodiments, computing device 1200 can be a non-mobile device (e.g., a desktop computer or another type of client device). Additionally, computing device 1200 can be a server device that includes cloud-based processing and storage capabilities.

[0132] like Fig.12 As shown in , computing device 1200 may include one or more processors 1202, memory 1204, storage device 1206, input / output interface 1208 (or "I / O interface 1208"), and communication interface 1210 that may be communicatively coupled via a communication infrastructure (e.g., bus 1212). Fig.12 The computing device 1200 is shown in FIG. Fig.12 The components illustrated in FIG. 1 are not intended to be limiting. Additional or alternative components may be used in other embodiments. Furthermore, in certain embodiments, computing device 1200 includes more than Fig.12 The components shown in the figure are fewer than those shown in the figure. Fig.12 Components of computing device 1200 shown in FIG. 1 are described.

[0133] In certain embodiments, processor(s) 1202 include hardware for executing instructions, such as hardware constituting a computer program. By way of example and not limitation, to execute instructions, processor(s) 1202 may receive (or fetch) instructions from internal registers, internal caches, memory 1204, or storage device 1206 and decode and execute the instructions.

[0134] The computing device 1200 includes a memory 1204 coupled to the processor(s) 1202. The memory 1204 may be used to store data, metadata, and programs for execution by the processor(s). The memory 1204 may include one or more of volatile memory and non-volatile memory, such as a random access memory ("RAM"), a read-only memory ("ROM"), a solid state disk ("SSD"), flash memory, phase change memory ("PCM"), or other types of data storage devices. The memory 1204 may be an internal memory or a distributed memory.

[0135] The computing device 1200 includes a storage device 1206, which includes a storage device for storing data and instructions. By way of example and not limitation, the storage device 1206 may include the non-transient storage medium described above. The storage device 1206 may include a hard disk drive (HDD), a flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices.

[0136] As shown, computing device 1200 includes one or more I / O interfaces 1208 that are provided to allow a user to provide input (such as user strokes) to computing device 1200, receive output from computing device 1200, and otherwise transfer data to and from computing device 1200. These I / O interfaces 1208 may include a mouse, keypad or keyboard, touch screen, camera, optical scanner, network interface, modem, other known I / O devices, or combinations of such I / O interfaces 1208. The touch screen may be activated with a stylus or a finger.

[0137] The I / O interface 1208 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In a particular embodiment, the I / O interface 1208 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or may serve as any other graphical content for a particular implementation.

[0138] The computing device 1200 may also include a communication interface 1210. The communication interface 1210 may include hardware, software, or both. The communication interface 1210 provides one or more interfaces for communicating between the computing device and one or more other computing devices or one or more networks (such as, for example, packet-based communications). As an example and not limitation, the communication interface 1210 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired-based network; or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network (such as WI-FI). The computing device 1200 may also include a bus 1212. The bus 1212 may include hardware, software, or both that connects the components of the computing device 1200 to each other.

[0139] In the foregoing description, the present invention has been described with reference to specific example embodiments of the present invention. Various embodiments and aspects of the present invention are described with reference to the details discussed herein, and the accompanying drawings illustrate various embodiments. The above description and the accompanying drawings are illustrative of the present invention and should not be construed as limiting the present invention. Several specific details are described to provide a thorough understanding of the various embodiments of the present invention.

[0140] Without departing from the spirit and basic characteristics of the present invention, the present invention may be embodied in other specific forms. The described embodiments are considered to be illustrative and non-restrictive in all aspects. For example, the method described herein can be performed with fewer or more steps / actions, or the steps / actions can be performed in different orders. Additionally, the steps / actions described herein can be repeated or performed in parallel with each other or with different instances of the same or similar steps / actions. Therefore, the scope of the present invention is indicated by the appended claims, rather than by the foregoing description. All changes within the meaning and scope of the equivalents of the claims are included within their scope.

Claims

1. A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause a computing device to: identifying a foreground image depicting at least one foreground object for a first background and a segmentation mask corresponding to the foreground image; identifying a background image depicting a second background; as well as Generating a composite digital image depicting the at least one foreground object against the second background based on the foreground image, the background image, and the segmentation mask by: Using a foreground encoder of a multi-stage fusion neural network, generating a foreground feature map based on the foreground image and the segmentation mask; Using the background encoder of the multi-stage fusion neural network, a background feature map is generated based on the background image and the segmentation mask; identifying a first layer-specific feature map generated by the foreground encoder and a second layer-specific feature map generated by the background encoder; providing the first layer-specific feature map and the second layer-specific feature map to a layer of a decoder of the multi-stage fusion neural network via a skip link; and The synthesized digital image is generated using the decoder based on the first layer-specific feature map and the second layer-specific feature map, wherein a layer of the foreground encoder corresponding to the first layer-specific feature map is at the same encoder level as a layer of the background encoder corresponding to the second layer-specific feature map.

2. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to: generate a reverse segmentation mask based on the segmentation mask corresponding to the foreground image, The step of generating the background feature map based on the background image and the segmentation mask comprises: The background feature map is generated based on the background image and the inverse segmentation mask.

3. The non-transitory computer-readable medium of claim 1 , wherein the instructions, when executed by the at least one processor, cause the computing device to identify the segmentation mask corresponding to the foreground image by generating the segmentation mask based on the foreground image using a foreground segmentation neural network.

4. The non-transitory computer-readable medium of claim 3, further comprising instructions that, when executed by the at least one processor, cause the computing device to: utilize a mask refinement neural network to modify the segmentation mask corresponding to the foreground image.

5. The non-transitory computer readable medium of claim 1, wherein the foreground image comprises a training foreground image, and the background image comprises a training background image; and The non-transitory computer-readable medium further includes instructions that, when executed by the at least one processor, cause the computing device to: train the multi-level fusion neural network to generate a synthetic digital image by: comparing the synthesized digital image to a target synthesized digital image to determine loss; and Parameters of the multi-stage fusion neural network are modified based on the determined loss.

6. The non-transitory computer-readable medium of claim 5, wherein the instructions, when executed by the at least one processor, cause the computing device to identify the foreground image by generating the foreground image using the multi-stage fusion neural network.

7. The non-transitory computer-readable medium of claim 5, further comprising instructions that, when executed by the at least one processor, cause the computing device to utilize the multi-stage fusion neural network to generate the target composite digital image.

8. The non-transitory computer-readable medium of claim 1, wherein the instructions, when executed by the at least one processor, cause the computing device to: identifying the background image depicting the second background by identifying the background image depicting one or more additional foreground objects for the second background; and The composite digital image depicting the at least one foreground object for the second background is generated by generating a digital image depicting the at least one foreground object and the one or more additional foreground objects for the second background.

9. A computing system comprising: at least one memory device comprising a multi-stage fusion neural network trained to generate a synthetic digital image, the multi-stage fusion neural network comprising a foreground encoder, a background encoder, and a decoder; at least one server device, the server device enabling the system to: identifying a foreground image depicting at least one foreground object for a first background; identifying a background image depicting a second background; generating a segmentation mask based on the foreground image using a foreground segmentation neural network; Using the foreground encoder of the multi-stage fusion neural network, generating a foreground feature map based on the foreground image and the segmentation mask; Using the background encoder of the multi-stage fusion neural network, generating a background feature map based on the background image and the segmentation mask; identifying a first layer-specific feature map generated by the foreground encoder and a second layer-specific feature map generated by the background encoder; as well as providing the first layer-specific feature map and the second layer-specific feature map to a layer of a decoder of the multi-stage fusion neural network via a skip link; as well as A synthetic digital image depicting the at least one foreground object against the second background is generated using the decoder based on the first layer-specific feature map and the second layer-specific feature map, wherein a layer of the foreground encoder corresponding to the first layer-specific feature map is at the same encoder level as a layer of the background encoder corresponding to the second layer-specific feature map.

10. The system of claim 9, wherein the at least one server device causes the system to train the multi-level fusion neural network to generate a synthetic digital image by: Using the multi-level fusion neural network, a target synthetic digital image is generated based on the easy training foreground image and the first training background image; Using the multi-level fusion neural network, generating a difficult training foreground image based on the easy training foreground image and the second training background image; generating a predicted synthetic digital image based on the difficult training foreground image and the first training background image using the multi-level fusion neural network; as well as Parameters of the multi-stage fusion neural network are modified based on a comparison of the predicted synthetic digital image with the target synthetic digital image.

11. The system of claim 10, wherein: The easy training foreground image includes a foreground object depicted against a solid color background; and The difficult training foreground image includes the foreground object depicted against a background from the second training background image, wherein the background varies in detail.

12. The system of claim 10, wherein the at least one server device enables the system to: comparing the predicted synthetic digital image to the target synthetic digital image to determine an L1 loss and a perceptual loss; and determining a combined loss based on the L1 loss and the perceptual loss, Wherein modifying the parameters of the multi-level fusion neural network based on the comparison between the predicted synthetic digital image and the target synthetic digital image comprises: The parameters are modified based on the combined loss.

13. The system of claim 12, wherein the at least one server device causes the system to determine the combined loss by: applying a weight to the perceptual loss to generate a weighted perceptual loss; and The L1 loss is combined with the weighted perceptual loss to generate the combined loss.

14. The system of claim 9, wherein the at least one server device enables the system to: generating a reverse segmentation mask based on the segmentation mask corresponding to the foreground image, and The background feature map is generated based on the background image and the segmentation mask by the following operation: the background feature map is generated based on the background image and the inverse segmentation mask.

15. The system of claim 9, wherein the at least one server device causes the system to: utilize a mask refinement neural network to modify a boundary of the at least one foreground object depicted in the segmentation mask based on the foreground image and the segmentation mask.

16. A computer-implemented method in a digital media environment for editing digital images, comprising: identifying a foreground image depicting at least one foreground object for a first background and a segmentation mask corresponding to the foreground image; identifying a background image depicting a second background; as well as Generating a composite digital image depicting the at least one foreground object against the second background based on the foreground image, the background image, and the segmentation mask by: Using a foreground encoder of a multi-stage fusion neural network, generating a foreground feature map based on the foreground image and the segmentation mask; Using the background encoder of the multi-stage fusion neural network, a background feature map is generated based on the background image and the segmentation mask; identifying a first layer-specific feature map generated by the foreground encoder and a second layer-specific feature map generated by the background encoder; as well as providing the first layer-specific feature map and the second layer-specific feature map to a layer of a decoder of the multi-stage fusion neural network via a skip link; and The synthesized digital image is generated using the decoder based on the first layer-specific feature map and the second layer-specific feature map, wherein a layer of the foreground encoder corresponding to the first layer-specific feature map is at the same encoder level as a layer of the background encoder corresponding to the second layer-specific feature map.

17. The computer-implemented method of claim 16, wherein identifying the segmentation mask corresponding to the foreground image comprises: The segmentation mask is generated based on the foreground image using a foreground segmentation neural network.

18. The computer-implemented method of claim 17, further comprising: The segmentation mask corresponding to the foreground image is modified using a mask refinement neural network.

Images