Object class inpainting in digital images using class-specific inpainting neural networks

Abstract

This disclosure relates to object class inpainting in digital images using class-specific inpainting neural networks. The disclosure relates to systems, methods, and non-transitory computer-readable media for generating inpainted digital images using class-specific cascaded modulation inpainting neural networks. For example, the disclosed system utilizes a class-specific cascaded modulation inpainting neural network including a cascaded modulation decoder layer to generate replacement pixels depicting a specific target object class. For instance, in response to a user selecting a replacement region and a target object class, the disclosed system utilizes a class-specific cascaded modulation inpainting neural network corresponding to the target object class to generate an inpainted digital image depicting instances of the target object class within the replacement region. Furthermore, in one or more embodiments, the disclosed system trains class-specific cascaded modulation inpainting neural networks corresponding to various target object classes (such as sky object classes, water object classes, ground object classes, or human object classes).

Description

Background Technology

[0001] In the field of digital image editing, computer-implemented models are becoming increasingly effective at generating realistic images from randomly sampled seeds or incomplete masked digital images. In fact, Generative Adversarial Networks (“GANs”), or patch-matching models, have revolutionized the digital image synthesis process, enabling the realistic rendering of complex scenes and the restoration of digital images with missing or defective pixels. However, despite the advancements in conventional digital image systems utilizing these models, conventional systems still suffer from numerous shortcomings, particularly in terms of the accuracy, efficiency, and flexibility of the computing devices used to implement them. Summary of the Invention

[0002] This disclosure describes one or more embodiments of systems, methods, and non-transitory computer-readable media that address one or more of the aforementioned or other problems in the art by utilizing class-specific inpainting neural networks to inpaint digital images to depict specific object classes. Specifically, in one or more implementations, the disclosed system utilizes a neural network design comprising an encoder that extracts multi-scale feature representations from an input image with apertures and a decoder with cascaded modulation layers at each resolution level. For example, in one or more embodiments, at each cascaded modulation layer of the decoder, the disclosed system applies global modulation to perform coarse semantic-aware structure synthesis, and then utilizes spatial modulation to adjust the feature maps in a spatially adaptive manner. The disclosed system trains one or more inpainting neural networks by masking specific object classes (such as sky regions) from training digital images. Thus, the cascaded modulation inpainting neural network learns to inpaint masked regions of the digital image using pixels depicting the specific object class. The disclosed system can utilize such class-specific inpainting neural networks in various user interface applications to efficiently, flexibly, and accurately regenerate or synthesize target object classes in digital images. Attached Figure Description

[0003] This disclosure describes one or more embodiments of the present disclosure with additional specificity and detail by referring to the accompanying drawings. The following paragraphs briefly describe these drawings, in which:

[0004] Figure 1 An example system environment in which a class-specific image restoration system according to one or more embodiments operates is shown;

[0005] Figure 2 An overview of generating repaired digital images using a class-specific cascaded modulation repair neural network according to one or more embodiments is shown;

[0006] Figure 3 An example architecture of a cascaded modulation repair neural network according to one or more embodiments is shown;

[0007] Figure 4 An example architecture of a cascaded modulation layer for a decoder of a cascaded modulation repair neural network according to one or more embodiments is shown;

[0008] Figure 5 An example architecture utilizing positional encoding in a cascaded modulation repair neural network is shown according to one or more embodiments;

[0009] Figure 6 An overview of training a class-specific cascaded modulation repair neural network according to one or more embodiments is shown;

[0010] Figure 7 A flowchart illustrating the training of a class-specific cascaded modulation repair neural network according to one or more embodiments is shown;

[0011] Figures 8A to 8C A user interface for generating restored digital images according to one or more embodiments is shown;

[0012] Figures 9A to 9C An additional user interface for generating restored digital images, according to one or more embodiments, is shown;

[0013] Figure 10 Example results are shown of generating inscribed digital images using a class-specific cascaded modulation insulator neural network trained to generate sky regions for digital images, according to one or more embodiments.

[0014] Figure 11 A table showing experimental results according to one or more embodiments is provided;

[0015] Figure 12 Additional experimental results tables according to one or more embodiments are shown;

[0016] Figure 13 A schematic diagram of a class-specific image restoration system according to one or more embodiments is shown;

[0017] Figure 14 A flowchart illustrating a series of actions for generating a repaired digital image using a class-specific cascaded modulation repair neural network, according to one or more embodiments, is shown; and

[0018] Figure 15 A block diagram of an example computing device according to one or more embodiments is shown. Detailed Implementation

[0019] This disclosure describes one or more embodiments of a class-specific image inpainting system that generates inpainted digital images using a class-specific inpainting neural network. In one or more embodiments, the class-specific image inpainting system utilizes a cascaded modulation-decoder layer that decomposes inference into multiple stages (e.g., global prediction and local thinning). For example, in each decoder layer, the class-specific image inpainting system begins with global code modulation that captures the global-scale image structure, followed by spatial adaptive modulation that thins the global prediction. Furthermore, the class-specific image inpainting system employs a unique method to train the class-specific inpainting neural network. Specifically, the class-specific image inpainting system generates class-specific digital images using a panoptic segmentation algorithm, and then trains the class-specific inpainting neural network using annotated class-specific regions as mask regions. By training using an adversarial loss under masking conditions, the class-specific image inpainting system learns the parameters of the class-specific inpainting neural network, which is capable of accurately, efficiently, and flexibly generating inpainted digital images depicting a specific target object class.

[0020] As previously described, in one or more implementations, class-specific image inpainting systems utilize cascaded modulation inpainting neural networks. For example, a class-specific image inpainting system uses multiple convolutional neural network encoder layers to process digital images at different scales / resolutions to generate coded feature vectors. Furthermore, in one or more implementations, the class-specific image inpainting system uses these coded feature vectors to generate image codes (e.g., global feature codes or other feature vectors) representing global features of the digital image. As mentioned above, in one or more implementations, the class-specific image inpainting system utilizes encoder layers including Fourier convolutional blocks to expand the encoder's receptive field.

[0021] Furthermore, class-specific image inpainting systems utilize unique cascaded modulator-decoder architectures to generate inpainted digital images. For example, each cascaded modulation layer includes a global modulation block and additional modulation blocks (such as a spatial modulation block or another global modulation block). In one or more embodiments, these modulation blocks implement different modulation operations to generate different feature map representations. Thus, for example, a global modulation block applies modulation based on global feature codes to an input global feature map to generate a new global feature map. Similarly, a spatial modulation block can apply spatial modulation (e.g., based on a spatial tensor and global feature codes) to an input local feature map to generate a new local feature map.

[0022] In some embodiments, the class-specific image inpainting system 102 utilizes different architectures for class-specific inpainting neural networks. For example, in one or more implementations, the class-specific image inpainting system 102 utilizes an inpainting neural network that includes encoder and decoder layers, but without a cascaded modulation-decoder layer. Therefore, the class-specific image inpainting system 102 can utilize a variety of class-specific inpainting neural networks.

[0023] As described above, the class-specific image inpainting system also learns parameters for a class-specific inpainting neural network. For example, the system utilizes a panoptic segmentation model to process a repository of digital images to segment objects corresponding to specific classes depicted in the digital images. The system filters digital images that depict the target object class and trains the class-specific inpainting neural network using corresponding masks. Specifically, the system uses the class-specific inpainting neural network to generate inpainted digital images from the class-segmented digital images. Then, the system utilizes a discriminator network to generate a realism prediction for the inpainted digital images. The system determines an adversarial loss from the realism prediction and uses this adversarial loss to modify the parameters of the class-specific inpainting neural network.

[0024] In one or more embodiments, the class-specific image inpainting system uses positional encoding when training and implementing the class-specific inpainting neural network. Specifically, the class-specific image inpainting system determines positional encodings of Fourier features that reflect different layers of the class-specific inpainting neural network. The class-specific image inpainting system injects these positional encodings into the input of the class-specific inpainting neural network and into each layer of the network (i.e., the encoder layer and the decoder layer) to enhance the model's structural predictive capabilities.

[0025] Once trained, the class-specific image inpainting system also utilizes a class-specific inpainting neural network to generate inpainted digital images. Specifically, the class-specific image inpainting system utilizes various user interfaces and corresponding workflows to leverage the class-specific inpainting neural network to generate inpainted digital images depicting the target object class. For example, the class-specific image inpainting system provides an initial digital image to the user interface. In response to user interaction with the digital image (e.g., drawing a new sky region or segmenting selections to replace an existing sky region), the class-specific image inpainting system utilizes the class-specific inpainting neural network to generate replacement pixels in instances of the target object class. Thus, the class-specific image inpainting system 102 can generate replacement regions reflecting target objects that are not present in the input (e.g., masked from the input image).

[0026] As mentioned above, conventional systems have many shortcomings or limitations, particularly in achieving accuracy, flexibility, and efficiency in computing devices. For example, when dealing with large holes in complex images, conventional systems often struggle to generate plausible image structures. For instance, conventional systems typically generate incomplete digital images with inauthentic content and visual artifacts. For example, while patch matching methods are generally effective for generating static textures or completing simple shapes, they fail to produce novel textures or image structures. Similarly, deep learning methods often struggle to generate content consistent with existing digital content both inside and outside the hole. In other words, conventional systems struggle to infer semantic cues from incomplete digital images while simultaneously propagating low-level visual features globally.

[0027] Furthermore, conventional systems struggle to generate replacement pixels for semantic regions that are completely missing in the input. For example, conventional systems are typically designed to borrow pixels from known regions. Therefore, these systems often fail to accurately represent missing (or masked) semantic regions in the input digital image.

[0028] These inaccuracies are often due to the inflexibility of conventional systems. For example, the inaccuracy in repairing large holes is partly due to the lack of flexible network structures that fail to capture long-term dependencies and high-level semantics of the image. Thus, for example, patch matching methods lack mechanisms to model high-level semantics to complete new semantic structures within the holes. Similarly, deep learning methods lack structured methods to acquire semantic information to complete global structures. In fact, a recent deep learning method (Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu, in Large scale image completion via co-modulated generative adversarial networks, arXiv preprint arXiv:2103.10428(2021) (hereinafter referred to as “CoModGAN”)) utilizes a joint modulation mechanism that uses global code modulation to decode encoded image features. However, this approach is limited in recovering spatial or other feature details and uses skip connections to pass invalid contextual features generated by the encoder to the decoder within the holes. Furthermore, many new repair models are inflexible because they are incompatible with state-of-the-art GAN architectures, such as the one described by Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila in “Analyzing and improving the image quality of StyleGAN,” Proc. CVPR (2020) (hereinafter referred to as StyleGan2), which is incorporated herein by reference in its entirety.

[0029] Furthermore, the encoding methods of conventional systems are also inaccurate. In fact, some conventional systems utilize fully convolutional models, which are affected by the slow growth of the effective receptive field in the early stages of the encoder. Therefore, utilizing cross-convolutions within the encoder can generate invalid features in the aperture regions, making feature correction in the decoding stage more challenging. Thus, conventional systems often generate additional inaccuracies when using convolutional encoder architectures.

[0030] Furthermore, conventional systems are often inflexible and unable to generate specific target object classes when generating replacement pixels. In fact, conventional generative models are often unaware of the specific objects generated when restoring digital images. Therefore, these models cannot generate replacement pixels consistent with the target object class required for a specific context. Several conventional systems have been developed for inserting sky regions within digital images. However, conventional systems often fail to generate new, novel skies, instead merely copying or replicating existing sky images. Moreover, conventional systems suffer from compatibility issues with non-sky regions. In particular, the sky regions inserted by conventional systems conflict with the geometry, lighting, and other visual features of the digital image.

[0031] Furthermore, conventional systems are typically inefficient and require substantial computing resources to implement. In fact, attempts to improve the accuracy and flexibility just discussed often exacerbate these inefficiencies. For example, improving the accuracy of deep learning methods typically leads to additional learning parameters and additional computational resources in memory, as well as increased processing power for training and implementing deep learning models.

[0032] In one or more embodiments, class-specific image inpainting systems offer a variety of improvements or advantages compared to conventional systems. For example, by utilizing cascaded modulation inpainting neural networks, one or more embodiments of class-specific image inpainting systems generate more realistic and accurate inpainted digital images. As explained in more detail below (e.g., regarding...), Figure 11 Experimental results show that, compared to conventional systems, example implementations of class-specific image restoration systems significantly improve accuracy.

[0033] Furthermore, class-specific image inpainting systems are capable of accurately generating semantic regions (such as sky regions) that are completely absent (e.g., masked) in the input digital image. In fact, by utilizing class-specific inpainting neural networks, class-specific image inpainting system 102 can completely replace sky regions (or other semantic regions) in a digital image while precisely matching the new regions with the contextual features of the rest of the digital image.

[0034] As described above, in one or more embodiments, class-specific image inpainting systems utilize cascaded modulation-decoder layers. For example, in some implementations, these cascaded modulation-decoder layers include global code modulation (capturing the global-scale image structure) and spatially adaptive modulation (refining global predictions in a spatially varied manner). Therefore, unlike conventional systems, in one or more implementations, class-specific image inpainting systems provide mechanisms to correct distorted local details so that the inpainted pixels are consistent globally and locally with the rest of the image. Furthermore, in some embodiments, class-specific image inpainting systems utilize modulation blocks (e.g., without instance normalization) to make the design compatible with newer GAN architectures such as StyleGAN2.

[0035] Furthermore, in one or more embodiments, the class-specific image inpainting system also utilizes a unique coding architecture to improve accuracy. For example, the class-specific image inpainting system utilizes Fast Fourier Convolutional Blocks within the encoder layer to expand the encoder's receptive field in the early stages, allowing the network encoder to better capture global structure. In fact, class-specific image inpainting system 102 utilizes Fast Fourier Convolutional Blocks at each encoder layer (at different resolutions) to propagate features in the early stages, which avoids generating invalid features within the apertures and improves the results.

[0036] In one or more embodiments, the class-specific image inpainting system further improves accuracy by utilizing positional encoding. Indeed, as described above, the class-specific image inpainting system generates positional encodings that reflect Fourier features for each feature map. The class-specific image inpainting system uses these positional encodings as input to the inpainting neural network and enhances the accuracy of structured predictions at the encoder and decoder layers.

[0037] Furthermore, class-specific image inpainting systems improve the accuracy and flexibility of generating replacement pixels for digital images depicting one or more instances of a specific target object class. In fact, unlike generating generic replacement pixels, class-specific image inpainting systems offer improved accuracy and flexibility in generating replacement pixels depicting the desired object class. Therefore, in one or more implementations, the client device selects the target object class, and the class-specific image inpainting system utilizes a corresponding class-specific inpainting neural network to generate replacement pixels depicting one or more instances of the desired object class. Moreover, by leveraging the class-specific inpainting neural network, the class-specific image inpainting system utilizes a data-driven generative model to generate a variety of novel regions consistent with the surrounding image content.

[0038] Furthermore, class-specific image inpainting systems improve accuracy and flexibility without sacrificing efficiency. In fact, as discussed in more detail below (e.g., regarding...), Figure 12In one or more implementations, class-specific image inpainting systems improve accuracy relative to conventional systems without increasing the number of parameters used to generate the inpainted digital image (in some cases, reducing the number of parameters). Therefore, class-specific image inpainting systems improve accuracy without sacrificing (in some cases, increasing) computer memory and processing power compared to conventional systems.

[0039] More details about class-specific image restoration systems will now be provided with reference to the accompanying drawings. For example, Figure 1 A schematic diagram of an example system environment for implementing a class-specific image restoration system 102 according to one or more embodiments is shown. About Figure 1 An overview of the class-specific image restoration system 102 has been described. A more detailed description of the components and processes of the class-specific image restoration system 102 will then be provided in conjunction with the accompanying drawings.

[0040] As shown in the figure, the environment includes (multiple) servers 104, client devices 108, a database 112, and a network 114. Each component of the environment communicates via network 114, and network 114 is any suitable network through which computing devices communicate. The following section combines... Figure 12 The example network will be discussed in more detail.

[0041] As described above, the environment includes client device 108. Client device 108 is a computing device among various computing devices, including smartphones, tablets, smart TVs, desktop computers, laptops, virtual reality devices, augmented reality devices, or related devices. Figure 12 Another computing device described. Although Figure 1 A single instance of client device 108 is shown, but in some embodiments, the environment includes multiple different client devices, each associated with a different user (e.g., a digital image editor). Client device 108 communicates with server(s) 104 via network 114, providing information to server(s) 104 instructing client device interactions (e.g., digital image selection, user interactions requesting the generation or modification of a digital image, or other inputs), and receiving information from server(s) 104, such as the generated restored digital image. Thus, in some cases, class-specific image restoration system 102 on server(s) 104 provides and receives information based on client device interactions made via client device 108.

[0042] like Figure 1As shown, client device 108 includes client application 110. Specifically, client application 110 is a web application, a native application installed on client device 108 (e.g., a mobile application, desktop application, etc.), or a cloud-based application, with all or part of its functionality performed by servers(s) 104. Based on instructions from client application 110, client device 108 presents or displays information to a user, including digital images, such as restoring digital images, masking digital images, and / or optional options for generating and editing digital images (e.g., indicating objects to be removed and / or restored). In some cases, client application 110 includes all or part of a class-specific image restoration system 102 and / or a class-specific cascaded modulation restoration neural network 116a to 116n (or other class-specific restoration neural networks).

[0043] like Figure 1 As shown, the environment includes multiple servers 104. The multiple servers 104 generate, track, store, process, receive, and transmit electronic data, such as client device interactions and / or indications of digital image pixels. For example, the multiple servers 104 receive data from client device 108 in the form of indications from client device interactions to generate a repaired digital image. In response, the multiple servers 104 transmit data to client device 108 to cause client device 108 to display or present the repaired digital image based on client device interactions.

[0044] In some embodiments, servers 104 communicate with client devices 108 to transmit and / or receive data via network 114, including client device interactions, repaired digital images, and / or other data. In some embodiments, servers 104 include distributed servers, wherein servers 104 include multiple server devices located across network 114 and in different physical locations. Servers 104 include content servers, application servers, communication servers, web hosting servers, multidimensional servers, or machine learning servers. Servers 104 further access and utilize database 112 to store and retrieve information, such as generative repair neural networks (e.g., class-specific cascaded modulation repair neural networks 116a to 116n), stored sample digital images for training, and / or generated repaired digital images.

[0045] like Figure 1As further shown, the servers 104 also include a class-specific image restoration system 102 as part of the digital content editing system 106. For example, in one or more implementations, the digital content editing device 106 is capable of storing, generating, modifying, editing, enhancing, providing, distributing, and / or sharing digital content, such as digital images. For example, the digital content editing system 106 provides client devices 108 with tools for generating and modifying digital images via client application 110.

[0046] In one or more embodiments, the server(s) 104 includes all or part of the class-specific image inpainting system 102. For example, the class-specific image inpainting system 102 operates on the server(s) 104 to train a generative inpainting neural network to generate inpainted digital images. In some cases, the class-specific image inpainting system 102 utilizes a class-specific cascaded modulation inpainting neural network, either locally on the server(s) 104 or from another network location (e.g., database 112), which includes one or more component neural networks, such as encoder neural networks, generator neural networks, and / or discriminator neural networks.

[0047] In some cases, client device 108 includes all or part of class-specific image inpainting system 102. For example, client device 106 generates, acquires (e.g., downloads), or utilizes one or more aspects of class-specific image inpainting system 102 from server(s) 104, such as class-specific cascaded modulation inpainting neural networks 116a to 116n. In practice, in some implementations, such as... Figure 1 As shown, class-specific image restoration system 102 is located wholly or partially on client device 108. For example, class-specific image restoration system 104 includes a web-hosted application that allows client device 108 to interact with server(s) 104. For instance, in one or more implementations, client device 108 accesses web pages supported and / or hosted by server(s) 104.

[0048] In one or more embodiments, client device 108 and servers(s)104 work together to implement class-specific image restoration system 102. For example, in some embodiments, servers(s)104 train one or more neural networks discussed herein and provide one or more neural networks to client device 108 for implementation (e.g., generating a restored digital image at client device 108). In some embodiments, servers(s)104 train one or more neural networks, client device 108 requests restoration of a digital image, servers(s)104 utilize one or more neural networks to generate the restored digital image and provide the restored digital image to client device 108. Furthermore, in some implementations, client device 108 assists in training one or more neural networks.

[0049] although Figure 1 A specific arrangement of the environment is shown, but in some embodiments, the environment has different component arrangements and / or may have different numbers or sets of components. For example, as described above, the class-specific image inpainting system 102 is implemented by a client device 108 (e.g., entirely or partially located thereon). Furthermore, in one or more embodiments, the client device 108 bypasses network 114 and communicates directly with the class-specific image inpainting system 102, and the class-specific cascaded modulation inpainting neural networks 116a to 116n are stored in database 112 and maintained by server(s) 104, client device 108, or a third-party device.

[0050] As previously described, in one or more embodiments, class-specific image inpainting system 102 utilizes a class-specific cascaded modulation inpainting neural network to generate inpainted digital images. For example, Figure 2 A class-specific image restoration system 102 according to one or more embodiments is shown, which generates a restored digital image 208 from a digital image 202 having a replacement region 204 using one or more of class-specific cascaded modulation restoration neural networks 116a to 116n.

[0051] like Figure 2 As shown, the class-specific image restoration system 102 identifies a digital image 202 having a replacement region 204. In one or more embodiments, the class-specific image restoration system 102 identifies the digital image 202 based on one or more user interactions at a client device. For example, the client device may select a digital image (e.g., from a digital image repository stored at the client device or a remote server). Furthermore, the class-specific image restoration system 102 may receive instructions on selecting a region of the digital image to be replaced, restored, or filled.

[0052] For example, replacement region 204 may include a region, portion, mask, or hole within a digital image to be replaced, covered, or filled with replacement pixels. In some embodiments, class-specific image inpainting system 102 identifies replacement region 204 based on a user's selection of pixels to be moved, removed, covered, or replaced from the digital image. For example, a client device may select distracting or unwanted objects or regions in a digital image. Class-specific image inpainting system 102 may remove or delete distracting or unwanted objects or regions and generate replacement pixels. In some cases, class-specific image inpainting system 102 identifies replacement region 204 by generating a digital image mask via a segmentation model (e.g., a segmentation neural network that identifies objects to be moved or removed).

[0053] The class-specific image restoration system 102 can identify the replacement region 204 in various ways. In some embodiments, the class-specific image restoration system 102 applies a segmentation algorithm. For example, the class-specific image restoration system 102 applies foreground, background, or salient object segmentation models. Similarly, in some embodiments, the class-specific image restoration system 102 applies a panoramic segmentation algorithm. In some embodiments, the class-specific image restoration system 102 applies a user-selected segmentation algorithm that segments digital objects from a digital image based on positive, negative, boundary, or region input. In some implementations, the class-specific image restoration system 102 provides various segmented objects for display and receives a user selection of one of the segmented objects as a replacement region 204.

[0054] Furthermore, in one or more implementations, the class-specific image inpainting system 102 also receives an indication of the target object class. For example, the class-specific image inpainting system 102 receives a selection of the replacement region 204 using a tool associated with the target object class (e.g., a sky fill tool or a water fill tool). Thus, in some embodiments, the class-specific image inpainting system 102 receives the selection of the replacement region 204 and the corresponding target object class via the same user interaction (e.g., a sky replacement segmentation tool for selecting an existing sky and indicating that the replacement region should be replaced with a new sky). In some embodiments, the class-specific image inpainting system 102 identifies the replacement region (e.g., using a segmentation algorithm) and receives a separate user interaction identifying the target object class for replacing the replacement region (e.g., selecting a ground target object from a plurality of selectable target object elements).

[0055] In some implementations, the class-specific image inpainting system 102 automatically determines the class of the target object. For example, the class-specific image inpainting system 102 may determine a classification corresponding to pixels in or around the replacement region. In one or more embodiments, the class-specific image inpainting system 102 utilizes classification to intelligently determine the class of the target object. Thus, for example, the class-specific image inpainting system 102 may determine that the replacement region 204 previously depicted sky pixels (or surrounding pixels indicating a sky region). In response, the class-specific image inpainting system 102 may utilize the class-specific cascaded modulation inpainting neural network 116a to generate a new sky.

[0056] As shown in the figure, class-specific image inpainting system 102 utilizes class-specific cascaded modulation inpainting neural network 116a to generate replacement pixels for replacement region 204. The class-specific cascaded modulation inpainting neural network 116a generates replacement pixels depicting instances of a target object class (e.g., depicting a new sky in response to an indication of a sky target object class). In some embodiments, the term neural network refers to a machine learning model trained and / or tuned based on inputs to generate predictions, determine classifications, or approximate unknown functions. For example, a neural network includes interconnected models of artificial neurons (e.g., hierarchically organized) that communicate and learn to approximate complex functions and generate outputs (e.g., generated digital images) based on multiple inputs provided to the neural network. In some cases, neural network refers to an algorithm (or set of algorithms) that implements deep learning techniques to model high-level abstractions in data. For example, neural networks include convolutional neural networks, recurrent neural networks (e.g., LSTM), graph neural networks, generative adversarial neural networks, or other architectures.

[0057] Relatedly, a generative adversarial neural network (or "GAN") comprises a neural network that is modulated or trained through an adversarial process to generate an output digital image (e.g., from an input digital image). In some cases, a GAN comprises multiple constituent neural networks, such as an encoder neural network and one or more decoder / generator neural networks. For example, the encoder neural network extracts latent codes from a noise vector or a digital image. The generator neural network (or a combination of generator neural networks) generates a modified digital image by combining the extracted latent codes (e.g., from the encoder neural network). During training, competing with the generator neural network, a discriminator neural network analyzes the generated digital image to generate a authenticity prediction by determining whether the generated digital image is real (e.g., from a stored set of digital images) or fake (e.g., not from a stored set of digital images). The discriminator neural network also enables the class-specific image inpainting system 102 to modify the parameters of the encoder neural network and / or one or more generator neural networks to ultimately generate a digital image that deceives the discriminator neural network into indicating that the generated digital image is a real digital image.

[0058] Along these lines, generative adversarial neural networks refer to neural networks with a specific architecture or purpose, such as generative inpainting neural networks. For example, generative inpainting neural networks include generative adversarial neural networks that repair or fill pixels in a digital image with replacement pixels. In some cases, generative inpainting neural networks repair digital images by filling in hole regions (represented by a digital image mask) that include pixels identified or otherwise designated as defective, missing, or undesirable. Indeed, as described above, in some embodiments, the digital image mask uses segmentation or masking to define replacement regions that indicate, overlay, cover, or delineate pixels in the digital image to be removed or replaced.

[0059] Therefore, the class-specific cascaded modulation inpainting neural network 116a includes a generative inpainting neural network that utilizes a decoder with one or more cascaded modulation decoder layers (e.g., trained to generate replacement pixels corresponding to the target object class). In fact, as... Figure 2 As shown, the class-specific cascaded modulation inpainting neural network 116a includes multiple cascaded modulation decoder layers 210 to 216. For example, a cascaded modulation decoder layer includes at least two connected (e.g., cascaded) modulation blocks used to modulate the input signal when generating the inpainted digital image. For instance, a cascaded modulation decoder layer may include a first global modulation block and a second global modulation block. Similarly, a cascaded modulation decoder layer may include a first global modulation block (which analyzes global features and utilizes a global spatial invariant method) and a second spatial modulation block (which analyzes local features using a spatial transformation method). Additional details regarding the modulation blocks (e.g., regarding...) will be provided below. Figure 3 , Figure 4 ).

[0060] As shown in the figure, in one or more implementations, the class-specific image inpainting system 102 trains multiple class-specific cascaded modulation inpainting neural networks 116a to 116n. In fact, the class-specific image inpainting system 102 trains different class-specific cascaded modulation inpainting neural networks 116a to 116n to generate inpainted digital images depicting different target object classes. For example, the first class-specific cascaded modulation inpainting neural network 116a corresponds to the sky object class, the second class-specific cascaded modulation inpainting neural network 114b corresponds to the ground object class, and the third class-specific cascaded modulation inpainting neural network 118n corresponds to the human object class. The class-specific image inpainting system 102 can also train class-specific cascaded modulation inpainting neural networks corresponding to different target object classes (e.g., water object class, ice object class, mountain object class, car object class, building / structure object class, road object class, tree object class, dog object class, or cat object class).

[0061] As shown in the figure, the class-specific image inpainting system 102 utilizes a class-specific cascaded modulation inpainting neural network 116a (and cascaded modulation decoder layers 210 to 216) to generate an inpainted digital image 208. Specifically, the class-specific cascaded modulation inpainting neural network 116a generates the inpainted digital image 208 by generating replacement pixels for a replacement region 204 corresponding to the target object class. As shown in the figure, the replacement region 204 is now filled with replacement pixels depicting a realistic instance of the target object class (e.g., sky) instead of the original replacement region 204.

[0062] As described above, in one or more implementations, the class-specific image inpainting system 102 utilizes a unique cascaded modulation inpainting neural network, which includes cascaded modulation decoder layers to generate inpainted digital images. Figure 3 An example architecture of a cascaded modulation repair neural network 302 according to one or more embodiments is shown.

[0063] As shown in the figure, the cascaded modulation inpainting neural network 302 includes an encoder 304 and a decoder 306. Specifically, the encoder 304 includes multiple convolutional layers 308a to 308n at different scales / resolutions. The class-specific image inpainting system 102 feeds a digital image input 310 (e.g., the encoding of the digital image) into the first convolutional layer 308a to generate a higher-scale (e.g., lower-resolution) encoded feature vector. The second convolutional layer 308b processes the encoded feature vector at a higher scale (lower resolution) and generates additional encoded feature vectors (at yet another higher scale / lower resolution). The class-specific image inpainting system 102 iteratively generates these encoded feature vectors until it reaches the final / highest-scale convolutional layer 308n and generates the final encoded feature vector representation of the digital image.

[0064] As shown in the figure, in one or more embodiments, the class-specific image inpainting system 102 generates a global feature code from the final encoded feature vector of the encoder 304. The global feature code includes a feature representation of the digital image from a global (e.g., high-level, high-scale, low-resolution) perspective. In particular, the global feature code may include a representation of the digital image that reflects the encoded feature vector at the highest scale / lowest resolution (or different encoded feature vectors that meet a threshold scale / resolution).

[0065] As shown in the figure, in one or more embodiments, the class-specific image inpainting system 102 applies a neural network layer (e.g., a fully connected layer) to the final encoded feature vector to generate a pattern code 312 (e.g., a pattern vector). Furthermore, the class-specific image inpainting system 102 generates a global feature code by combining the pattern code 312 with a random pattern code 314. Specifically, the class-specific image inpainting system 102 utilizes a neural network layer (e.g., a multilayer perceptron) to process the input noise vector to generate a random pattern code. The neural network layer maps the input noise vector to the random pattern code 314. The class-specific image inpainting system 102 combines the random pattern code 324 with the pattern code 312 (e.g., concatenate, add, or multiply) to generate a global feature code 316. Although... Figure 3 A specific method for generating global feature code 316 is shown, but the class-specific image inpainting system 102 can utilize various different methods to generate global feature codes representing the encoded feature vector of encoder 304 (e.g., without style code 312 and / or random style code 314).

[0066] As described above, the class-specific image restoration system 102 can generate image codes using the encoder 304. Image codes refer to the coded representation of a digital image. Therefore, image codes can include one or more coded feature vectors, style codes, and / or global feature codes.

[0067] In one or more embodiments, the class-specific image inpainting system 102 utilizes multiple Fourier convolutional encoder layers to generate image codes (e.g., encoded feature vectors, pattern codes 312, and / or global feature codes 316). For example, the Fourier convolutional encoder layer (or fast Fourier convolution) includes convolutional layers that include nonlocal receptive fields and cross-scale fusion within convolutional units. Specifically, fast Fourier convolution can include three computations in a single computational unit: a local branch performing small kernel convolution, a semi-global branch processing spectral stacked image patches, and a global branch processing image-level spectra. These three branches complementarily handle different scales. Furthermore, fast Fourier convolution can include a multi-branch aggregation process for cross-scale fusion. For example, in one or more embodiments, class-specific image inpainting system 102 utilizes fast Fourier convolutional layers, as described by Lu Chi, Borui Jiang and Yadong Mu in “Fast fourier convolution, Advances in Neural Information Processing Systems”, 33 (2020), which is incorporated herein by reference in its entirety.

[0068] Specifically, in one or more embodiments, the class-specific image inpainting system 102 utilizes a Fourier convolutional encoder layer for each of the encoder convolutional layers 308a to 308n. Thus, the class-specific image inpainting system 102 utilizes different Fourier convolutional encoder layers with different scales / resolutions to generate encoded feature vectors with improved, nonlocal receptive fields.

[0069] The operation of encoder 304 can also be described using variables or equations to demonstrate the functionality of cascaded modulation inpainting neural network 302. For example, as described above, cascaded modulation inpainting neural network 302 is an encoder-decoder network with a proposed cascaded modulation block for image inpainting in its decoding stage. Specifically, cascaded modulation inpainting neural network 302 starts with encoder E, which takes a portion of the image and a mask as input to produce a multi-scale feature map ranging from the input resolution to a 4×4 resolution.

[0070]

[0071] in The features are generated at scales 1 ≤ i ≤ L (where L is the highest scale or resolution). The encoder is implemented by a set of span-2 convolutions with residual connections.

[0072] In generating the highest scale features Then, the fully connected layer and L2 normalization generate global style codes. The input is represented globally. In parallel with the encoder, an MLP-based mapping network generates a random pattern code w from normalized random Gaussian noise z to simulate the randomness of the generation process. Furthermore, the class-specific image inpainting system 102 combines w with s to generate a final global code g = [s; w] for decoding. As described above, the class-specific image inpainting system 102 can utilize the final global code as image encoding for the digital image.

[0073] As mentioned above, in some implementations, the effective receptive field of a fully convolutional model grows slowly, especially in the early stages of the network. Therefore, utilizing cross-convolutions within the encoder can generate invalid features within the aperture regions, making feature correction in the decoding stage more challenging. Fast Fourier Convolution (FFC) can help early layers achieve a receptive field covering the entire image. However, conventional systems only use FFC at the bottleneck layer, which is computationally demanding. Furthermore, shallow bottleneck layers cannot effectively capture global semantic features. Therefore, in one or more implementations, the class-specific image inpainting system 102 replaces the convolutional blocks in the encoder with FFC targeting the encoder layer. FFC enables the encoder to propagate features in the early stages, thus addressing the problem of generating invalid features within the apertures, which helps improve the results.

[0074] like Figure 3As further shown, the cascaded modulation inpainting neural network 302 also includes a decoder 306. As illustrated, the decoder 306 includes multiple cascaded modulation layers 320a to 320n. The cascaded modulation layers 320a to 320n process input features (e.g., input global feature maps and input local feature maps) to generate new features (e.g., new global feature maps and new local feature maps). Specifically, each of the cascaded modulation layers 320a to 320n operates at a different scale / resolution. Thus, the first cascaded modulation layer 320a acquires input features at a first resolution / scale and generates new features at a lower scale / higher resolution (e.g., via upsampling as part of one or more modulation operations). Similarly, additional cascaded modulation layers operate at lower scales / higher resolutions until the inpainted digital image is generated at the output scale / resolution (e.g., lowest scale / highest resolution).

[0075] Furthermore, each cascaded modulation layer in a cascaded modulation layer comprises multiple modulation blocks. For example, regarding Figure 3 The first cascaded modulation layer 320a includes a global modulation block and a spatial modulation block. Specifically, the class-specific image inpainting system 102 performs global modulation for the input features of the global modulation block. Furthermore, the class-specific image inpainting system 102 performs spatial modulation for the input features of the spatial modulation block. By performing both global and spatial modulation within each cascaded modulation layer, the class-specific image inpainting system 102 refines the global location to generate a more accurate inpainted digital image.

[0076] As shown in the figure, cascaded modulation layers 3320a to 320n are cascaded because the global modulation block is fed into the spatial modulation block. Specifically, the class-specific image inpainting system 102 performs spatial modulation at the spatial modulation block based on features generated at the global modulation block. For example, in one or more embodiments, the class-specific image inpainting system 102 utilizes the global modulation block to generate intermediate features. The class-specific image inpainting system 102 then uses convolutional layers (e.g., a 2-layer convolutional affine parameter network) to convert the intermediate features into spatial tensors. The class-specific image inpainting system 102 then uses the spatial tensors to modulate the input features analyzed by the spatial modulation block.

[0077] For example, Figure 4 Additional details regarding the operation of the global modulation block and the spatial modulation block, according to one or more embodiments, are provided. Specifically, Figure 4 Global modulation block 402 and spatial modulation block 403 are shown. For example... Figure 4 As shown, the global modulation block 412 includes a first global modulation operation 404 and a second global modulation operation 416. Furthermore, the spatial modulation block 413 includes a global modulation operation 401 and a spatial modulation operation 410.

[0078] For example, a modulation block (or modulation operation) includes a computer-implemented process for modulating (e.g., scaling or shifting) an input signal based on one or more conditions. For instance, a modulation block might involve amplifying certain features while simultaneously canceling / normalizing these amplifications to preserve operations within the generative model. Therefore, a modulation block (or modulation operation) could, for example, include a modulation layer, a convolutional layer, and a normalization layer. The modulation layer scales each input feature of the convolution, and the normalization removes the effects of scaling from the statistics of the output feature map of the convolution.

[0079] In fact, because the modulation layer modulates feature statistics, a modulation block (or modulation operation) typically includes one or more methods to handle these statistical changes. For example, a modulation block (or modulation operation) may include a computer-implemented process that normalizes features using batch normalization or instance normalization. Modulation is achieved by scaling and shifting the activations based on affine parameters predicted from the input conditions. Similarly, some modulation processes replace feature normalization with demodulation. Therefore, a modulation block (or modulation operation) may include a modulation layer, a convolutional layer, and a demodulation layer. For example, in one or more embodiments, the modulation block (or modulation operation) includes the modulation methods described in StyleGan2. A modulation block may include one or more modulation operations.

[0080] Furthermore, a global modulation block (or global modulation operation) refers to a modulation block (or modulation operation) that modulates an input signal in a spatially invariant manner. For example, a global modulation block (or global modulation operation) performs modulation based on global features of a digital image (e.g., features that do not spatially vary in feature maps or image coordinates). Therefore, for example, a global modulation block includes a modulation block that modulates the input signal based on image codes (e.g., global feature codes) generated by an encoder. A global modulation block may include multiple global modulation operations.

[0081] A spatial modulation block (or spatial modulation operation) refers to a modulation block (or modulation operation) that modulates an input signal in a spatially varying manner (e.g., according to a spatially varying feature map). Specifically, a spatial modulation block (or spatial modulation operation) may utilize a spatial tensor to modulate the input signal in a spatially varying manner. Therefore, in one or more embodiments, a global modulation block applies global modulation, where the affine parameters are consistent across spatial coordinates. A spatial modulation block applies a spatially varying affine transformation that varies across spatial coordinates. In some embodiments, a spatial modulation block may include both a spatial modulation operation and another modulation operation (e.g., a global modulation operation and a spatial modulation operation).

[0082] For example, spatial modulation operations may include spatially adaptive modulation, as described by Taesung Park, Ming Yu Liu, Ting Chun Wang, and Jun Yan Zhu in “Semantic image synthesis with spatially-adaptive normalization,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019), which is incorporated herein by reference in its entirety (hereinafter referred to as Taesung). In some embodiments, spatial modulation operations utilize spatial modulation operations with a different architecture than Taesung's, including modulation convolution demodulation pipelines, as described in more detail below.

[0083] Therefore, regarding Figure 4 The class-specific image inpainting system 102 utilizes a global modulation block 402. As shown, the global modulation block 412 includes a first global modulation operation 404 and a second global modulation operation 416. Specifically, the first global modulation operation 404 processes the input global feature map 412. For example, the input global feature map 412 includes feature vectors generated by a cascaded modulation inpainting neural network that reflect global features (e.g., high-level features or features corresponding to the entire digital image). Thus, for example, the global feature map 412 includes feature vectors that reflect global features generated from previous global modulation blocks in the cascaded decoder layer. The global feature map 412 may also include feature vectors corresponding to encoded feature vectors generated by the encoder (e.g., in the first decoder layer, the class-specific image inpainting system 102 may utilize encoded feature vectors, pattern codes, global feature codes, constants, noise vectors, or other feature vectors as input).

[0084] As shown in the figure, the first global modulation operation 404 includes a modulation layer 404a, an upsampling layer 404b, a convolutional layer 404c, and a normalization layer 404d. Specifically, the class-specific image inpainting system 102 uses the modulation layer 404a to perform global modulation on an initial global feature map 412 based on a global feature code 414 (e.g., a global feature code 316). Specifically, the class-specific image inpainting system 102 applies a neural network layer (i.e., a fully connected layer) to the global feature code 414 to generate a global feature vector 416. The class-specific image inpainting system 102 then modulates the initial global feature map 412 using the global feature vector 406.

[0085] Furthermore, the class-specific image inpainting system 102 applies an upsampling layer 404b (e.g., to modify the resolution scale). Additionally, the class-specific image inpainting system 102 applies a convolutional layer 404c. Furthermore, the class-specific image inpainting system 102 applies a normalization layer 404d to complete a first global modulation operation 404. As shown, the first global modulation operation 404 generates global intermediate features 418. Specifically, in one or more embodiments, the class-specific image inpainting system 102 generates global intermediate features 418 by combining (e.g., concatenating) the output of the first global modulation operation 404 with an encoded feature vector 420 (e.g., from a convolutional layer of an encoder with a matching scale / resolution).

[0086] As shown in the figure, the class-specific image inpainting system 102 also utilizes a second global modulation operation 406. Specifically, the class-specific image inpainting system applies the second global modulation operation 406 to global intermediate features 418 to generate a new global feature map 422. Specifically, the class-specific image inpainting system 102 applies a global modulation layer 406a to the global intermediate features 418 (e.g., based on global feature vector 416). Furthermore, the class-specific image inpainting system 102 applies a convolutional layer 406b and a normalization layer 406c to generate the new global feature map 422. As shown in the figure, in some embodiments, the class-specific image inpainting system 102 applies a spatial offset when generating the new global feature map 424.

[0087] In addition, such as Figure 4 As shown, the class-specific image inpainting system 102 utilizes a spatial modulation block 403. Specifically, the spatial modulation block 413 includes a global modulation operation 408 and a spatial modulation operation 410. The global modulation operation 418 processes the input local feature map 424. For example, the input local feature map 424 includes feature vectors generated by the cascaded modulation inpainting neural network that reflect local features (e.g., low-level, specific, or spatially varied features). Thus, for example, the local feature map 424 includes feature vectors reflecting local features generated from previous spatial modulation blocks in the cascaded decoder layer. The global feature map 412 may also include feature vectors corresponding to encoded feature vectors generated by the encoder (e.g., in the first decoder layer, the class-specific image inpainting system 102 may utilize encoded feature vectors, style codes, noise vectors, or other feature vectors).

[0088] As shown in the figure, the class-specific image inpainting system 102 utilizes a global modulation operation 408 to generate local intermediate features 426 from a local feature map 424. Specifically, the class-specific image inpainting system 102 applies a modulation layer 408a, an upsampling layer 408b, a convolutional layer 408c, and a normalization layer 408d. Furthermore, in one or more embodiments, the class-specific image inpainting system 102 applies spatial offset and broadcast noise to the output of the global modulation operation 408 to generate the local intermediate features 426.

[0089] like Figure 4 As shown, the class-specific image inpainting system 102 generates a new local feature map 428 using a spatial modulation operation 410. Specifically, the spatial modulation operation 420 modulates local intermediate features 426 based on global intermediate features 418. More specifically, the class-specific image inpainting system 102 generates a spatial tensor 430 from the global intermediate features 418. For example, the class-specific image inpainting system 102 applies a convolutional affine parameter network to generate the spatial tensor 430. In particular, the class-specific image inpainting system 102 applies a convolutional affine parameter network to generate an intermediate spatial tensor. The class-specific image inpainting system 102 combines the intermediate spatial tensor with the global feature vector 416 to generate the spatial tensor 430. The class-specific image inpainting system 102 uses the spatial tensor 420 to modulate the local intermediate features 426 (using the spatial modulation layer 410a) and generates a modulation tensor.

[0090] As shown in the figure, the class-specific image inpainting system 102 also applies a convolutional layer 410b to the modulation tensor. In particular, the convolutional layer 410b generates a convolutional feature representation from the modulation tensor. Furthermore, the class-specific image inpainting system 102 applies a normalization layer 410c to the convolutional feature representation to generate a new local feature map 428.

[0091] Although shown as normalization layer 410c, in one or more embodiments, the class-specific image inpainting system 102 applies a demodulation layer. For example, the class-specific image inpainting system 102 applies a modulation convolution demodulation pipeline (e.g., general normalization rather than instance normalization). This approach avoids potential artifacts (e.g., water droplet artifacts) caused by instance normalization. In fact, the demodulation / normalization layer comprises a layer that scales each output feature map by uniform demodulation / normalization values ​​(e.g., instance normalization using a data correlation constant instead of uniform standard deviation).

[0092] like Figure 4 As shown, in some embodiments, the class-specific image inpainting system 102 also applies a shift tensor 432 and broadcast noise to the output of spatial modulation operation 410. For example, spatial modulation operation 420 generates normalized / demodulated features. The class-specific image inpainting system 102 also generates the shift tensor 432 by applying an affine parameter network to global intermediate features 418. The class-specific image inpainting system 102 combines the normalized / demodulated features, the shift tensor 422, and / or broadcast noise to generate a new local feature map 428. In one or more embodiments, as shown, the class-specific image inpainting system 102 also combines noise modulation to generate a new local feature map 428.

[0093] After generating new global feature map 422 and new local feature map 428, the class-specific image inpainting system 102 proceeds to the next cascaded modulation layer in the decoder. For example, the class-specific image inpainting system 102 utilizes the new global feature map 422 and new local feature map 428 as input features for additional cascaded modulation layers at different scales / resolutions. The class-specific image inpainting system 102 then utilizes the additional cascaded modulation layers to generate additional feature maps (e.g., utilizing additional global modulation blocks and additional spatial modulation blocks). The class-specific image inpainting system 102 can iteratively process the feature maps using the cascaded modulation layers until the final scale / resolution is reached to generate the inpainted digital image.

[0094] although Figure 4 Global modulation block 402 and spatial modulation block 403 are shown, but in some embodiments, class-specific image inpainting system 102 utilizes a global modulation block followed by (e.g., cascaded to) another global modulation block. For example, class-specific image inpainting system 102 replaces spatial modulation block 403 with an additional global modulation block. In such embodiments, class-specific image inpainting system 102 replaces with skip connections. Figure 4 The APN (and spatial tensor) and corresponding spatial modulation are shown. For example, class-specific image inpainting system 102 utilizes global intermediate features to perform global modulation on local intermediate vectors. Therefore, class-specific image inpainting system 102 can utilize a first global modulation block and a second global modulation block.

[0095] As mentioned above, the decoder can also be described using variables and equations to illustrate the operation of the cascaded modulation repair neural network. For example, as described above, the decoder stacks a series of cascaded modulation blocks to modify the input feature map. Upsampling is performed. Each cascaded modulation block takes the global code g as input to modulate features based on the global representation of the local image. Furthermore, the class-specific image inpainting system 102 provides a mechanism for correcting local errors after predicting the global structure.

[0096] Specifically, class-specific image inpainting system 102 utilizes cascaded modulation blocks to address the challenge of generating coherent global and local features. At a high level, class-specific image inpainting system 102 follows the approach of: i) decomposing global and local features to separate local details from the global structure, and ii) predicting the cascade of global and spatial modulations of local details from the global structure. In one or more implementations, class-specific image inpainting system 102 utilizes spatial modulation generated from global codes for better prediction (e.g., and discards instance normalization to make the design compatible with StyleGAN2).

[0097] More specifically, cascaded modulation draws global and local features from previous scales. and And the global code g is used as input, and new global features are generated at the next scale / resolution. and local features In order to Generate new global code The class-specific image inpainting system 102 utilizes a global code modulation level that includes a modulation convolution demodulation process. This generates upsampled features X.

[0098] Due to the limited expressive power of the global vector g in representing 2D visual details, and the inconsistency between the inside and outside of the aperture, global modulation can generate distorted features inconsistent with the context (such as those related to the aperture). Figure 5 (Discussed in more detail). To perform compensation, the class-specific image inpainting system 102 utilizes spatial modulation to generate more accurate features. Specifically, the spatial modulation modulates the input local features in a spatially adaptive manner, using X as a spatial code and g as a global code.

[0099] Furthermore, the class-specific image inpainting system 102 utilizes a unique spatial modulation and demodulation mechanism to avoid potential "water droplet" artifacts caused by instance normalization in conventional systems. As shown in the figure, spatial modulation follows a modulation-convolution demodulation pipeline.

[0100] Specifically, for spatial modulation, the class-specific image inpainting system 102 generates a spatial tensor A0 = APN(Y) from feature X via a two-layer convolutional affine parameter network (APN). Simultaneously, the class-specific image inpainting system 102 generates a global vector α = fc(g) from a global code g with fully connected layers (fc) to capture the global context. The class-specific image inpainting system 102 generates a final spatial tensor A = A0 + α as a broadcast sum of A0 and α for use with element-wise multiplication. The intermediate feature Y of the scaling block:

[0101]

[0102] Furthermore, for convolution, the modulation tensor Convolution with a 3×3 learnable kernel K yields:

[0103]

[0104] For spatially-aware demodulation, the class-specific image inpainting system 102 applies demodulation steps to calculate the normalized output. Specifically, the class-specific image inpainting system 102 can assume that the input feature Y is an independent random variable with unit variance, and that the expected variance of the output remains unchanged after modulation, i.e. Therefore, this gives the demodulation calculation:

[0105]

[0106] in These are the demodulation coefficients. The class-specific image restoration system 102 can implement the above equation using standard tensor operations.

[0107] In one or more implementations, the class-specific image inpainting system 102 also adds spatial offsets and broadcast noise. For example, the class-specific image inpainting system 102 normalizes features. A shift tensor B = APN(X) is added to the feature X and the broadcast noise n generated by another affine parameter network (APN) to produce new local features.

[0108]

[0109] As described above, in some embodiments, the class-specific image inpainting system 102 also modulates noise. Specifically, the class-specific image inpainting system 102 samples noise from a normal distribution. Furthermore, the noise modulation factor intensity is calculated using 3×3 convolution. And it generates modulation noise n′. Then, the class-specific image inpainting system 102 determines local features by adding spatial bias and noise according to the following formula:

[0110]

[0111] For example, consider the following example pseudocode for spatial modulation operation 410. Specifically, the affine parameter network (APN) is implemented as a 3-layer convolutional network that takes X as input to generate scaling parameter A and shift parameter B.

[0112] def APN(X):

[0113] #1×1 Input Layer

[0114] tl = self.conV1_1x1(X)

[0115] #3×3+1×1 Intermediate Layer

[0116] t2 = self.conV2_3x3(t1)

[0117] t2 = t2 + self.conv2_1x1(t1)

[0118] #1×1 Output Layer

[0119] A = self.conv_A_1x1(t)

[0120] B = self.conv_B_1x1(t)

[0121] Return A, B

[0122] Next, spatial modulation will feature Figure X Y is modulated using Y and the global code g as inputs:

[0123] import torch.nn.functional as F

[0124] def spatial_mod(X, Y, g, W, noise):

[0125] bs = X.size(0) # Batch size

[0126] #Get Space Code

[0127] A, B = self.APN(X)

[0128] #Merge with global codes

[0129] A=A+self.fc(g).reshape(bs,-1,1,1)

[0130] #Spatial Modulation

[0131] Y = Y.mul(A)

[0132] #conv

[0133] Y = F.conv2d(Y, w)

[0134] #Spatial perception normalization

[0135] w = w.unsqueeze(0)

[0136] A_avg_var=A.square().mean([2, 3])

[0137] reshape(bs, 1, -1, 1, 1)

[0138] D=(w.square().mul(A_avg_var)

[0139] .sum(dim=[2,3,4])+1e-8

[0140] ).rsqrt()

[0141] Y=Y.mul(D.reshape(bs,-1,1,1))

[0142] #Add bias and noise

[0143] Y = Y + B + noise

[0144] return Y

[0145] In one or more embodiments, the class-specific image inpainting system 102 uses a neural network as described in “DIGITAL IMAGEINPAINTING UTILIZING A CASCADED MODULATION INPAINTING NEURAL NETWORK”, App. No. 17 / 661,985, filed on May 4, 2022, which is incorporated herein by reference.

[0146] As described above, in one or more embodiments, the class-specific image inpainting system 102 also utilizes positional encoding to enhance structural prediction. For example, Figure 5 The generation and utilization of position codes according to one or more embodiments are illustrated. Specifically, Figure 5 The diagram shows the insertion of position codes 502a to 502n into the various layers of the cascaded modulation repair neural network 302.

[0147] Location encoding comprises a digital representation of the place or position of an item in a sequence. Specifically, location encoding comprises a finite-dimensional representation (e.g., a vector or tensor) of the position of an item in the sequence. Thus, a model can utilize location encoding to determine the place or position of a value in a sequence. Therefore, in one or more implementations, the location encoding has the same dimension as the sequence in question. For example, the size of the location encoding matches the resolution / size of a feature vector or feature map (e.g., a global feature map or a local feature map). To address variable length and scaling issues, in one or more embodiments, class-specific image inpainting system 102 utilizes location encoding comprising a matrix or other digital representation of finite length / dimension and a fixed range of values ​​(e.g., between a predetermined set of values).

[0148] In one or more implementations, the class-specific image inpainting system 102 utilizes Fourier features as positional encoding. For example, the class-specific image inpainting system 102 uses the following Fourier features to positionally encode entries (e.g., entries in a feature map):

[0149] PE=[sin(ω0i), cos(ω0i),..., sin(ω0j), cos(ω0i),...,]

[0150] ω0 = 1 / size

[0151] ω1=2 / siz

[0152] ω2=3 / size

[0153] ω n =1

[0154] Where i is a discrete horizontal position in the sequence (e.g., in a feature map), j is a discrete vertical position in the sequence (e.g., in a feature map), ω is the frequency of change used to encode the position (from 1 / size to 1), and n is the dimension of the position encoding (and the corresponding feature map). Therefore, the sin(ω0i), cos(ω0i), ..., components reflect the height dimension of the feature map entries, while the sin(ω0j), sin(ω0j), ..., components refer to the width dimension of the feature map entries. As described above, in one or more implementations, the total dimension of the position encoding matches the resolution / dimensionality of the corresponding sequence (e.g., the feature map). Therefore, each layer of the cascaded modulation instigation neural network 302 can utilize position encoding with a dimension matching the feature vector of that layer.

[0155] In one or more embodiments, the positional encoding equation described above reflects the positional encoding of a single entry. Therefore, in one or more implementations, the class-specific image inpainting system 102 generates positional encodings that include an individual positional encoding matrix of entries (e.g., vectors) of the feature map. In one or more embodiments, the class-specific image inpainting system 102 combines the individual positional encodings with individual entries (e.g., feature vectors) of the feature map (e.g., concatenation, addition, multiplication, etc.).

[0156] For example, positional encodings 502a to 502b include positional encodings for each entry corresponding to the feature vector at each corresponding layer of the network. For example, in one or more implementations, the class-specific image inpainting system 102 utilizes a first encoder layer 308a to process a 512×512 input feature set. The class-specific image inpainting system 102 combines the input feature vector with positional encodings 502a (e.g., 512×512 positional encodings) whose dimensions correspond to those of the first encoder layer 308a.

[0157] As shown in the figure, the first encoder layer 308a generates a feature vector, which is then analyzed by the second encoder layer 308 at different resolutions / dimensions (e.g., 256×256). The class-specific image inpainting system 102 generates a positional code 502b with dimensions corresponding to the second encoder layer 308b. Furthermore, the class-specific image inpainting system 102 combines the positional code 502b with the feature vector generated by the first encoder layer 308a. The second encoder layer 308b then processes this combined positional feature vector. Additionally, as shown in the figure, the class-specific image inpainting system 102 uses a similar method to generate and utilize positional codes 502c and 502d with additional encoder layers 308c and 308n.

[0158] Furthermore, the class-specific image inpainting system 102 also generates and utilizes positional codes of the cascaded modulator-decoder layers 502e to 502n. For example, the class-specific image inpainting system 102 combines the positional codes 502e to 502n with global and local feature maps at each layer of the cascaded modulator-decoder layers 502e to 502n. Specifically, the class-specific image inpainting system 102 generates positional codes 502e to 502n with different dimensions corresponding to the resolution / dimension of the cascaded decoder layers 320a to 320n. For example, in one or more embodiments, if the cascaded decoder layer 320b has an 8×8 resolution / dimension, the class-specific image inpainting system 102 utilizes 8×8 positional codes for positional coding 502e.

[0159] In one or more implementations, the class-specific image inpainting system 102 reuses one or more positional codes from the encoder layer for the decoder layer. For example, the class-specific image inpainting system 102 uses the same dimensions for the first encoder layer 308a as for the final decoder layer 320n. Because they have the same dimensions, in one or more implementations, the class-specific image inpainting system 102 uses the same positional codes for positional codes 502a and 502n. In other embodiments, the class-specific image inpainting system 102 generates separate positional codes.

[0160] As described above, in one or more embodiments, the class-specific image inpainting system 102 also utilizes a unique method to train a class-specific cascaded modulation inpainting neural network. For example, Figure 6 An overview of a class-specific image inpainting system 102 for training a class-specific cascaded modulation inpainting neural network according to one or more embodiments is shown.

[0161] Specifically, Figure 6 The diagram illustrates a series of actions performed by a class-specific image inpainting system 102 during the training of a class-specific cascaded modulation inpainting neural network. Specifically, as shown, the class-specific image inpainting system 102 performs action 602, receiving digital images depicting object classes. For example, in one or more implementations, the class-specific image inpainting system 102 performs action 602 by accessing a repository of training digital images. The class-specific image inpainting system 102 can identify the training digital images depicting object classes by utilizing segmentation models, such as panoramic segmentation models that identify objects and corresponding object segments.

[0162] In addition, such as Figure 6As shown, the class-specific image inpainting system 102 also performs an action 606 of generating a predicted inpainted digital image. For example, the class-specific image inpainting system 102 performs action 606 by providing a digital image depicting the object class to a cascaded modulation inpainting neural network and generating the inpainted digital image. In some implementations, the class-specific image inpainting system 102 utilizes a class-segmentation digital image that blocks, masks, or segments instances of the object class from the digital image. For example, the class-specific image inpainting system 102 uses a mask determined by a panoptic segmentation model to block or remove pixels depicting instances of the object class to generate a class-segmentation digital image. The class-specific image inpainting system 102 then uses the cascaded modulation inpainting neural network to generate the predicted inpainted digital image from the class-segmentation digital image.

[0163] As shown in the figure, the class-specific image inpainting system 102 also performs action 606 to modify the parameters of the cascaded modulation inpainting neural network to generate the class-specific cascaded modulation inpainting neural network 608. Specifically, the class-specific image inpainting system 102 uses the predicted inpainted digital image (from action 604) to modify the parameters of the cascaded modulation inpainting neural network. For example, as shown in the figure, the class-specific image inpainting system 102 determines the adversarial loss from the inpainted digital image by utilizing a decoder neural network. For instance, the class-specific image inpainting system 102 generates a realism prediction by utilizing the decoder neural network and determines the adversarial loss from the realism prediction. Then, the class-specific image inpainting system 102 learns the parameters of the cascaded modulation inpainting neural network from the adversarial loss.

[0164] As shown in the figure, by modifying the parameters of the cascaded modulation inpainting neural network in this way, the class-specific image inpainting system 102 generates a class-specific cascaded modulation inpainting neural network 608. In fact, by training the class-specific cascaded modulation inpainting neural network using class-segmented digital images, the class-specific cascaded modulation inpainting neural network learns to generate inpainted digital images depicting a specific object class.

[0165] The class-specific image inpainting system 102 can generate various different class-specific cascaded modulation inpainting neural networks. In fact, as shown in the figure, the class-specific image inpainting system 102 can generate class-specific cascaded modulation inpainting neural networks trained to generate sky object classes, ground object classes, water object classes, and / or human object classes. The class-specific image inpainting system 102 can select an appropriate class-specific cascaded modulation inpainting neural network from a plurality of class-specific cascaded modulation inpainting neural networks in response to a specific request for inpainting a digital image. For example, if a client device identifies a replacement region with a sky object class indication, the class-specific image inpainting system 102 can select a class-specific cascaded modulation inpainting neural network trained to generate a sky region (e.g., segmenting a digital image from a sky-specific class). Similarly, in response to a client device selecting a replacement region with a ground object class indication, the class-specific image inpainting system 102 can select a class-specific cascaded modulation inpainting neural network trained to generate a ground region (e.g., segmenting a digital image from a ground-specific class).

[0166] For example, Figure 7 Additional details are provided regarding a class-specific image inpainting system 102 for training a class-specific cascaded modulation inpainting neural network according to one or more embodiments. As shown, the class-specific image inpainting system 102 identifies a digital image 702 and utilizes a panoramic segmentation model 704 to identify a digital image depicting an object class 706 and a segmentation mask 708. The panoramic segmentation model includes a computer-implemented model for assigning semantic labels to pixels in the digital image. For example, the panoramic segmentation model includes a machine learning model that predicts a semantic label for each pixel in the digital image, thereby segmenting the digital image into semantically labeled regions. In some implementations, the class-specific image inpainting system 102 utilizes a neural network panoramic segmentation model. For example, in one or more implementations, class-specific image inpainting system 102 utilizes a panoptic segmentation neural network, as described by Y.Li, H.Zhao, X.Qi, L.Wang, Z.Li, J.Sun, and J.Jia in “Full Convolutional Networks for Panoptic Segmentation”, CVPR 2021, arXiv:2012.00720v2.

[0167] For example, class-specific image inpainting system 102 uses panoramic segmentation model 704 to identify all digital images depicting pixels with sky semantic labels. Class-specific image inpainting system 102 uses these sky digital images as digital images depicting object class 706. Furthermore, class-specific image inpainting system 102 also identifies those pixels depicting sky regions and generates segmentation mask 708 from these pixels. Therefore, class-specific image inpainting system 102 generates segmentation mask 708 to block or cover instances of the object class.

[0168] By applying segmentation mask 708, class-specific image inpainting system 102 generates a class-segmented digital image. Specifically, class-specific image inpainting system 102 segments object instances from a digital image depicting object class 706. For example, the class-segmented digital image may include a digital image and a mask that covers one or more instances of the object class depicted in the digital image.

[0169] like Figure 7 As shown, in one or more embodiments, the class-specific image inpainting system 102 also generates and applies an extended segmentation mask 710. For example, in one or more examples, the class-specific image inpainting system 102 applies an extension operation to a segmentation mask 708 to generate the extended segmentation mask 710. For example, the extension operation may include a computer-implemented model or process for extending the mask or mask pixels. For instance, the extension operation may extend the mask by a predetermined number of pixels (e.g., three or five pixels) to expand the size of the mask.

[0170] Since the segmentation mask 708 may not cover all pixels of an object class, in some embodiments, the class-specific image inpainting system 102 applies an extension operation and generates an extended segmentation mask 710 to reduce the likelihood that the class-segmented digital image will include pixels corresponding to the object class. Thus, for example, segmentation of the sky region might leave a small perimeter around the pixels depicting the sky. By applying the extension operation, the class-specific image inpainting system 102 can segment / cover / block such perimeters of pixels when generating the class-segmented digital image.

[0171] As shown in the figure, the class-specific image inpainting system 102 utilizes a cascaded modulation inpainting neural network 712 to process digital images depicting object class 706 and segmentation mask 708 (or extended segmentation mask 710). Specifically, the class-specific image inpainting system 102 generates a class-segmented digital image (by providing the digital image depicting object class 706 and segmentation mask 708 as input to the cascaded modulation inpainting neural network 712). As described above, the cascaded modulation inpainting neural network 712 utilizes an encoder layer and a cascaded modulation decoder layer to generate a repaired digital image 714.

[0172] Furthermore, the class-specific image restoration system 102 utilizes the restored digital image 714 to determine an adversarial loss 720. Specifically, the class-specific image restoration system 102 utilizes a discriminator neural network 716. As described above, the discriminator neural network 716 analyzes the input digital image and generates a authenticity prediction. For example, the discriminator neural network 716 can acquire a real digital image (e.g., a digital image not generated by the cascaded modulation restoration neural network 712) and predict whether the digital image is real or fake. In one or more embodiments, the class-specific image restoration system 102 compares this authenticity prediction with an authenticity label (e.g., real or fake) to determine the adversarial loss. The class-specific image restoration system 102 utilizes this adversarial loss to train the discriminator neural network.

[0173] Similarly, the class-specific image inpainting system 102 can also utilize a discriminator neural network 716 to analyze the digital image generated by the cascaded modulation inpainting neural network 712. In fact, as shown, the class-specific image inpainting system 102 uses the discriminator neural network 716 to generate a authenticity prediction 718 from the inpainted digital image 714. The class-specific image inpainting system 102 compares the authenticity prediction 718 with the authenticity label (e.g., false label) of the inpainted digital image 714 to determine an adversarial loss 720.

[0174] Furthermore, as shown in the figure, the class-specific image inpainting system 102 also utilizes adversarial loss 720 to modify the parameters of the cascaded modulation inpainting neural network 712 and / or the discriminator neural network 716. For example, the class-specific image inpainting system 102 uses gradient descent and backpropagation techniques to modify the internal parameter weights of each layer of the cascaded modulation inpainting neural network 712 and the discriminator neural network 716. In this way, the discriminator neural network 716 becomes better at distinguishing between real digital images and pseudo-digital images. Furthermore, the cascaded modulation inpainting neural network 712 becomes better at generating inpainted digital images corresponding to the object class. Therefore, the class-specific image inpainting system 102 learns parameters that make the cascaded modulation inpainting neural network become a class-specific cascaded modulation inpainting neural network 712.

[0175] In one or more embodiments, the class-specific image inpainting system 102 utilizes masking regularization to learn parameters for a generative inpainting neural network. Specifically, the class-specific image inpainting system 102 utilizes modified regularization techniques, such as R1 regularization specifically tailored for inpainting digital images. For example, the class-specific image inpainting system 102 modifies the R1 regularization term to avoid penalizing portions of the image and thus better separates the input conditions from the generated output. In some cases, the class-specific image inpainting system 102 modifies the R1 regularization using a digital image mask to form a masking R1 regularization term. By utilizing masking regularization, in one or more embodiments, the class-specific image inpainting system 102 reduces or eliminates the detrimental effects of calculating regularization on the digital image background. In one or more embodiments, the class-specific image inpainting system 102 utilizes a training method as described in U.S. Patent Application No. 17 / 650967, filed February 14, 2022, entitled “Learning Parameters Forgive Inpainting Nural Networks Using Object-Aware Training and Masked Regularization of Generative Inpainting Neural Networks,” which is incorporated herein by reference in its entirety. In one or more embodiments, the class-specific image inpainting system 102 avoids the use of reconstruction loss (such as perceptual loss).

[0176] although Figures 2 to 7 The class-specific image inpainting system 102, utilizing a cascaded modulation inpainting neural network, is illustrated. It can leverage various inpainting neural networks. For example, the class-specific image inpainting system 102 can utilize the unique training methods described above with respect to the various inpainting neural networks referenced in this disclosure. In fact, in some embodiments, the class-specific image inpainting system 102 trains and utilizes a class-specific inpainting neural network having encoder and decoder layers but not including a cascaded modulation layer. Therefore, the above description of the class-specific cascaded modulation inpainting neural network can also be implemented using a class-specific inpainting neural network with an architecture lacking a cascaded modulation decoder layer.

[0177] also. Figures 2 to 7The diagram illustrates the generation of undepicted digital images for specific classes (such as sky, water, ground, etc.), which may also include object types within those specific classes. Thus, for example, class-specific image inpainting system 102 may utilize a first class-specific inpainting neural network trained to generate cloudy skies (first class) and a second class-specific inpainting neural network trained to generate blue skies (second class). Therefore, class-specific image inpainting system 102 may utilize different class-specific inpainting neural networks to generate different types of classes or different subclasses (e.g., storm, embers, blue, cloudy). In one or more embodiments, class-specific image inpainting system 102 determines the specific type of class based on user input. For example, class-specific image inpainting system 102 may receive textual user input (e.g., “cloudy sky”) or a digital image depicting a specific type of class (e.g., a digital image depicting a cloudy sky), and then generate replacement pixels for the inpainted digital image depicting that type of class. Each of these types is included in the primary term “class” as used herein.

[0178] As described above, in one or more embodiments, the class-specific image restoration system 102 also generates various user interfaces for generating restored digital images. For example, Figure 8A A digital image 806 is shown depicted through a user interface 804 of a client device 802. In response to user interaction via the user interface 806, a class-specific image restoration system 102 can identify replacement areas of the digital image.

[0179] For example, Figure 8BA user interface 804 is shown for identifying the sky region of a digital image 806. Specifically, the class-specific image inpainting system 102 receives user interactions (e.g., user selections) of the sky region 808, such as clicks, presses, or other selection events. In one or more embodiments, the class-specific image inpainting system 102 utilizes a segmentation model to identify the sky region 808. For example, the class-specific image inpainting system 102 utilizes a segmentation model that identifies salient objects, segments objects based on user selection, and / or segments all objects in the digital image. For example, the class-specific image inpainting system 102 utilizes the model described in Deep GrabCut For Object Selection by Ning Xu et al., published July 14, 2017, which is incorporated herein by reference in its entirety. Alternatively, the class-specific image inpainting system 102 utilizes one or more models described in the following documents: U.S. Patent Application Publication No. 2019 / 0130229, filed October 31, 2017, entitled “DeepSalient Content Neural Networks for Efficient Digital Object Segmentation”; U.S. Patent Application No. 16 / 035,410, filed July 13, 2018, entitled “Automatic Trimap Generation and Image Segmentation”; or U.S. Patent No. 10,192,129, filed November 18, 2015, entitled “Utilizing Interactive Deep Learning to Select Objects in Digital VisualMedia”, each of which is incorporated herein by reference in its entirety.

[0180] In this way, the class-specific image restoration system 102 identifies the replacement area. In one or more embodiments, the class-specific image restoration system 102 also identifies the target object class corresponding to the replacement area. For example, the class-specific image restoration system 102 may receive (via user interface 804) a user selection of a specific target object class (e.g., sky, ground, water) to be included in the replacement pixels of the replacement area. In some embodiments, the class-specific image restoration system 102 identifies the target object class based on the selection of elements (e.g., via buttons or radio buttons on user interface 804). In some embodiments, the class-specific image restoration system 102 identifies the target object class based on the selected tool. For example, the class-specific image restoration system 102 may receive a user selection of a "sky replacement tool". When the sky replacement tool is selected, the class-specific image restoration system 102 may receive a selection of a sky region 808, and (because the user has already identified the sky replacement tool) the class-specific image restoration system 102 may identify the target object class as the sky object class.

[0181] In some implementations, the class-specific image inpainting system 102 determines the target object class by analyzing the digital image and / or the replacement region. For example, the class-specific image inpainting system 102 may utilize a panoptic segmentation model to analyze the digital image and determine that the pixels of the replacement region correspond to a specific semantic category. The class-specific image inpainting system 102 may then use this semantic category as the target object class. Therefore, when identifying a replacement region depicting the sky, the class-specific image inpainting system 102 may select the sky target object class. Similarly, when identifying a replacement region depicting a human, the class-specific image inpainting system 102 may select the human target object class.

[0182] After identifying the sky region 808 and the target object class, the class-specific image inpainting system 102 utilizes a class-specific inpainting neural network to generate an inpainted digital image depicting a new instance of the object class. Specifically, the class-specific image inpainting system 102 selects a class-specific inpainting neural network corresponding to the target object class. Therefore, when determining the sky target object class, the class-specific image inpainting system 102 selects a class-specific inpainting neural network trained to generate the sky region. Similarly, when determining the ground object class (e.g., grass, sand object class), the class-specific image inpainting system 102 selects a class-specific inpainting neural network trained to generate the ground region.

[0183] The class-specific image inpainting system 102 also utilizes a class-specific inpainting neural network to generate an inpainted digital image. Specifically, the class-specific image inpainting system 102 generates a masked digital image and provides the masked digital image to the class-specific inpainting neural network. For example, the class-specific image inpainting system 102 generates a mask covering a sky region 808 and applies the mask to a digital image 806 to cover or block the sky region 806. In one or more embodiments, the class-specific image inpainting system 102 also applies an extension operation to generate an extended mask. In fact, as described above (regarding...), Figure 7 The class-specific image inpainting system 102 can apply an extension operation to the mask to generate an extended mask. The class-specific image inpainting system 102 can then utilize the extended mask to generate a masked digital image for use by the class-specific inpainting neural network. In fact, the class-specific image inpainting system 102 utilizes the class-specific inpainting neural network to generate an inpainted digital image depicting a new instance of the target object.

[0184] For example, Figure 8C A user interface 804 is shown, depicting a restored digital image 812 with replacement pixels 810 having a replacement sky region 808. As shown, the class-specific image inpainting system 102 utilizes a class-specific inpainting neural network to generate replacement pixels depicting the sky instance. By utilizing the class-specific inpainting neural network, the class-specific image inpainting system 102 generates a novel synthetic sky region that blends with the remaining scene in the digital image 806. Furthermore, the class-specific image inpainting system 104 provides client devices with the flexibility to control the content of the replacement pixels 810.

[0185] although Figures 8A to 8C The example illustrates replacing a sky region with a new sky region, but the class-specific image inpainting system 102 can also generate a user interface for adding instances of a target object class that were not previously depicted in the replacement area. For example, the class-specific image inpainting system 102 can generate a sky region in an area of ​​a digital image where there was no sky previously. Similarly, the class-specific image inpainting system 102 can generate a water region in an area of ​​a digital image where water was not previously depicted.

[0186] For example, Figure 9A A digital image 906 is shown displayed via a user interface 904 of a client device 902. The digital image 904 depicts a natural scene with sky and mountains, but no water. A class-specific image restoration system 102 can generate a restored digital image that depicts novel, generated pixels that depict water. For example, as... Figure 9BAs shown, the class-specific image restoration system 102 receives user interaction (e.g., drawing or painting) on ​​the replacement area 908. The class-specific image restoration system 102 also identifies the water target object class. Specifically, the class-specific image restoration system 102 identifies the water target object class based on the selection of a water replacement tool (or the selection of different water class selection elements).

[0187] like Figure 9C As shown, the class-specific image inpainting system 102 utilizes a class-specific inpainting neural network to generate replacement pixels 912 that depict an instance of a target object class (e.g., a lake). Specifically, even if the digital image 906 does not depict water, the class-specific inpainting neural network generates a lake that blends into the surrounding environment of the digital image. Therefore, the class-specific image inpainting system 102 can receive user input (e.g., brush or region input) and generate replacement pixels for the inpainted digital image 910 that depict one or more novel instances of the target object class.

[0188] As described above, the researchers conducted experiments on class-specific image inpainting system 102 to analyze the results of example implementations utilizing class-specific cascaded modulation inpainting neural networks. For example, Figure 10 An example restored digital image generated by an example implementation of class-specific image restoration system 102 is shown. Specifically, class-specific image restoration system 102 generates restored digital images 1002c and 1004c from original digital images 1002b and 1004b. Specifically, class-specific image restoration system 102 generates masked digital images 1002a and 1004a from original digital images 1002b and 1004b, and utilizes an example class-specific cascaded modulation restoration neural network to generate restored digital images 1002c and 1004c from masked digital images 1002a and 1004a. As shown, restored digital images 1002c and 1004c depict new, synthetic instances of the target object class, which are seamlessly integrated with the context of original digital images 1002b and 1004b.

[0189] The researchers also conducted additional objective experiments to compare the class-specific image inpainting system 102 with conventional systems. For example, they performed image inpainting experiments on the Places2 dataset at a resolution of 512×512. An experimental example of training the class-specific image inpainting system 102 (“CM-GAN”) using the Adam optimizer is shown. The learning rate and batch size were set to 0.001 and 32, respectively. CM-GAN takes resized images as input so that the model can predict the global structure of the images. The researchers applied flip enhancements to increase the number of training samples.

[0190] For numerical evaluation, the researchers calculated PSNR, SSIM, Frchet initiation distance (FID), and perceptual image patch similarity distance (LPIPS). The researchers also used paired / unpaired initial identifier scores (P-IDS / U-IDS)

[56] for evaluation. As shown in the figure, the researchers compared the results of CoModGAN, Lama, and ProFill with those of various other systems, including those described in the following:

[0191] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky, in Resolution-robust large mask in painting with fourier convolutions, arXiv preprint arXiv:2109.07161 (2021) (hereinafter “LaMa”);

[0192] Yu Zeng, Zhe Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, and Huchuan Lu in High-resolution image inpainting with iterative confidence feedback and guided upsampling, arXiv preprint arXiv:2005.11742(2020) (hereinafter “ProFill”);

[0193] Yu Zeng, Zhe Lin, Huchuan Lu, and Vishal M. Patel in Cr-fill: Generative image inpainting with auxiliary contextual reconstruction, Proceedings of the IEEE International Conference on ComputerVision (2021) (hereinafter “CRFill”);

[0194] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huangin, Free-form image inpainting with gated convolution, Proceedings of the IEEE International Conference on Computer Vision, pages 4471–4480 (2019) (hereinafter “DeepFill v2”);

[0195] Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li in Generating diverse structure for image inpainting with hierarchical vq-vae, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10775–10784 (2021) (hereinafter “DiverseStructure”).

[0196] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and Mehran Ebrahimi in Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212(2019) (hereinafter “EdgeConnect”);

[0197] Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao in High-fidelity pluralistic image completion with transformers, arXiv preprint arXiv:2103.14031

[0198] (2021)(hereinafter "ICT");

[0199] Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang, and Zhan Xu, in Contextual residual aggregation for ultra high-resolution image inpainting, Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pages 7508–7517 (2020)

[0200] (“HiFill”);

[0201] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H. Li, Shan Liu, and Ge Li, in Structureflow: Image inpainting via structure-aware appearance flow, IEEE International Conference on Computer Vision (ICCV) (2019)

[0202] (hereinafter “StructureFlow”); and

[0203] Yibing Song, Wei Huang, Hongyu Liu, Bin Jiang and Chao Yang in Rethinking image inpainting via a mutual encoder-decoder with feature equalizations, Proceedings of the European Conference on Computer Vision (2020) (hereinafter “MEDFE”).

[0204] Figure 11 Results relative to these other systems are presented. The results show that the experimental implementation of the class-specific image inpainting system 102 (CM-GAN) significantly outperforms all other methods in terms of FID, U-IDS, and P-IDS. Compared to LaMa, CM-GAN reduces FID by more than 50% from 3.864 to 1.749, and is similar in terms of LPIPS, which can be explained by the typically blurred results of LaMa versus the clearer results of CM-GAN.

[0205] also, Figure 12 The inference complexity of various models is illustrated. As shown in the figure, experimental embodiments of the class-specific image inpainting system 102 have a similar number (sometimes fewer) of parameters to CoModGAN and LaMa.

[0206] Looking at it now Figure 13 This will provide additional details regarding the components and functionality of the class-specific image restoration system 102. Specifically, Figure 13An example schematic diagram of a class-specific image inpainting system 102 on an example computing device 1300 (e.g., one or more of client devices 108 and / or servers 104). Figure 13 As shown, the class-specific image restoration system 102 includes an incomplete digital image manager 1302, an encoder manager 1304, a cascaded modulation decoder manager 1306, a restored digital image manager 1308, a user interface manager 1310, a training engine 1312, and a storage manager 1314.

[0207] As described above, the class-specific image restoration system 102 includes an incomplete digital image manager 1302. Specifically, the incomplete digital image manager 1302 acquires, identifies, receives, generates, and / or utilizes incomplete digital images. For example, as described above, the incomplete digital image manager 1302 can receive an incomplete digital image including a digital image with holes or masks representing replacement areas. Furthermore, the incomplete digital image manager 1302 can also receive an indication of a target object class for filling the replacement areas.

[0208] As further described, the class-specific image restoration system 102 includes an encoder manager 1304. Specifically, the encoder manager 1304 manages, trains, maintains, executes, implements, applies, or utilizes an encoder of a cascaded modulation restoration neural network. For example, the encoder manager 1304 uses the techniques described above to generate coded feature vectors (e.g., global feature codes) corresponding to incomplete digital images.

[0209] As shown in the figure, the class-specific image inpainting system 102 also includes a cascaded modulation decoder manager 1306. Specifically, the cascaded modulation decoder manager 1306 trains, manages, maintains, executes, implements, or applies a cascaded modulation decoder to the cascaded modulation inpainting neural network. For example, the cascaded modulation decoder manager 1306 applies multiple cascaded modulation layers (as described above) to generate global and local feature maps for determining replacement pixels in the replacement region. The encoder manager 1304 and / or the cascaded modulation decoder manager 1306 can also select a class-specific cascaded modulation inpainting neural network corresponding to a specific target object class (e.g., from multiple class-specific cascaded modulation inpainting neural networks corresponding to different target object classes).

[0210] The class-specific image restoration system 102 also includes a restoration digital image manager 1308. For example, the restoration digital image manager 1308 can identify, generate, provide, and / or display a restored digital image. For instance, the restoration digital image manager 1308 can identify replacement pixels (generated by the cascaded modulation decoder manager 1306) to generate a restored digital image.

[0211] The class-specific image restoration system 102 also includes a user interface manager 1310. The user interface manager 1310 can manage, generate, monitor, and / or provide user interfaces. For example, the user interface manager can receive user input from one or more user interfaces (e.g., user selections for digital images, replacement areas, and / or target object classes). Similarly, the user interface manager 1310 can provide user interface elements for display via a display device of a client device. For example, the user interface manager 1310 can provide restored digital images for display via a graphical user interface and target object class selection elements.

[0212] The class-specific image inpainting system 102 also includes a training engine 1312. The training engine 1313 can teach, learn, and / or train machine learning models, such as class-specific cascaded modulation inpainting neural networks. For example, as described above, the training engine 1312 can modify the parameters of the class-specific cascaded modulation inpainting neural network based on predicted inpainted digital images generated from class-segmented training digital images (e.g., using adversarial loss).

[0213] The class-specific image restoration system 102 also includes a storage manager 1314. The storage manager 1314 operates in conjunction with or includes one or more memory devices (such as database 112) that store various data, such as digital images 1314a (e.g., incomplete or restored digital images) and / or (multiple) cascaded modulation generative neural networks 1314b (e.g., various parameters / layers of encoders and decoders as described above for different class-specific cascaded modulation restoration neural networks trained for different object classes). For example, the storage manager 1314 may include: a memory device comprising digital images depicting object classes; and a discriminator neural network and a cascaded modulation restoration neural network comprising an encoder and a decoder, wherein the decoder comprises multiple cascaded modulation layers.

[0214] In one or more embodiments, each component of the class-specific image restoration system 102 communicates with each other using any suitable communication technology. Furthermore, the components of the class-specific image restoration system 102 communicate with one or more other devices, including the aforementioned client devices. It will be appreciated that although the components of the class-specific image restoration system 102... Figure 13 The components are shown as separate, but any sub-component can be combined into fewer components, such as a single component, or divided into more components for a specific implementation. Furthermore, although the class-specific image restoration system 102 is described... Figure 13 The components, but at least some of the components used to perform operations in conjunction with the class-specific image restoration system 102 described herein can be implemented on other devices in the environment.

[0215] Components of the class-specific image restoration system 102 include software, hardware, or both. For example, components of the class-specific image restoration system 102 include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices (e.g., computing device 1300). When executed by one or more processors, the computer-executable instructions of the class-specific image restoration system 102 cause the computing device 1300 to perform the methods described herein. Alternatively, components of the class-specific image restoration system 102 include hardware, such as a dedicated processing device for performing a particular function or group of functions. Additionally or alternatively, components of the class-specific image restoration system 102 include a combination of computer-executable instructions and hardware.

[0216] Furthermore, for example, components of the class-specific image restoration system 102 that perform the functions described herein can be implemented as parts of a standalone application, modules of an application, plugins to an application including a content management application, library functions or functions that can be called by other applications, and / or cloud computing models. Therefore, components of the class-specific image restoration system 102 can be implemented as parts of a standalone application on a personal computing device or mobile device. Alternatively or additionally, components of the class-specific image restoration system 102 can be implemented in any application that allows the creation and delivery of content to users, including but not limited to… Experience Manager and Creative Applications in, for example and “ADOBE”, “ADOBEEXPERIENCE MANAGER”, “CREATIVE CLOUD”, “PHOTOSHOP”, “LIGHTROOM” and “INDESIGN” are registered trademarks or trademarks of Adobe Inc. in the U.S. and / or other countries.

[0217] Figures 1 to 13 The corresponding text and examples provide a variety of systems, methods, and non-transitory computer-readable media for training generative inpainting neural networks through object-aware training and / or masking regularization to achieve accurate digital image inpainting. In addition to the above, embodiments can also be described according to flowcharts including actions for achieving specific results. For example, Figure 14 A flowchart illustrating an example sequence or series of actions according to one or more embodiments is shown.

[0218] Although Figure 14 Actions according to a particular embodiment are shown, but alternative embodiments may omit, add, reorder, and / or modify them. Figure 14 Any action shown. Figure 14The action can be performed as part of a method. Alternatively, a non-transitory computer-readable medium can include instructions that, when executed by one or more processors, cause a computing device to perform... Figure 14 The action. In another embodiment, the system can perform... Figure 14 The actions described herein can be performed in parallel with each other or repeated or executed in parallel with different instances of the same or other similar actions.

[0219] Figure 14 A series of example actions 1400 for generating repaired digital images using a class-specific (cascaded modulation) repair neural network are illustrated. Specifically, the series of actions 1400 includes action 1402 of receiving an indication of a replacement region of the digital image and a target object class. For example, in one or more embodiments, action 1402 includes receiving an indication of the replacement region of the digital image and the target object class via a user interface of a client device.

[0220] For example, in one or more embodiments, receiving instructions on a replacement region and a target object class includes: providing a digital image for display via a user interface; and receiving a user selection corresponding to the replacement region via the user interface using a selection tool corresponding to the target object class. Furthermore, in one or more embodiments, action 1402 includes determining the replacement region using a segmentation model and the user selection.

[0221] Furthermore, a series of actions 1400 includes an action 1404 of generating replacement pixels using a class-specific (cascaded modulation) repair neural network. For example, in one or more embodiments, action 1404 includes generating replacement pixels for the replacement region using a class-specific (cascaded modulation) repair neural network corresponding to the class of the target object.

[0222] In one or more implementations, action 1404 includes generating a mask corresponding to the replacement region; and generating replacement pixels from the mask and the digital image using a class-specific (cascaded modulation) instigation neural network. Furthermore, in some implementations, action 1404 includes generating replacement pixels using a class-specific (cascaded modulation) instigation neural network corresponding to at least one of a sky object class, a water object class, a ground object class, or a human object class.

[0223] For example, in one or more embodiments, generating replacement pixels using a class-specific (cascaded modulation) inpainting neural network includes generating image encoding using an encoder layer of the class-specific (cascaded modulation) inpainting neural network. Furthermore, generating image encoding using an encoder layer of the class-specific (cascaded modulation) inpainting neural network includes: generating positional encodings corresponding to different resolutions of the encoder layer; and generating multiple encoded feature vectors using the encoder layer and the positional encodings.

[0224] Furthermore, in one or more implementations, generating replacement pixels involves generating replacement pixels from an image encoder using a cascaded modulation decoder layer that utilizes a class-specific cascaded modulation repair neural network.

[0225] Furthermore, a series of actions 1400 includes action 1406 of providing a repaired digital image including: replacing pixels such that the repaired digital image depicts an instance of a target object class. For example, in one or more embodiments, action 1406 includes providing a repaired digital image including replacing pixels for display via a client device such that the repaired digital image depicts an instance of a target object class within a replacement area.

[0226] In one or more implementations, a series of actions 1400 includes: receiving an instruction for a sky replacement region of a replacement digital image via a user interface of a client device; generating a plurality of sky replacement pixels for the sky replacement region using a class-specific (cascaded modulation) inpainting neural network trained to generate sky regions for the digital image; and providing the inpainted digital image, including the plurality of sky replacement pixels within the sky replacement region, for display via the client device.

[0227] For example, in one or more implementations, a series of actions 1400 includes determining a sky replacement region from a digital image using a segmentation model. Furthermore, in one or more embodiments, the series of actions 1400 includes selecting a specific class (cascaded modulation) repair neural network trained to generate the sky region from a plurality of specific class (cascaded modulation) repair neural networks based on an indication of a sky replacement region replacement.

[0228] Furthermore, in some implementations, a series of actions 1400 includes generating sky-replacement pixels from image encoding using a cascaded modulation decoder layer of a class-specific cascaded modulation repair neural network. Additionally, in one or more implementations, generating sky-replacement pixels includes generating positional codes corresponding to different resolutions of the cascaded modulation decoder layers.

[0229] Furthermore, in one or more implementations, a series of actions 1400 includes generating sky-replacement pixels by utilizing a cascaded modulation decoder layer of a class-specific cascaded modulation repair neural network, image encoding, and position encoding.

[0230] In some implementations, a series of actions 1400 includes different sets of actions (i.e., different from...). Figure 14 (The set of actions shown). For example, in some implementations, a series of actions 1400 includes: generating a class-segmented digital image by segmenting instances of an object class from a digital image; generating multiple predicted inscribed digital images for the object class from the class-segmented digital image using a (cascaded modulation) inscription neural network; and modifying the parameters of the (cascaded modulation) inscription neural network using a discriminator neural network and an adversarial loss to generate a class-specific (cascaded modulation) inscription neural network.

[0231] For example, in some embodiments, instances of segmenting an object class from multiple images include: determining a segmentation mask corresponding to the object class from a digital image; generating an extended segmentation mask from the segmentation mask using an extension operation; and segmenting the object class from the digital image using the extended segmentation mask.

[0232] Similarly, in some implementations, generating repaired digital images with multiple predictions for an object class involves generating image encodings using Fourier convolutional encoder layers of an encoder that utilizes a class-specific (cascaded modulation) repair neural network.

[0233] Furthermore, in one or more embodiments, generating multiple predictive inpainted digital images for an object class includes generating multiple predictive inpainted digital images from an image encoding using a cascaded modulation layer of a class-specific cascaded modulation inpainting neural network, wherein a given cascaded modulation layer includes a global modulation block and a spatial modulation block.

[0234] Furthermore, in some implementations, modifying the parameters of the (cascaded modulation) inpainting neural network to generate class-specific (cascaded modulation) inpainting neural networks includes: using a discriminator neural network to generate a realism prediction from a predicted inpainted digital image; and determining an adversarial loss based on the realism prediction.

[0235] In one or more embodiments, generating a class-segmented digital image includes segmenting an instance of one of the following classes from a digital image: a sky object class, a water object class, a ground object class, or a human object class.

[0236] Embodiments of this disclosure may include or utilize a dedicated or general-purpose computer including computer hardware, such as one or more processors and system memory, as discussed in more detail below. Embodiments within the scope of this disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more processes described herein may be implemented at least in part as instructions contained in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any media content access device described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., memory, etc.) and executes those instructions to perform one or more processes, including one or more of the processes described herein.

[0237] Computer-readable media can be any available medium that can be accessed by a general-purpose or special-purpose computer system. A computer-readable medium storing computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of this disclosure may include at least two distinct types of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0238] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drive (“SSD”) (e.g., RAM-based), flash memory, phase-change memory (“PCM”), other types of memory, other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0239] "Network" is defined as one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or other communication connection (wired, wireless, or a combination of wired and wireless), the computer correctly recognizes the connection as a transmission medium. Transmission media may include networks and / or data links that can be used to carry desired program code in the form of computer-executable instructions or data structures and are accessible by a general-purpose or special-purpose computer. Combinations of the foregoing should also be included within the scope of computer-readable media.

[0240] Furthermore, upon arrival at various computer system components, program code in the form of computer-executable instructions or data structures can be automatically transferred from the transmission medium to a non-transitory computer-readable storage medium (device) (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or a non-volatile computer storage medium (device) at the computer system. Therefore, it should be understood that a non-transitory computer-readable storage medium (device) can be included in computer system components that also (or even primarily) utilize the transmission medium.

[0241] Computer-executable instructions include, for example, instructions and data that, when executed at a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or group of functions. In some embodiments, the computer-executable instructions execute on a general-purpose computer to turn the general-purpose computer into a special-purpose computer that implements the elements of this disclosure. The computer-executable instructions may be, for example, binary, intermediate format instructions (e.g., assembly language), or even source code. Although the subject matter has been described in language specific to structural features and / or methodological behavior, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as exemplary forms for implementing the claims.

[0242] Those skilled in the art will understand that this disclosure can be implemented in network computing environments with a variety of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. This disclosure can also be practiced in distributed system environments, where both local and remote computer systems, linked via a network (via a hardwired data link, a wireless data link, or a combination of hardwired and wireless data links), perform tasks. In a distributed system environment, program modules can reside on both local and remote memory storage devices.

[0243] The embodiments of this disclosure can also be implemented in a cloud computing environment. In this specification, "cloud computing" is defined as a model for implementing on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be used in the market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly configured via virtualization and released with low management effort or service provider interaction, and then scaled accordingly.

[0244] Cloud computing models can be composed of various features, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, and measurement services. Cloud computing models can also expose various service models, such as Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). Cloud computing models can also be deployed using different deployment models, such as private clouds, community clouds, public clouds, and hybrid clouds. In this specification and claims, “cloud computing environment” means the environment in which cloud computing is employed.

[0245] Figure 15An example computing device 1500 (e.g., computing device 1000, client device 108, and / or / multiple servers 104) is illustrated in block diagram form. The computing device 1500 can be configured to perform one or more of the processes described above. It should be understood that a class-specific image restoration system 102 may include an implementation of the computing device 1500. Figure 15 As shown, the computing device may include a processor 1502, a memory 1504, a storage device 1506, an I / O interface 1508, and a communication interface 1510. Furthermore, the computing device 1500 may include input devices such as a touchscreen, a mouse, and in some embodiments, a touchscreen. Figure 15 The components shown are fewer or more components. A more detailed description will follow. Figure 15 The components of the computing device 1500 shown.

[0246] In a particular embodiment, processor(s) 1502 includes hardware for executing instructions, such as those that constitute a computer program. By way of example and not limitation, in order to execute instructions, processor(s) 1502 may retrieve (or fetch) instructions from internal registers, internal caches, memory 1504, or storage device 1506, and decode and execute the instructions.

[0247] Computing device 1500 includes memory 1504 coupled to processor(s) 1502. Memory 1504 can be used to store data, metadata, and programs executed by processor(s). Memory 1504 can include one or more types of memory, such as random access memory (“RAM”), read-only memory (“ROM”), solid-state drive (“SSD”), flash memory, phase-change memory (“PCM”), or other types of data storage devices. Memory 1504 can be internal or distributed memory.

[0248] Computing device 1500 includes storage device 1506, which includes memory for storing data or instructions. By way of example and not limitation, storage device 1506 may include the non-transitory storage medium described above. Storage device 1506 may include a hard disk drive (HDD), flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices.

[0249] The computing device 1500 also includes one or more input or output (“I / O”) devices / interfaces 1508, which are provided to allow a user to provide input (such as user strokes) to the computing device 1500, receive output from the computing device 1500, and otherwise transfer data to and from the computing device 1500. These I / O devices / interfaces 1508 may include a mouse, keypad or keyboard, touchscreen, camera, optical scanner, network interface, modem, other known I / O devices, or combinations of such I / O devices / interfaces 1508. The touchscreen can be activated using a writing device or a finger.

[0250] I / O device / interface 1508 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, device / interface 1508 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0251] The computing device 1500 may also include a communication interface 1510. The communication interface 1510 may include hardware, software, or both. The communication interface 1510 may provide one or more interfaces for communication (e.g., packet-based communication) between the computing device and one or more other computing devices 1500 or one or more networks. By way of example and not limitation, the communication interface 1510 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks such as Wi-Fi. The computing device 1500 may also include a bus 1512. The bus 1512 may include hardware, software, or both for coupling components of the computing device 1500 to each other.

[0252] In the foregoing description, the invention has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the invention have been described with reference to the details discussed herein, and various embodiments are illustrated in the accompanying drawings. The foregoing description and drawings are illustrative of the invention and should not be construed as limiting the invention. Numerous specific details have been described to provide a thorough understanding of various embodiments of the invention.

[0253] This invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects as illustrative rather than restrictive. For example, the methods described herein may be performed with fewer or more steps / actions, or the steps / actions may be performed in a different order. Furthermore, the steps / actions described herein may be repeated or performed in parallel with each other, or in parallel with different instances of the same or similar steps / actions. Therefore, the scope of this invention is indicated by the appended claims rather than by the foregoing description. All variations within the meaning and scope of the equivalent claims should be included within its scope.

Claims

1. A non-transitory computer-readable medium storing instructions thereon, the instructions, when executed by at least one processor, causing the at least one processor to perform an operation, the operation comprising: The replacement area and target object class of the digital image are received via the user interface of the client device; Replacement pixels for the replacement region are generated using a class-specific repair neural network corresponding to the target object class, the class-specific repair neural network having multiple cascaded modulation layers, wherein a given cascaded modulation layer includes a global modulation block and a spatial modulation block: Using the global modulation block of the multiple cascaded modulation layers, modulation is applied based on global feature codes to capture global prediction; as well as Using the spatial modulation block of the multiple cascaded modulation layers, spatial modulation is applied to refine the global prediction; as well as A repaired digital image, including the replacement pixels, is provided for display via the client device, such that the repaired digital image depicts an instance of the target object class within the replacement region.

2. The non-transitory computer-readable medium of claim 1, wherein receiving the instruction regarding the replacement region and the target object class comprises: The digital image is provided for display via the user interface; as well as The user selects the replacement area via the user interface using a selection tool corresponding to the target object class.

3. The non-transitory computer-readable medium according to claim 2, further comprising: The replacement region is determined using a segmentation model and the user's selection.

4. The non-transitory computer-readable medium of claim 1, further comprising instructions, which, when executed by the at least one processor, cause the at least one processor to perform an operation, the operation comprising: A global feature map is generated using the global modulation block, so that modulation can be applied based on the global feature code; as well as The spatial modulation block is used to perform spatial modulation using the global feature map.

5. The non-transitory computer-readable medium of claim 1, further comprising instructions, which, when executed by the at least one processor, cause the at least one processor to perform an operation, the operation comprising: The replacement pixels are generated using a class-specific repair neural network corresponding to at least one of the following: sky object class, water object class, ground object class, or human object class.

6. The non-transitory computer-readable medium of claim 1, wherein generating the replacement pixel using the class-specific repair neural network comprises: Image encoding is generated by repairing the encoder layer of a neural network using class-specific cascaded modulation.

7. The non-transitory computer-readable medium of claim 6, wherein generating the image encoding using the encoder layer of the class-specific cascaded modulation insulator neural network comprises: Generate position codes corresponding to different resolutions of the encoder layer; as well as Multiple encoded feature vectors are generated using the encoder layer and the position encoding.

8. The non-transitory computer-readable medium of claim 6, wherein generating the replacement pixel comprises: The replacement pixels are generated using a cascaded modulation decoder layer of the class-specific cascaded modulation insufflation neural network derived from the image encoding.

9. A computer-implemented method, comprising: The interaction between the user and the digital image is received via the user interface of the client device, the interaction including an instruction to replace the sky replacement area of ​​the digital image; Using a panoramic segmentation model, the sky target object class is determined based on the indication of the sky replacement region in the replaced digital image; A class-specific cascaded modulation insulation neural network is selected based on the indicated sky target object class, the class-specific cascaded modulation insulation neural network being trained to generate sky regions for digital images; Sky-replaced pixels for the sky-replaced region are generated by utilizing the class-specific cascaded modulation inpainting neural network trained to generate sky regions for digital images, wherein the class-specific cascaded modulation inpainting neural network has multiple cascaded modulation layers, wherein a given cascaded modulation layer includes a global modulation block and a spatial modulation block: Using the global modulation block of the multiple cascaded modulation layers, modulation is applied based on global feature codes to capture global prediction; as well as Using the spatial modulation block of the multiple cascaded modulation layers, spatial modulation is applied to refine the global prediction; as well as A restored digital image, including the sky-replacement pixels within the sky-replacement area, is provided for display via the client device.

10. The computer-implemented method according to claim 9, further comprising: The sky replacement area is determined based on the portion of the digital image selected by the user input.

11. The computer-implemented method according to claim 9, further comprising: Based on the instruction to replace the sky replacement region, the class-specific cascaded modulation repair neural network trained to generate the sky region is selected from a plurality of class-specific cascaded modulation repair neural networks.

12. The computer-implemented method according to claim 9, further comprising: The sky replacement pixels are generated from the image encoding using the cascaded modulation decoder layer of the class-specific cascaded modulation repair neural network.

13. The computer-implemented method of claim 12, wherein generating the sky replacement pixels comprises: Generate position codes corresponding to the different resolutions of the cascaded modulation decoder layers.

14. The computer-implemented method according to claim 13, further comprising: The sky replacement pixels are generated using the cascaded modulation decoder layer, the image encoding, and the position encoding of the class-specific cascaded modulation repair neural network.

15. A system comprising: One or more memory devices; as well as One or more processors are configured to cause the system to: The replacement area and target object class of the digital image are received via the user interface of the client device; Replacement pixels for the replacement region are generated using a class-specific repair neural network corresponding to the target object class, the class-specific repair neural network having multiple cascaded modulation layers, wherein a given cascaded modulation layer includes a global modulation block and a spatial modulation block: Using the global modulation block of the multiple cascaded modulation layers, modulation is applied based on global feature codes to capture global prediction; as well as Using the spatial modulation block of the multiple cascaded modulation layers, spatial modulation is applied to refine the global prediction; as well as A repaired digital image, including the replacement pixels, is provided for display via the client device, such that the repaired digital image depicts an instance of the target object class within the replacement area.

16. The system of claim 15, wherein receiving the instruction of the replacement region and the target object class comprises: The digital image is provided for display via the user interface; as well as The user selects the replacement area via the user interface using a selection tool corresponding to the target object class.

17. The system of claim 16, wherein the one or more processors are further configured to cause the system to determine the replacement region using a segmentation model and the user selection.

18. The system of claim 15, wherein the one or more processors are further configured to cause the system to: A global feature map is generated using the global modulation block to apply modulation based on the global feature code; and The spatial modulation block is used to perform spatial modulation using the global feature map.

19. The system of claim 15, wherein generating the replacement pixel using the class-specific repair neural network comprises: Image encoding is generated by repairing the encoder layer of a neural network using class-specific cascaded modulation.

20. The system of claim 19, wherein generating the image code using the encoder layer of the class-specific cascaded modulation repair neural network comprises: Generate position codes corresponding to different resolutions of the encoder layer; as well as Multiple encoded feature vectors are generated using the encoder layer and the position encoding.

Images