AGILEGAN-based stylization method for amplifying style area

Abstract

The disclosed methods and systems for magnifying a stylized region of an image include receiving an input image, generating a first stylized image based on the input image using a first generative adversarial network (GAN) generator, normalizing the input image, and generating a second stylized image based on the first stylized image using a second generative adversarial network (GAN) generator. And generating a second stylized image based on the normalized input image generator using a second generative adversarial network (GAN), mixing the first stylized image and the second stylized image to obtain a third stylized image, and providing the third stylized image as output.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application No. 17 / 807,527, filed on June 17, 2022, entitled AGILEGAN-BASED STYLIZATION METHOD TO ENLARGE A STYLE REGION, the entire contents of which are incorporated herein by reference in their entirety. Background Art

[0003] Portraiture, the art of depicting a subject's appearance, is an important art form dating back to the dawn of civilization. It has evolved beyond realistic depictions to more creative interpretations, with a variety of styles such as abstract art, cubism, and cartoons. Automatically stylized portraiture has experienced rapid development in recent years due to advances in deep learning. Early methods involving neural style convincingly demonstrated the ability to transfer texture style from sample source to target images, transforming real photos into paintings by Van Gogh or Picasso. However, when it comes to portraiture, these methods have largely failed to capture the important geometric dependencies of different portrait styles, and thus lack in stylization quality.

[0004] Image-to-image translation methods were later introduced to “convert” images from a source domain to a target domain, either in a supervised manner using paired datasets or in an unsupervised setting using unpaired datasets. These methods have been explored for portrait stylization, e.g., self-to-anime and cartoons. AgileGAN is a state-of-the-art framework that generates high-quality style portraits via inversion-consistent transfer learning, while providing greater flexibility in creating high-quality and high-resolution (1024×1024) portrait stylization models, requiring only a limited number of style samples (~100), and can be trained in a short time (~1 hour).

[0005] However, using the AgileGAN-based portrait stylization workflow, the generated image is stylized in a region that is fixed and limited to the user's face. For example, when a user uploads an input image, the system needs to crop and / or otherwise normalize the input image to generate a corresponding output image with a style region. Typically, the style region is relatively smaller than the original input image and only covers part of the user's hair region and / or neck region (if any). Therefore, an AgileGAN-based stylization method is needed to enlarge the style region of the image.

[0006] With respect to these and other general considerations, embodiments have been described.Although relatively specific problems have been discussed, the examples described herein should not be limited to solving the specific problems identified in the background above. Summary of the invention

[0007] According to examples of the present disclosure, a refinement method for GAN is described, and more specifically, a portrait stylization workflow based on AgileGAN is described. In an example, after training a stylized generator using, for example, the AgileGAN method, a post-processing step may be included to enlarge the stylized region of the image. An advantage of the present disclosure may be to enlarge the stylized region of the image to include hair regions, neck regions, headwear, or other accessories that may not be fully stylized otherwise.

[0008] In some examples, a method for enlarging a style region of an image is disclosed. The method includes receiving an input image; using a first generative adversarial network (GAN) generator, generating a first stylized image based on the input image; normalizing the input image; using a second generative adversarial network (GAN) generator, generating a second stylized image based on the normalized input image; mixing the first stylized image and the second stylized image to obtain a third stylized image; and providing the third stylized image as an output.

[0009] In some examples, the method further includes extracting a first face parsing mask from the first stylized image and extracting a second face parsing mask from the second stylized image. The blending of the first stylized image and the second stylized image is based on the first face parsing mask and the second face parsing mask.

[0010] In some examples, the mixture of the first stylized image and the second stylized image comprises a Gaussian mixture.

[0011] In some examples, the third stylized image includes a first set of pixels corresponding to a hair region and a neck region of the first stylized image, and a second set of pixels corresponding to a face region of the second stylized image.

[0012] In some examples, normalization includes one or more of cropping or scaling the input image.

[0013] In some examples, the first GAN generator is different from the second GAN generator.

[0014] In some examples, the first GAN generator and the second GAN generator are AgileGAN generators.

[0015] In some examples, the method also includes receiving a plurality of sample style images, and training the first GAN generator and the second GAN generator using transfer learning based on the received plurality of sample style images.

[0016] In some examples, a system is disclosed. The system includes at least one processor and a memory storing instructions, and the instructions, when executed by the at least one processor, cause the system to perform a set of operations. The set of operations implements any method described herein.

[0017] In some examples, a non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium includes instructions executable by one or more processors to cause the one or more processors to implement any of the methods described herein.

[0018] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0020] Figure 1 An example of a data flow process for enlarging a style region of an image according to an example of the present disclosure is described.

[0021] Figure 2 Instructions along the Figure 1 Examples of input images and corresponding stylized images for various parts of the data flow process.

[0022] Figure 3 Details of an example method of enlarging a style region of an image according to some examples of the present disclosure are described.

[0023] Figure 4 A block diagram is provided to illustrate the physical components (eg, hardware) of a computing system with which aspects of the present disclosure may be practiced and according to examples of the present disclosure.

[0024] Figure 5 One aspect of a system architecture for processing data received at a computing system from a remote source according to examples of the present disclosure is described. DETAILED DESCRIPTION

[0025] In the following detailed description, reference is made to the accompanying drawings forming part thereof, and specific embodiments or examples are shown by way of illustration in the accompanying drawings. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. The embodiments may be practiced as methods, systems, or devices. Therefore, the embodiments may take the form of hardware implementations, complete software implementations, or implementations combining software and hardware aspects. Therefore, the following detailed description should not be regarded as restrictive, and the scope of the present disclosure is limited by the appended claims and their equivalents.

[0026] Stylizing facial images in an artistic way has been explored in the context of non-photorealistic rendering. Early methods relied on low-level histogram matching using linear filters. Neural style transfer produced early exciting results via deep learning by matching feature statistics in convolutional layers. Since then, several improvements have been proposed that aim to implement local patterns in deep feature space via Markov random fields (MRFs), extend style transfer to video, and improve quality by imposing temporal constraints. While these methods can achieve generally convincing results for several artistic styles, they generally fail in styles that involve significant geometric deformation of facial features, such as cartooning. For more general stylization, image-to-image (I2I) conversion can be used to convert an input image from a source domain to a target domain.

[0027] Conditional generative adversarial networks (GANs) can be implemented to learn input-to-output mappings. Similar ideas have been applied to various tasks, such as applying sketches to photos and applying attributes to images. For example, the well-known cycle consistency loss in CycleGAN has been proposed to improve the network training stability of unpaired settings. Unsupervised methods have also been used for cartooning. In addition, CycleGAN has been extended to cross-domain anime portrait generation, and other unsupervised methods have incorporated attention modules and learnable normalization functions for cartoon face generation, where their attention-guided models can flexibly control the amount of change in shape and texture. GANs have been used to synthesize images that ideally match the distribution of training data sets via adversarial training. GANs have been applied to various fields, including but not limited to image restoration, image manipulation, and texture synthesis. Various advances have been used to improve the architecture, synthesis quality, and training stability of GANs.

[0028] Since GANs are typically designed to generate realistic images by sampling from a known distribution in a latent space, GAN inversion solves the complementary problem of finding the most accurate latent code that will reconstruct an image given an input image. One approach is optimization-based, which directly optimizes the latent code to minimize the pixel-by-pixel reconstruction loss of a single input instance. Another approach is learning-based, in which a deterministic model is trained by minimizing the difference between the input and synthesized images. Other work combines these optimization and learning-based methods by learning an encoder that produces a good initialization for subsequent optimization. In addition to image reconstruction, some examples also use inversion when performing image manipulation. For example, a hybrid approach can encode an image into a semantically actionable domain for image editing. In addition, a general Pixel2Style2Pixel (PSP) encoder has been proposed; such an encoder is based on a dedicated identity loss for embedding images in several real image conversion tasks such as restoration and super-resolution.

[0029] As mentioned above, when a user uploads an input image, the system needs to crop and / or otherwise normalize the input image to generate a corresponding output image with a style region. Typically, the style region is relatively smaller than the original input image and only covers part of the user's hair region and / or neck region (if any). Therefore, an AgileGAN-based stylization method is needed to enlarge the style region of the image to include the entire hair region, neck region, headwear, and / or other accessories that may not otherwise be properly stylized.

[0030] According to an example of the present disclosure, a method for enlarging a style region of an image is described, and more specifically, a portrait stylization workflow based on AgileGAN is described. In an example, after training a stylized generator using, for example, an AgileGAN method, a post-processing step may be included to enlarge the style region to include a hair region, a neck region, a headdress, and / or other accessories that may not be stylized in other ways. The method may include receiving an input image, generating a first stylized image based on the input image, normalizing the input image, generating a second stylized image based on the normalized input image, mixing the first stylized image and the second stylized image together to obtain a third stylized image, the third stylized image including an enlarged style region relative to the second stylized image, and providing the third stylized image as an output. At least one benefit of using such a method may include ensuring that the output stylized image is enlarged to include the entire hair region, neck region, headdress, and / or other accessories.

[0031] Figure 1An example of a data flow process 100 for obtaining a generated image having a texture consistent with a style sample image using a latent code mixing operation according to an example of the present disclosure is depicted. In an example, during a first operation 102, one or more GAN generators 104 (such as a first GAN generator 104a and a second GAN generator 104b), and more specifically one or more AgileGAN generators, may be trained using a plurality of images depicting a desired style.

[0032] The starting baseline for training the AgileGAN generator includes using a pre-trained GAN model (e.g., a StyleGAN-based model and / or a StyleGAN2 model). The pre-trained GAN model (e.g., a StyleGAN-based model and / or a StyleGAN2 model) exhibits the property that if a random sample from a Gaussian distribution in the Z latent space is collected, the model can generate an image that fits the original training distribution (e.g., the original training distribution of the data set). Training the AgileGAN model can include two stages. The task of training involves using an image as input; therefore, determining the corresponding latent vector of the GAN model (e.g., a StyleGAN-based model and / or a StyleGAN2 model). A front-end encoder such as a hierarchical variational encoder is trained to map an input image (e.g., sample image 106) to a latent space while keeping the back-end GAN generator fixed. In a second process, starting from a copy of a pre-trained GAN model (e.g., a StyleGAN-based model and / or a StyleGAN2 model), the pre-trained GAN model (e.g., a StyleGAN-based model and / or a StyleGAN2 model) is fine-tuned so that samples from a Gaussian distribution in a latent space can generate images that better fit the style sample.

[0033] In the example, the two training stages are performed independently and can be trained in parallel. However, structurally, the two training stages share the pivot latent space and are also jointly anchored by the fixed GAN generator 104. As previously mentioned, by separating the inversion training and image generation training into two stages, the training of the AgileGAN model does not require a paired dataset; therefore, the separation of training also enables higher resolution by reducing the computational load and making the backpropagation process more effective and efficient. Instead of fine-tuning the architecture of the AgileGAN model, new style domains can be incorporated by fine-tuning the generator.

[0034] Conventional autoencoders generally lack the ability to generate new images because the resulting latent space is discontinuous. In order to force the autoencoder utilized during training the GAN generator 104 to generate a continuous latent space, an output vector of a mean and an output vector of a standard deviation are utilized. Training the hierarchical variational encoder includes optimizing the Kullback-Leibler divergence (e.g., a mean close to 0 and a standard deviation close to 1) as well as image reconstruction and other losses that may depend on the mean and standard deviation. The standard deviation and the mean may be sampled to generate a latent z vector corresponding to one of the multiple input images. While a typical variational autoencoder includes an encoder εθ and a decoder Gφ (e.g., the GAN generator 104) with respective parameters θ and φ, which are jointly trained to minimize the reconstruction error between an input image x (e.g., an image of the plurality of training images 106) and an output image x (e.g., an image generated by the GAN generator 104), the hVAE for inversion uses a fixed original pre-trained GAN model (e.g., a StyleGAN-based model and / or a StyleGAN2 model) as a decoder Gφo (e.g., the GAN generator 418) and the hVAE is trained to learn the posterior distribution q(z|x).

[0035] Additional details for training the GAN generator 104 and the hierarchical variational autoencoders may be found in U.S. patent application serial number 17 / 321,384, entitled “A HIGH-RESOLUTION PORTRAIT STYLIZATION FRAMEWORKS USING A HIERARCHICAL VARIATIONAL ENCODER” and / or U.S. patent application serial number 17,534,631, entitled “AGILEGAN-BASED REFINEMENT METHOD AND FRAMEWORK FOR CONSISTENT TEXTURE GENERATION,” the contents of which are incorporated herein by reference for their teachings and all purposes.

[0036] During the second operation 108 , the input image 110 may be inserted into the first GAN generator 104a . The first GAN generator 104a may be an AgileGAN generator trained using the method discussed above. The first GAN generator 104a generates a first stylized image 114a based on the input image 110 .

[0037] Also during the second operation 108, the input image 110 can be normalized, for example, by a normalizer engine or component 112. The normalizer component 112 can be trained or otherwise configured to crop and / or scale the input image 110. For example, the normalizer component 112 can align the eye position within a pre-specified area or on a pre-specified size to center the eye position. In addition, the normalizer component 112 can crop the input image 110 to a desired size, such as a square-sized image of, for example, 1024×1024. Additionally or alternatively, the normalizer 112 can enlarge or reduce the size of the input image 110.

[0038] The input image 110 may be input to the second GAN generator 104b after being normalized by the normalizer component 112. The second GAN generator 104b may be the same as the first GAN generator 104a (e.g., they are trained using the same or similar process and data set). Alternatively, the second GAN generator 104b may be different from the first GAN generator 104a (e.g., the second GAN generator 104b may be trained using a different training process and / or a different data set). After the input image 110 is normalized, the second GAN generator 104b generates a second stylized image 114b based on the input image 110.

[0039] Each of the first stylized image 114a and the second stylized image 114b may be input into a corresponding extractor engine or component 116a, 116b. The extractor components 116a and / or 116b may each be trained or otherwise configured to extract one or more face parsing masks 118a, 118b from the corresponding first stylized image 114a or the second stylized image 114b. The masks 118a, 118b may correspond to one or more features of the corresponding first stylized image 114a or the second stylized image 114b that have been stylized. For example, each of the first stylized image 114a or the second stylized image 114b may include a corresponding hair region 120a, 120b and a neck region 122a, 122b. The hair region 120b and / or the neck region 122b of the mask 118b may be smaller than the hair region 120a and / or the neck region 122a of the mask 118a. Thus, while the normalization component 112 may allow for more accurate and / or detailed stylization of facial portions of the input image 110, the corresponding mask 118b may lack stylization of the neck and / or hair portions, as shown in the neck region 122b and the hair region 120b.

[0040] The extractor components 116 and / or 116b may be trained or otherwise configured to perform semantic image segmentation, wherein features of a user's face are trained to be identified based on traditional landmarks (e.g., bone structure, contrast, expected location of features, etc.). The extractor components 116 and / or 116b may include one or more machine learning models, such as a neural network, that are trained with information about semantic facial regions of an image at the pixel level.

[0041] The masks 118a and 118b corresponding to the first stylized image 114a and the second stylized image 114b, respectively, may be input into the selective mixer engine or component 124 to generate the third stylized image 126. The third stylized image 126 may be obtained, for example, from mixing the first stylized image 114a and the second stylized image 114b together based on the first mask 120a and the second mask 120b. Specifically, the second stylized image 114b (e.g., the stylized area of ​​the input image 110) may be enlarged via the selective mixer component 124 to include a portion of the first stylized image 114a. The portion of the first stylized image 114a that is mixed with the second stylized image 114b may be a portion of the hair region 120a and a portion of the neck region 122a corresponding to the mask 118a. As discussed previously herein, the hair region 120a and the neck region 122a may be larger than the hair region 120b and the neck region 120b; therefore, blending the portions of the first stylized image 114a corresponding to the hair region 120a and the neck region 122a will enlarge the stylized region of the second stylized image 114b.

[0042] The selective mixer component 124 may mix the first stylized image 114a and the second stylized image 114b together to generate or otherwise obtain the third stylized image 126. The selective mixer component 124 may perform Gaussian mixing, Gaussian blurring, or Gaussian smoothing to mix (e.g., convolve) the first stylized image 114a and the second stylized image 114b together. For example, one or more boundaries of the first stylized image 114b may be defined along the top and bottom of the mask 118b. A first set of pixels corresponding to the hair region 120a and the neck region 122a of the first stylized image 114a may be mixed with a second set of pixels corresponding to the facial region of the second stylized image 114b (e.g., between the neck region 122b and the hair region 120b). The first set of pixels and the second set of pixels may be mixed (e.g., convoluted) along one or more boundaries of the first stylized image 114b. One of ordinary skill in the art may recognize additional or alternative methods for mixing images together.

[0043] According to some examples of the present disclosure, it is expected that the hair regions 120a, 120b may include headwear or other accessories (e.g., necklaces, earrings, scarves, etc.). In such examples, the stylized region of the second stylized image 114 may be enlarged to include a stylized version of the headwear or other accessories, such as by Figure 2 As shown in the examples provided, and otherwise understood by one of ordinary skill in the art.

[0044] The third stylized image 126 may be provided as an output. For example, the third stylized image 126 may be provided as an output to a computing device, such as a computer program product described later herein. Figure 4 The computing device 400 discussed above. After the third stylized image 126 is output, it may be displayed, such as on a display screen or via a user interface. Additionally or alternatively, the third stylized image may be output for further processing according to methods or systems known to those of ordinary skill in the art and / or learned from experimentation with the systems and methods disclosed herein.

[0045] Typically, the selective mixer component 124 allows for enlarging the stylized region of the second stylized image 114b so that the hair and neck portions of the image are additionally and / or completely included in the stylization. The mechanisms disclosed herein facilitate generating enlarged stylized regions that include components of an input image such as hair, neck, headwear, etc.

[0046] Figure 2 Instructions along the Figure 1 1 and 2. The example grid 200 includes a first input image 202a, a second input image 202b, and a third input image 202c. The first input image 202a, the second input image 202b, and the third input image 202c may be similar to the reference image 200. Figure 1 Described input image 110. For example, the first input image 202a, the second input image 202b, and the third input image 202c may each have a hair region, a neck region, and a face region between the hair region and the neck region.

[0047] The first input image 202a, the second input image 202b, and the third input image 202c may be normalized (e.g., via the normalizer component 112) and inserted into a GAN generator (e.g., the GAN generator 104b). Subsequently, a stylized region may be generated for each of the input images 202a, 202b, and 202c. The example grid 200 includes a first input image 202a having a first stylized region 204a, a second input image 202b having a second stylized region 204b, and a third input image 202c having a third stylized region 204c.

[0048] As shown with respect to the images 202a, 202b, and 202c having stylized regions 204a, 204b, and 204c, the stylized regions 204a, 204b, and 204c do not extend completely through the person's hair, neck, and headwear in each of the respective input images 202a, 202b, and 202c. With specific reference to the first input image 202a having the first stylized region 204a, the neck region of the person in the first input image 202a is not completely stylized. Additionally, the user's headwear is not completely stylized. With specific reference to the second input image 202b having the second stylized region 204b, the neck region of the person in the second input image 202b is not completely stylized. Additionally, with specific reference to the third input image 202c having the third stylized region 204c, the neck region and hair region of the person in the third input image 202c are not completely stylized. Therefore, the stylized regions 204a, 204b, and 204c of the input images 202a, 202b, and 202c need to be enlarged.

[0049] According to the mechanisms disclosed herein, the stylized regions 204a, 204b, and 204c may be enlarged or extended via post-processing. For example, the input images 202a, 202b, and 202c may be input, such as directly input, into a GAN generator (e.g., GAN generator 104a) without normalization. Subsequently, a stylized region may be generated for each of the first input image 202a, the second input image 202b, and the third input image 202c, as shown in FIG. Figure 1 In addition, the stylized regions may be blended with the stylized regions 204a, 204b, and 204c, respectively (e.g., via the selective mixer component 124) to create enlarged stylized regions. For example, the example grid 200 includes a first input image 202a having a first enlarged stylized region 206a, a second input image 202b having a second enlarged stylized region 206b, and a third input image 202c having a third enlarged stylized region 206c. The first enlarged stylized region 206a, the second enlarged stylized region 206b, and the third enlarged stylized region 206c may be generated or otherwise obtained via selective blending between the first stylized region based on the input image and the second stylized region based on the normalized input image.

[0050] As shown with respect to the images 202a, 202b, and 202c having the enlarged stylized regions 206a, 206b, and 206c, the enlarged stylized regions 206a, 206b, and 206c extend completely through the person's hair, neck, and headwear in each of the respective input images 202a, 202b, and 202c. With specific reference to the first input image 202a having the first stylized region 204a, the neck region of the person in the first input image 202a is completely stylized. Additionally, the user's headwear is completely stylized. With specific reference to the second input image 202b having the second stylized region 204b, the neck region and hair region of the person in the second input image 202b are completely stylized. Additionally, with specific reference to the third input image 202c having the third stylized region 204c, the neck region and hair region of the person in the third input image 202c are completely stylized. Thus, the mechanisms disclosed herein facilitate application of post-processing methods that enlarge the stylized region to include the entire neck region, hair region, headwear, and / or other accessories (e.g., necklaces, earrings, scarves, etc.) present in the input image.

[0051] Figure 3 Depicted is an exemplary method 300 for enlarging a stylistic region of an image according to some examples of the present disclosure. Figure 3 300. Generally, the method 300 begins at 302 and ends at 316. The method 300 may include more or fewer steps, or may differ from Figure 3 The steps shown are arranged in the order of the steps. Method 300 may be performed as a set of computer executable instructions executed by a computer system and encoded or stored on a computer readable medium. In an example, aspects of method 300 are performed by one or more processing devices such as a computer or server. In addition, method 300 may be performed by gates or circuits associated with a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SOC), a neural processing unit, or other hardware device. Hereinafter, reference will be made to the method 300 in conjunction with the method 300. Figure 1 , Figure 2 , Figure 4 and Figure 5 The described systems, components, modules, software, data structures, user interfaces, examples, etc. are used to explain the method 300.

[0052] Method 300 begins at 302, where flow may proceed to 304. At 304, an input image may be received. For example, a user may provide an input image and / or the system may select an input image from a storage location. The input image may correspond to input image 110 ( Figure 1 ) and / or input images 202a, 202b and / or 202c ( Figure 2). The input image may be provided by a user. Additionally or alternatively, the input image may be generated or otherwise obtained by a computing device (e.g., computing device 400). For example, a user may capture an image using a camera of a computing device, and the input image may be generated based on visual data received by the camera.

[0053] The process proceeds to 306, where a first stylized image is generated based on the input image received at 304. The first stylized image may be generated using a first generative adversarial network generator (GAN) such as Figure 1 As described above, the first GAN may be an AgileGAN trained using a plurality of images depicting a desired style.

[0054] Flow proceeds to operation 308, where the input image is normalized (e.g., via Figure 1 Normalization may include one or more of cropping or scaling the input image. For example, an image of a face may be cropped and / or scaled to the center of the eyes of the face within a predetermined region. Additionally or alternatively, scaling of the input image may include enlarging or reducing the input image to match a predetermined format for input into a generative adversarial network generator.

[0055] The process proceeds to operation 310, where a second stylized image is generated based on the normalized input image. The second stylized image may be similar to Figure 2 The second stylized image can be generated using a second generative adversarial network generator (GAN) such as Figure 1 The second GAN 104b is generated by the second GAN 104b. As described previously herein, the second GAN can be an AgileGAN trained using a plurality of images depicting a desired style. In some examples, the second GAN can be similar or identical to the first GAN of operation 306. Alternatively, in some examples, the second GAN can be different from the first GAN of operation 306. For example, the second GAN can be trained using a process and / or data set that is different from the process and / or data set on which the first GAN was trained.

[0056] The process proceeds to operation 312, where the first stylized image and the second stylized image are blended to obtain a third stylized image. The third stylized image may include an enlarged stylized region relative to the second stylized region. The first stylized image and the second stylized image may be blended using a selective blender such as Figure 1The first stylized image and the second stylized image may be blended via a Gaussian blend, a Gaussian blur, or a Gaussian smoothing to blend (e.g., convolve) the first stylized image and the second stylized image together. For example, one or more boundaries of the first stylized image may be defined along the top and bottom of the extracted mask. A first set of pixels corresponding to a hair region and a neck region of the first stylized image may be blended with a second set of pixels corresponding to a facial region of the second stylized image (e.g., between the neck region and the hair region). The first set of pixels and the second set of pixels may be blended (e.g., convoluted) along one or more boundaries of the first stylized image. One of ordinary skill in the art may recognize additional or alternative methods for blending images together.

[0057] Flow proceeds to operation 314, where the third stylized image is provided as an output. For example, the third stylized image 126 may be provided as an output to a computing device, such as a computer program product described later herein. Figure 4 The computing device 400 discussed above. After the third stylized image 126 is output, it may be displayed, such as on a display screen or via a user interface. Additionally or alternatively, the third stylized image may be output for further processing according to methods or systems known to those of ordinary skill in the art and / or learned from experimentation with the systems and methods disclosed herein.

[0058] Typically, blending the first stylized image and the second stylized image allows for enlarging the stylized region of the second stylized image so that a hair portion, a neck portion, and / or a headdress or other accessory of the image are additionally and / or completely included in the stylization. The mechanisms disclosed herein facilitate generating an enlarged stylized region that includes components of an input image, such as hair, neck, headdress, accessories, etc.

[0059] Method 300 may end at 316. Alternatively, method 300 may return to operation 302 to create an iterative process of receiving an input image and providing an output image having an enlarged style region according to aspects described herein.

[0060] Figure 4 400 . A block diagram is provided to illustrate the physical components (e.g., hardware) of a computing system 400 with which aspects of the present disclosure may be practiced. The computing device components described below may be suitable for the computing and / or processing devices described above. In a basic configuration, the computing system 400 may include at least one processing unit 402 and a system memory 404. Depending on the configuration and type of the computing device, the system memory 404 may include, but is not limited to, volatile storage devices (e.g., random access memory (RAM)), non-volatile storage devices (e.g., read-only memory (ROM)), flash memory, or any combination of such memories.

[0061] The system memory 404 may include an operating system 405 and one or more program modules 406 suitable for running software applications 420, such as one or more components supported by the system described herein. As an example, the system memory 404 may include a first GAN generator 421, a second GAN generator 422, a normalizer 423, an extractor 424, and a selective mixer 425. The first GAN generator 421 and the second GAN generator 422 may be connected to the first GAN generator 104a and the second GAN generator 104b ( Figure 1 ) is the same or similar and may refer to the AgileGAN generator as previously described. Normalizer 423 may be the same as normalizer 112 ( Figure 1 ) is the same or similar. Extractor 424 may be the same as extractor 116a and / or 116b ( Figure 1 ) is the same or similar. The selective mixer 425 can be the same as the selective mixer 124 ( Figure 1 ) are the same or similar. For example, operating system 405 can be suitable for controlling the operation of computing system 400.

[0062] Furthermore, the examples of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application, and are not limited to any particular application or system. Figure 4 408. The computing system 400 may have additional features or functionality. For example, the computing system 400 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data. Figure 4 4 is illustrated by a removable storage device 409 and a non-removable storage device 410.

[0063] As described above, a number of program modules and data files may be stored in system memory 404. When executed on processing unit 402, program module 406 (e.g., software application 420) may perform processes including, but not limited to, the various aspects described herein. Other program modules may be used in accordance with various aspects of the present disclosure. In addition, examples of the present disclosure may be practiced in discrete electronic components of a circuit, a packaged or integrated electronic chip containing logic gates, a circuit utilizing a microprocessor, or on a single chip containing electronic components or a microprocessor. For example, embodiments of the present disclosure may be implemented via a processor or a processor. Figure 4Each or many of the components described in the embodiment may be integrated into a system on a chip (SOC) on a single integrated circuit for practice. Such SOC devices may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operated via the SOC, the functions described herein regarding the ability of the client to switch protocols may be operated via dedicated logic integrated with other components of the computing system 400 on a single integrated circuit (chip). Embodiments of the present disclosure may also be practiced using other technologies capable of performing logical operations (such as AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present disclosure may be practiced within a general-purpose computer or within any other circuit or system.

[0064] The computing system 400 may also have one or more input devices 412, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. The one or more input devices 412 may include an image sensor. Output device(s) 414 may also be included, such as a display, a speaker, a printer, etc. The above devices are examples and other devices may be used. Figure 4 As shown, computing system 400 may include one or more communication connections 416 that allow communication with other computing devices / systems 450. Examples of suitable communication connections 416 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports. Computing system 400 may represent one or more computing systems on which data flow process 100 and / or method 300 are implemented.

[0065] The term computer-readable media as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, or program modules. System memory 404, removable storage device 409, and non-removable storage device 410 are all examples of computer storage media (e.g., memory storage devices). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other article that can be used to store information and can be accessed by the computing system 400. Any such computer storage media may be part of the computing system 400. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0066] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0067] Figure 5 One aspect of a system architecture for processing data received at a computing system from a remote source such as a personal computer 504, a tablet computing device 506, or a mobile computing device 508 is illustrated. The personal computer 504, the tablet computing device 506, or the mobile computing device 508 may include one or more applications 520; such applications may include, but are not limited to, a first GAN generator 421, a second GAN generator 422, a normalizer 423, an extractor 424, and a selective mixer 425. The content at the server device 502 may be stored in different communication channels or other storage types. For example, a directory service, a web portal, a stylized image service, an instant messaging store, or a social networking service may be used to store various documents.

[0068] As described above, one or more of the aforementioned program modules 406 or software applications 420 may be employed by the server device 502 and / or the personal computer 504, the tablet computing device 506, or the mobile computing device 508. For example, the server device 502 may include the first GAN generator 421, the second GAN generator 422, the normalizer 423, the extractor 424, and the selective mixer 425 described previously.

[0069] The server device 502 can provide data to and from client computing devices such as a personal computer 504, a tablet computing device 506, and / or a mobile computing device 508 (e.g., a smartphone) over a network 515. By way of example, the computer system described above can be embodied in a personal computer 504, a tablet computing device 506, and / or a mobile computing device 508 (e.g., a smartphone). In addition to receiving graphics data that can be used for pre-processing at a graphics originating system or post-processing at a receiving computing system, any of these examples of computing devices can obtain content from a repository 516.

[0070] In addition, the aspects and functions described herein may be operated on a distributed system (e.g., a cloud-based computing system), where application functions, memory, data storage and retrieval, and various processing functions may be remotely operated from one another on a distributed computing network (such as the Internet or an intranet). The user interface and various types of information may be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, the user interface and various types of information may be displayed and interacted with on a wall onto which the user interface and various types of information are projected. Interaction with multiple computing systems through which embodiments of the present invention may be practiced includes key input, touch screen input, voice or other audio input, gesture input where the associated computing device is equipped with a detection (e.g., camera) function for capturing and interpreting user gestures to control the functions of the computing device, and the like.

[0071] In addition, the aspects and functions described herein can be operated on a distributed system (e.g., a cloud-based computing system), where application functions, memory, data storage and retrieval, and various processing functions can be remotely operated from each other on a distributed computing network (such as the Internet or an intranet). The user interface and various types of information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, the user interface and various types of information can be displayed and interacted with on a wall onto which the user interface and various types of information are projected. Interaction with multiple computing systems through which embodiments of the present invention can be practiced includes key input, touch screen input, voice or other audio input, gesture input where the associated computing device is equipped with a detection (e.g., camera) function for capturing and interpreting user gestures to control the functions of the computing device, and the like.

[0072] For example, the above text describes various aspects of the present disclosure with reference to the block diagrams and / or operational descriptions of the methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions marked in the boxes may not occur in the order shown in any flow chart. For example, two boxes shown in succession may actually be executed substantially simultaneously, or the boxes may sometimes be executed in the opposite order, depending on the functions / actions involved.

[0073] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the present disclosure in any way. The various aspects, examples and details provided in this application are considered to be sufficient to convey the best mode of possession and enable others to make and use the claimed disclosure. The claimed disclosure should not be interpreted as being limited to any aspect, example or detail provided in this application. Whether combined or shown and described individually, it is intended to selectively include or omit various features (of structure and method) to produce an embodiment with a specific feature set. The description and illustration of this application have been provided, and those skilled in the art can envision the changes, modifications and alternative aspects that fall within the spirit of the broader aspects of the overall inventive concept embodied in this application, which do not depart from the broader scope of the claimed disclosure.

Claims

1. A method for enlarging a style region of an image, the method comprising: receiving an input image; Using a first generative adversarial network (GAN) generator, generate a first stylized image based on the input image; Normalizing the input image; Using a second generative adversarial network (GAN) generator, generate a second stylized image based on the normalized input image; Mixing the first stylized image and the second stylized image to obtain a third stylized image; as well as The third stylized image is provided as output.

2. The method according to claim 1, further comprising: extracting a first facial parsing mask from the first stylized image; as well as extracting a second face parsing mask from the second stylized image, Wherein the blending of the first stylized image and the second stylized image is based on the first face parsing mask and the second face parsing mask. 3 . The method of claim 1 , wherein the mixture of the first stylized image and the second stylized image comprises a Gaussian mixture. 4 . The method of claim 3 , wherein the third stylized image comprises a first set of pixels corresponding to a hair region and a neck region of the first stylized image, and a second set of pixels corresponding to the face region of the second stylized image.

5. The method of claim 1, wherein the normalization comprises: One or more of cropping or scaling is performed on the input image.

6. The method of claim 1, wherein the first GAN generator is different from the second GAN generator.

7. The method of claim 1, wherein the first GAN generator and the second GAN generator are AgileGAN generators.

8. The method according to claim 1, further comprising: receiving a plurality of sample style images; as well as Based on the received multiple sample style images, the first GAN generator and the second GAN generator are trained using transfer learning.

9. A system comprising: at least one processor; a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations comprising: receiving an input image; Using a first generative adversarial network (GAN) generator, generate a first stylized image based on the input image; Normalizing the input image; Using a second generative adversarial network (GAN) generator, generate a second stylized image based on the normalized input image; mixing the first stylized image and the second stylized image to obtain a third stylized image; and The third stylized image is provided as output.

10. The system of claim 9, wherein the set of operations further comprises: extracting a first facial parsing mask from the first stylized image; as well as extracting a second face parsing mask from the second stylized image, Wherein the blending of the first stylized image and the second stylized image is based on the first face parsing mask and the second face parsing mask.

11. The system of claim 9, wherein the mixture of the first stylized image and the second stylized image comprises a Gaussian mixture.

12. The system of claim 9, wherein the third stylized image comprises a first set of pixels corresponding to a hair region and a neck region of the first stylized image, and a second set of pixels corresponding to the face region of the second stylized image.

13. The system of claim 9, wherein the normalization comprises one or more of cropping or scaling the input image.

14. The system of claim 9, wherein the first GAN generator is different from the second GAN generator.

15. The system of claim 9, wherein the first GAN generator and the second GAN generator are AgileGAN generators.

16. The system of claim 9, wherein the set of operations further comprises: receiving a plurality of sample style images; as well as Based on the received multiple sample style images, the first GAN generator and the second GAN generator are trained using transfer learning.

17. A non-transitory computer-readable storage medium comprising instructions executable by one or more processors to cause the one or more processors to: receiving an input image; Using a first generative adversarial network (GAN) generator, generate a first stylized image based on the input image; Normalizing the input image; Using a second generative adversarial network (GAN) generator, generate a second stylized image based on the normalized input image; Mixing the first stylized image and the second stylized image to obtain a third stylized image; as well as The third stylized image is provided as output.

18. The computer-readable storage medium of claim 17, wherein the instructions further cause the one or more processors to: extracting a first face parsing mask from the first stylized image; and extracting a second face parsing mask from the second stylized image, Wherein the blending of the first stylized image and the second stylized image is based on the first face parsing mask and the second face parsing mask.

19. The computer-readable storage medium of claim 17, wherein the instructions further cause the one or more processors to: receiving a plurality of sample style images; and Based on the received multiple sample style images, the first GAN generator and the second GAN generator are trained using transfer learning.

20. The computer-readable storage medium of claim 17, wherein the third stylized image comprises a first set of pixels corresponding to a hair region and a neck region of the first stylized image, and a second set of pixels corresponding to the face region of the second stylized image.

Images