METHODS, APPARATUS, AND SYSTEMS INCORPORATING GANs MINIMIZING ATTRIBUTE ENTANGLE USING SELF-CORRECTING GAN SPACES

Abstract

METHODS, APPARATUS, AND SYSTEMS INTEGRATING GANs MINIMIZING ATTRIBUTE ENTANGLE USING SELF-CORRECTING GAN SPACES Methods, apparatus, and systems achieve disentanglement in semantic editing using GAN-based models. Self-correcting (low-density) latent code samples are projected into the original latent space and the editing directions corrected through retraining based on resulting high-density and low-density regions in the modified latent space. Leveraging the original significant directions and semantic region-specific layers, the operations interpolate the original latent codes to generate images with minority combinations of attributes, and then reverse these samples back into the original latent space.In accordance with embodiments, the operations can be applied to pre-existing methods that learn meaningful latent directions. Attribute disentanglement is improved with added small amounts of samples from low-density regions. The resulting GAN models with disentangled editing directions are useful in a variety of applications, including virtual reality applications, virtual try-on-off (VTO) applications, or other virtual try-on applications that simulate an effect of a product or service, among other applications. Figure for abstract: none.

Description

Title of the invention: METHODS, APPARATUS AND SYSTEMS INCORPORATING GANs MINIMIZING ATTRIBUTE ENTANGLE USING SELF-CORRECTING GAN SPACES Technical field

[0001] This disclosure relates to image processing using artificial intelligence (AI), particularly using generative adversarial networks (GANs), and applications incorporating such image processing such as image-to-image translation, text-to-image translation, augmented reality applications, and virtual try-on applications, among others. CONTEXT

[0002] GANs can synthesize realistic images with the learned latent space shown to encode rich semantic information with various interpretable directions. However, due to the unstructured nature of the learned latent space, a bias is inherited from the training data where specific groups of visual attributes tend to appear together. For example, attributes that tend to appear together include age and glasses or women and lipsticks. Therefore, interpolation following editing directions of one attribute could result in changes entangled with other attributes. To address this issue, previous works usually adjust the learned directions to minimize changes in other attributes. However, such known approaches may still fail on highly correlated features.

[0003] It is desirable to achieve improved attribute disentanglement and an improved GAN framework that provides greater disentanglement. SUMMARY

[0004] Methods, apparatuses, and systems achieve disentanglement in semantic editing using GAN-based models. Self-corrected (low-density) latent code samples are generated and projected into the original latent space, and editing directions are corrected by retraining based on both high-density and low-density regions in the modified latent space (e.g., as fused with the self-corrected latent code samples). By leveraging the original meaningful directions and semantic region-specific layers, in one embodiment, the operations interpolate the original latent codes to generate images with minority combinations of attributes, and then invert these samples back to the original latent space. In accordance with embodiments, the operations can be applied to pre-existing methods that learn meaningful latent directions. The resulting GAN models with disentangled editing directions are useful in a variety of applications, including virtual reality applications, virtual try-on-top (VTO) applications, or other virtual try-on applications that simulate an effect of a product or service, among other applications. Attribute disentanglement is improved with added small amounts of samples from low-density regions (e.g., 30 to 1000 samples). Brief description of the drawings

[0005] [Fig IA, IB] Figures 1A and 1B are each a graphical representation of GAN latent spaces where [Fig.1A] shows an original latent space and IB a self-corrected latent space, in accordance with one embodiment.

[0006] [Fig 2A, 2B] Figures 2A and 2B are tangle plots showing absolute values ​​of tetrachoric correlations between each pair of attributes in the original FFHQ dataset ([Fig.2A]) and the StyleGAN2-generated images ([Fig.2B]).

[0007] [Fig 3A, 3B] Figures 3A and 3B are example visualizations of the latent vector entanglement of glasses and age, which likely share similar decision boundaries and interpolation directions. [Fig.3A] shows age groupings equipped with PCAs trained on glasses samples and [Fig.3B] shows a project of self-corrected samples (young person with glasses and old person without glasses) on the same axes.

[0008] [Fig 4A-4E] Figures 4A, 4B, 4C, 4D, and 4E are graphical representations of attribute dependence (AD) (the lower the better) where each y-axis represents the average AD and each respective x-axis shows delta logits for, respectively, lipstick, glasses, age, beard, and gender.

[0009] [Fig.5] [Fig.5] is a block diagram of a graphical user interface (GUI) for producing a synthesized image from a source image using a controllable GAN, in accordance with one embodiment.

[0010] [Fig.6] [Fig.6] is a block diagram of a computer system, according to one embodiment.

[0011] [Fig.7] [Fig.7] is a flowchart showing operations in accordance with one embodiment hereof.

[0012] The present concept(s) is / are better described by means of certain of their embodiments, which are described herein with reference to the accompanying drawings, in which like reference numerals refer to like features throughout. It is to be understood that the term invention, when used herein, is intended to refer to the inventive concept underlying the embodiments described below and not just the embodiments themselves. It is also to be understood that the general inventive concept is not limited to the illustrative embodiments described below and that the descriptions that follow are to be read in that light. DETAILED DESCRIPTION

[0013] Recent advances in GANs [9], such as the StyleGAN working group [13, 14, 12] and BigGAN [6], amplify a remarkable success for photorealistic image synthesis. In addition to a variety of real-world applications such as image-to-image translation [11, 25] or text-to-image translation [24, 15], another working group [3, 4, 23, 10, 19, 7] that studies the interpretability of GANs has attracted increasing attention in the research community. Training GANs produces respective learned latent spaces that react to the respective training datasets used in training. A learned latent space is a multidimensional space that represents the training dataset observed during training. That is, the latent space for the model encodes the observed data in a meaningful way through a spatial representation.The latent space comprises a number of latent codes (e.g., typically data vectors) representing individual data points in the latent space. Training examples such as individual images in the training data with similar features, such as similar facial attributes, are often represented in the latent space "closer" to each other, e.g., in a cluster. Noted works investigate the learned latent space by identifying semantically meaningful directions and interpolating along the learned directions. However, there remain challenges in perfectly disentangling correlated features such as age and glasses while obtaining valid feature orderings.Thus, semantic face editing seeks to manipulate the facial attributes of a given image with the expectation that the model can change the target attribute while retaining other information of the input face.

[0014] Previous work has widely adopted three approaches to obtain disentangled controls: learned direction orthogonalization

[19] , semantic mask-based controls

[23] , and gradient-based channel filtering [7]. Learned direction orthogonalization

[19] follows the assumption that, for any learned direction, a change along another orthogonal axis should not affect the feature for the learned direction. For example, if A and B are two orthogonal directions that define two hyperplanes, the change along axis A should not affect its distance from the hyperplane of B. This can be achieved through projection of one direction onto another or optimization-based corrections. In practice, as shown in

[19] , learned directions are often found to be orthogonal, but entangled in the embedding vector space due to the highly dimensional nature of the space. The second type of strategy

[23] uses the information contained in the semantic mask and disentangles features in different semantic regions. While this shows effectiveness on more localized attributes, it does not generate orders for global attributes such as gender or age. Gradient-based channel filtering [7] selects channels based on their importance with respect to a target attribute.Mathematically, by taking the gradient with respect to each attribute, [7] selects the channels with the maximum impact on the target attribute while filtering out the channels with the maximum impact on other attributes. However, this might fail if two attributes are highly correlated and share almost the same set of channels for decisions.

[0015] Therefore, questions arise, for example, whether the original GAN ​​space is "entangled"? Is the GAN model even capable of generating images that never or barely appeared in the training data, for example, images of men with lipstick or women with beards? It is assumed that the absence of such training data results in a non-uniformly distributed density in the learned latent space (as shown in Figures 1A and 1B), which leads to the bias of the identified directions. Figures 1A and 1B are each graphical representations of respective GAN latent spaces 100A and 100B in accordance with embodiments. [Fig. 1A] represents a latent space determined using a known approach to training a GAN model while [Fig.[1B] represents a latent space determined using an approach for training a GAN model in accordance with an embodiment hereof where self-correction is used. Figures 1A and 1B represent differences in entanglement in the (facial) attributes of glasses and age between the known approach and the approach using self-correction in accordance with an embodiment hereof. The contour lines (heavily dashed lines) at 100A and 100B indicate a density of different subregions in the learned latent space. The training images 102A and 102B represent, respectively, a face of an elderly person with glasses and a face of an elderly person without glasses. The training dataset includes more faces of elderly people with glasses than without glasses. Following (e.g., conventional) training of the GAN model, as shown in [Fig.1A], the face of . elderly person without glasses 102B is associated with a low-density region 104A of the latent space 100A. As shown in [Fig.lB] after training using a self-correction approach in accordance with one embodiment herein, the image 102B is shown associated with a high-density region 104B of the space 100B. Figures 1A and 1B show an input image 106 comprising a younger face without glasses and depict how the respective trained GAN models would interpolate an aging effect applied to the input image 106. [Fig.lA] shows the interpolation (represented by an arrowed dotted line 108A) of the input image to the sub-region associated with elderly faces with glasses, exhibiting an entanglement of the attributes glasses and age. However, [Fig.lB] shows the interpolation (line 108B) to subregion 104B with a high density of elderly faces without glasses, avoiding clutter.High-density regions are open to direct sampling through a known distribution, such as the standard sampling process for GANs by passing standard Gaussian noise over the network. In contrast, low-density regions are regions on which the network is not well trained and are therefore extremely difficult to sample using the standard procedure mentioned above. Minority combinations of attributes include those that do not exist or represent less than 5% of all possible combinations in the training data, e.g., for the lipstick and gender combination, the example combination of man with lipstick does not exist in the training data.

[0016] To support this intuition, the empirical results of the correlation between different pairs of attributes in the original image distribution and how this affects the learned GAN space are therefore shown in Figures 1A and 1B. Inspired by these empirical results, a new framework called SC2GAN is shown and described to achieve disentangled orders. The samples manipulated with GAN inversion methods are projected into the low-density regions of the learned latent space to obtain a more balanced latent space distribution. It is shown that the interpretable directions re-learned by different methods with the proposed framework applied here would be corrected towards the correct grouping, as shown in [Fig.1B], 1. Related GAN Work#

[0017] GANs [9] are one of the most powerful generative models that have been widely used to tackle various tasks related to real-world imaging [13, 15, 14]. This section provides an overview of two different categories of approaches to control GAN ​​outputs, as well as the line of work that incorporates real images into the GAN latent space. 1. Image Editing with Conditional GANs#

[0018] By incorporating loss terms related to the class label during training, conditional GANs obtain explicit controls over the image generation process [16, 11, 18], which are capable of generating images of classes specified by a user with a class label as input. Nevertheless, they lack controls over multiple attributes, so the entanglement problem may still occur. In the field of facial image generation, methods have been proposed to obtain finer-grained controls over multiple attributes [8, 20]. These approaches translate 3D face rendering controls, i.e., 3DMM parameters [5], into the GAN framework and are capable of controlling expressions, pose, and lighting while preserving identity. However, the controls that can be learned by these methods are limited to parameterizing facial attributes of existing 3D models. 1. Interpolation in GAN latent space#

[0019] Unlike conditional GANs, another line of work [3, 19, 10, 23] explores controls on output image semantics in trained unlabeled GANs. They showed that this GAN latent space encodes rich semantic information with many meaningful directions, interpolating along these gives human-interpretable changes in the output semantics. InterFaceGAN

[19] uses pre-trained image classifiers to cluster latent codes corresponding to different semantics, and trains state vector machines (SVMs) on these samples to learn the editing direction. Grad-Control [7] works similarly by training fully connected layers on a small amount of labeled latent codes, and taking the classifier gradient directions as the meaningful path in the latent space.GANSpace

[10] works in an unsupervised manner by performing PCA on generator features, and by regressing linear directions in the latent space corresponding to the principal components, which correspond to human-interpretable changes in the image space. StyleSpace

[23] learns finer-grained commands by computing latent channels activated exclusively for semantic regions defined by pre-trained semantic segmentation networks.

[0020] Although many significant directions have been discovered, during interpolation, entanglement in attributes, i.e., the change of the target affects other attributes assumed to be independent by humans, often occurs. This phenomenon could be attributed to the nature of the learned latent space, i.e., groups of visual attributes are not guaranteed to be uniformly distributed in the training data, so the generator captures this property and implicitly encodes it in its latent space. To solve this problem,

[19] proposes to adjust the editing directions and minimize the change in the entangled attributes by orthogonalizing the target direction of the entangled attribute through a projection, while [7] filters the salient latent channels to predict the entangled attribute during interpolation.

[0021] Although the aforementioned techniques achieve some success, disentanglement remains difficult when the correlation between the attributes is significantly high and after adjustment, the resulting direction brings only trivial changes in the target, or there are only very few channels left for the target attribute. The approach of

[23] suffers less from the entanglement problem because it focuses on attributes belonging to localized semantic regions, but it lacks the ability to edit global attributes such as age that require changes across the entire image. Unlike approaches that directly adjust biased directions, according to the embodiment(s) herein, the present methods, apparatuses, systems, and techniques propose to use such directions and correct biased latent code distributions to learn more disentangled directions. 1. Inversion GAN#

[0022] GAN inversion embeds real-world images into the GAN latent space, which can then be edited with latent space interpolation [1, 2, 14, 21, 22]. There are two main categories of GAN inversion: Optimization-based methods [1, 14, 2], which sample from the original latent space and optimize the latent code to match the output to the real image target, or encoder-based methods [21, 22], which aim to reverse the generation process and learn the reverse mapping from image space to latent space, using training on a large number of code-latent image pairs.A common challenge for GAN inversion is the trade-off between distortion (i.e., target-likeness) and editability (i.e., how close the inverted code must be to the original latent distribution for latent interpolation directions to be applicable), and different regularization schemes have been proposed to handle these trade-offs [26, 21]. In contrast, according to the embodiments herein, naive latent optimization of VF latent codes on VF + interpolation outputs is employed, to achieve little distortion as the results faithfully represent minority distributions in the original W-space. 2. Methodology

[0023] According to embodiments, this section describes obtaining disentangled commands in the VF StyleGAN space. Observations of entanglement in the latent space are first discussed, followed by a more detailed analysis that quantifies these phenomena. Finally, a framework is proposed that learns more disentangled editing directions by generating latent code samples located in low-density areas to reduce biases in the original latent sample distribution.

[0024] 2.1. Observation of the latent space entanglement

[0025] The disentanglement of the StyleGAN VL space here is motivated by observations of correlations between pairs of attributes in StyleGAN outputs. GANs map latent codes z from a known ZQ distribution to an image space x G with the mapping function g : Z X. In StyleGAN

[13] , instead of directly feeding z to the generator blocks, the output is driven by a function of w — M(z) °ù is a multi-layer perceptron network with 8 layers. The vectors w are named style vectors lying in the VF space GP^, and each vector w is repeated 18 times for a vector w+ and fed to the generator layers at different resolutions to generate the final image with + which has been shown to enable powerful feature drives at different levels of abstraction.Nevertheless, multiple works [13, 19, 10] have found that, unlike the original Z-distribution, the VL-space distribution is distorted because it captures the correlations between attributes in the training data, resulting in low-density regions for minority attribute groups.

[0026] To visualize these effects, the aggregated faces generated by StyleGAN-FFHQ for a set of attributes were computed, displayed (not shown), and observed. In addition, faces averaged from a merged dataset containing equal amounts of original W samples and autocorrected samples (e.g., from VF+ edits) were computed, visualized (not shown), and observed. The observations reveal the entanglement between different pairs of attributes in the first set of averaged faces, e.g., most women smile while most men do not, and older people are likely to wear glasses.By projecting data onto low-density regions in the original clusters, e.g., edited images of elderly people not wearing glasses or men with lipstick, the corrected distributions show less entanglement in the second set of averaged faces. The result provides intuition for the entanglement problem that many previous works that learn semantic editing directions in VF [19, 7, 10] have suffered from, where interpolation for editing one attribute yields changes in multiple attributes.

[0027] 2.2. Latent Space Entanglement Analysis

[0028] Figures 2A and 2B are graphical representations of the entanglement 200, 202 showing absolute values ​​of tetrachoric correlations between each pair of attributes in the original FFHQ dataset ([Fig.2A]) and the images generated by StyleGAN2 ([Fig.2B]).

[0029] To further explain the above results, latent space entanglement was analyzed from both the training data and the learned latent distribution perspectives. First, the original FFHQ training data for StyleGAN was analyzed. With the FFHQ feature dataset

[17] , Figure 2A presents the measured correlations between each pair of attributes, which exhibit non-trivial correlations between attributes such as glasses and age in alignment with the observations of the averaged faces from the first set. Then, VF space was analyzed, leveraging knowledge from pre-trained image classifiers

[13] . In one embodiment, an image bank of 500k synthetic images was created and pseudo-labels were predicted for each attribute of interest. A same correlation matrix was then calculated as presented in Figure 2B.The high correlations between some attribute pairs make it difficult to learn disentangled editing directions. For example, for glasses and age, since the high-density region for elderly contains mostly samples of elderly code wearing glasses, it is very likely that when interpolating the young latent code without glasses to the elderly direction, glasses will be added. This essentially corresponds to the overlaps between the separation boundaries learned from the data generated from VF and the original distribution.

[0030] Overlaps between separation boundaries can be visualized using principal component analysis (PCA) techniques and graphs. For example, a graph can be generated to visualize the latent vector entanglement of eyeglasses and age, which likely share similar decision boundaries and interpolation directions. Figures 3A and 3B are example visualizations of the latent vector entanglement of eyeglasses and ages 300 and 302, which likely share similar decision boundaries and interpolation directions. [Fig. 3A] shows age groupings equipped with PCAs trained on eyeglass samples and [Fig. 3B] shows a project of self-corrected samples (young person with eyeglasses and old person without eyeglasses) on the same axes.

[0031] In an example as shown in Figures 3A and 3B, age groupings equipped with PCAs trained on eyeglass samples can be determined and shown graphically. In [Fig.3A] an example of visualizing the latent vector entanglement of eyeglasses and age, which likely share similar decision boundaries and interpolation directions, is demonstrated. [Fig.3A] shows age groups equipped with PCAs trained on glasses samples. In [Fig.3B], the self-corrected samples (young person with glasses and elderly person without glasses) are projected onto the same axes.

[0032] As discussed in

[19] , for such large entanglement, orthogonalizing the edit direction through a projection does not work well, as it also eliminates the target direction. Similarly, the salient channels proposed in [7] for the two attributes also overlap significantly, making channel filtering likely to fail to disentangle the attributes. Therefore, according to one embodiment, this problem is approached from another angle, by asking the question: Instead of manipulating the learned edit directions, if one could obtain low-density samples, e.g., elderly people without glasses in VF space, and create a less biased training distribution for the edit direction, would the newly trained direction be more disentangled?

[0033] 2 .3. Learning disentangled directions

[0034] According to one embodiment, in order to learn disentangled directions in VF, a method called SC2GAN is introduced, which corrects the bias in the VF distribution via self-corrected latent code samples. According to one embodiment, given an entangled editing direction in VF, the method first interpolates the VF codes in VF+ following this direction to obtain edited images with localized changes corresponding to minority attribute groups, then projects the edited images into the VF space and re-learns the editing directions with the self-corrected latent codes fused with the original VF space training data.

[0035] Spatial editing in VF+. As observed in several previous works [23, 10, 7], compared to VF, StyleGAN's VF+ space allows for more localized controls, with VF codes contributed to layers at different resolutions controlling different levels of abstraction, experiments show that the entanglement problem can be mitigated with spatial editing in VF+.

[0036] In particular, with a learned entangled editing VF direction, interpolation along the same direction in attribute-specific VF + layers achieves better disentanglement than interpolation in VF, resulting in images with the correct minority attribute groups, such as elderly people who do not wear glasses. However, VF + interpolation has limited capability, as changes are mostly limited to specific semantic regions. When editing attributes involving global deformation, limiting changes to specific VF + layers sometimes results in an absence of the desired target effect. In contrast, VF space modifies the image at a global level with wider ranges of changes available. Nevertheless, W + interpolation remains useful because it allows access to StyleGAN-generated images with minority attribute clusters. It is assumed (without prejudice or limitation) that if these images can be reconstructed from latent codes in W space, then such a clustering of latent codes represents the low-density region needed to correct for the tangled editing W direction.

[0037] Obtaining self-corrected samples in W. To verify the above assumption, naive latent optimization is employed and the W+ interpolation results are projected into W-space. It is observed that the inverted W-codes faithfully reconstruct the VF+ editing results and preserve minority attribute groups well. In other words, these latent codes are self-corrected (disentangled) samples based on the original entangled editing directions, and they can be merged with the original W-space samples to create a more balanced distribution for retraining the editing directions. The operations in Table 1 describe how to formally obtain self-corrected latent code samples, according to one embodiment.

[0038] [Table 1]

[0039] Algorithm lObtain a self-correcting sample w' in W from w for attribute a with semantic editing direction fa

[0040] 1: Latent sample w corresponding to w +

[0041] 2: Generator G, inversion function Inv

[0042] 3: Set of W + layer indices relevant for editing ala — {i}

[0043] 4: Interpolation step size step '

[0044] 5: For idxi E. Ia:

[0045] 6: + fs\

[0046] 7: w

[0047] 8: Return w'

[0048] According to lines 1 and 2 of Table 1, the operations provide a latent space code sample (e.g. w) in VF space corresponding to an associated code (e.g. w + ) in VF + space such as to control the generation of an edited image sample (i.e. instances of an edited image) for attribute a along an editing direction (e.g. a semantic direction represented by f ). The attribute a is determined as one that represents a minority attribute in the distribution, i.e. it is associated with a low density region of the latent space and is entangled with at least one other attribute. The generator G produces an edited image sample and an inversion function Inv inverts such a generated (return) image sample into latent code samples (e.g. w') in VF space.

[0049] According to row 3 of Table 1, the operations provide the applicable parameters for producing an edited image according to the structure of the GAN generator, for example, according to StyleGAN techniques in W+ space. In rows 4 to 6 or of Table 1, the operations interpolate along the editing direction according to the function f using the latent code sample and step size interpolation parameters and steps 5 to construct a vector (latent code sample) K'+ with which to generate the edited image sample.

[0050] On lines 7 and 8, the generator produces the edited image samples g(m' + ) from the latent code samples w+ as interpolated via lines 4 to 6 of Table 1. The image samples are inverted to produce a new latent code sample w' for merging into the VF space for relearning the edit directions using the merged latent space where the distribution (of latent space codes) is more uniform. The directions are re-learned according to the technique of the associated Generator. For example, the edit directions are re-learned in the same way as the original edit directions, but from the merged latent space.

[0051] Assuming that the original VF space distribution has the entanglement problem between the attribute and a2, where the high density regions mainly contain latent codes w with the semantics + and + CC-) in image space, so it is likely that changing the sign of ai by interpolation in VF causes the opposite change in a2. However, since these two attributes are controlled by VF+ layers at different levels, VF+ interpolation for editing results in images with the semantics + ) and _ j. By means of an inversion, the codes w' having the same set of semantics are obtained in VF space (merged space including w' codes after inversion) so that the strength of a2 in each cluster ai can be balanced by merging the original w codes with the self-corrected codes. By retraining the edit direction for with the corrected distribution in VF, is decoupled from the signs of a\ and disentangled and global commands are obtained. 3. Experiences

[0052] In this section, the framework here is applied to existing supervised methods that learn edit directions based on VF space latent code samples to obtain more disentangled directions.

[0053] 3.1. Configuration of Experiments

[0054] Models. Experiments were performed on the W-space of StyleGAN2

[14] pre-trained on FFHQ

[13] with SVM-based

[19] and gradient-based [7] editing directions. Samples of 500k images were made and pseudo-labels were obtained for the attributes gender, smile, glasses, age, lipstick and beard with pre-trained attribute classifiers

[13] .

[0055] Learning the original directions. Since the framework, according to the embodiments, performs retraining of the learned editing directions, the latent codes W are first sampled based on the images with the largest / smallest logits of the classifier and follow one of the methods i) SVM-based editing directions

[19] or ii) gradient-based editing directions [7] to learn the original W-space editing directions.

[0056] Learning disentanglement directions. With the original edit directions learned, the operations in Table 1 are applied to each method and the set of latent code samples to obtain the self-corrected samples. The directions are retrained (using method i) and method ii) respectively) from scratch using the merged dataset containing both self-corrected samples and the original VF codes. Implementation details can be found below in the Experiment Details section.

[0057] 3.2. Manipulation of disentangled attributes

[0058] Qualitative results (not shown) of attribute manipulation for gender, age, glasses, lipstick, and beard have been prepared and presented for observation and comparison. For details regarding the latent interpolation methods,

[19] and [7] were trained on the same set of 100 latent code samples with the highest classifier confidence for each attribute. For

[10] and

[23] , the original directions as presented in the original paper were used, selecting the channel for “gray hair” as the Age+ channel for

[23] . When retraining the SVMs for

[19] , the negated values ​​of the original scores are extracted from the W-space training data and used as pseudo-scores for the self-corrected samples.The original directions learned by Grad-Control [7] and InterFaceGAN

[19] were compared to the edit directions after applying the self-correction framework to both processes. The results were also compared to processes to which the framework was not applicable. GANSpace

[10] learns meaningful directions in W by applying PCA to generator features and requires manual examination for semantic meanings, while StyleSpace

[23] finds locally activated semantic channels in the space that are VF+ layers with affine transformations applied. For global attributes (age and gender) and local attributes (lipstick and beard), the framework . self-correction framework amplifies disentanglement for InterFaceGAN

[19] and Grad-Control [7]. For example, disentangled aging effects without glasses were obtained and the female direction was decoupled from the smile. GANSpace and StyleSpace suffer little from the entanglement problem, but the amount of change they make to global attributes is extremely limited, e.g., StyleSpace fails to synthesize more female effects and GANSpace lacks the ability to generate aging effects. Meanwhile, for local attributes, with the self-correction framework applied, InterFaceGAN and Grad-Control achieve similar performance to GANSpace and StyleSpace, which operate in much higher-dimensional spaces.

[0059] 3.3 Quantitative Results: Entanglement Analysis

[0060] The entanglement level was quantitatively measured with the Attribute Dependence (AD) proposed by

[23] . Experiments on the attributes were performed to show the non-trivial entanglement in the previous analysis, which are gender, glasses, age, lipstick, and beard. 3000 test images with all the attributes of interest (gender, smile, glasses, age, lipstick, beard) lying around the decision boundaries of the attribute classifiers were sampled. The images were divided into 5 test sets for calculating the AD. The operations to calculate the AD on each attribute in one embodiment are as follows.

[0061] For each set of images with target attribute ae A, with an editing method fa, where A represents all attributes, interpolate the original latent image codes following

[19] and [7] for d = 6 in 9 steps.

[0062] For each interpolation result in step \ calculate x ~ which represents the absolute change in the target attribute logit, normalized by the population standard deviation and obtain the x-values ​​for the AD plot.

[0063] For each interpolation result at step s, also calculate _ iy M , which represents the average of the absolute change in the other “ IÆ1 attribute logits, normalized by each population standard deviation, and obtain the values ​​for the AD plot.

[0064] Then group the pairs (x,y) with their x values ​​into buckets of (0, 0.25], (0.25, 0.5],..., (1.75, 2], and plot the midpoint for each bucket as the final x value, the average of the values ​​in each bucket as the final value.

[0065] Figures 4A-4E are graphical representations of the DA (400, 402, 404, 406, and 408, respectively) where each y-axis represents the average DA across all graphs and the x-axis shows delta logits for, respectively, lipstick, glasses, age, beard, and gender. Figures 4A-4E show that, with the self-correction framework applied, disentanglement improves significantly for Grad-Control [7] in all attributes. For InterFaceGAN

[19] , adding the self-corrected data significantly decreases the DA for lipstick editing, while for the rest of the attributes, the improvement is not as obvious as that of Grad-Control [7]. It is assumed (without prejudice or limitation) that this is because [7] supports training with small amounts of data, and adding the self-corrected data has significant impacts on the decision boundaries, while

[19] requires large amounts of training samples, thus less effect of the added self-corrected code on the learned SVM.

[0066] 3.4. Manipulation of real images

[0067] Experimental results were observed for age and lipstick editions following InterFaceGAN

[19] with and without applying SC2GAN (an embodiment of the self-correction framework). The proposed approach achieves better disentanglement while preserving image identity. 4. Ablation studies

[0068] Number of self-corrected samples. It has been qualitatively demonstrated how the number of self-corrected samples fused with the original VF training data affects the overall editing directions learned by [7] through a presentation of edited images (not shown), as the method of [7] can only be trained on a small dataset. With more self-corrected samples added, the original entanglement with the glasses is further minimized, as the glasses do not appear with similar aging effects present during interpolation.

[0069] Direct sampling of balanced data. Another approach that obtains the latent codes of sparse area is to sample directly from these regions based on the pseudo-labels, such as from an image bank. However, it has been demonstrated qualitatively through a presentation of edited images (not shown) that, while some clutter can be mitigated with this approach, training with these samples could result in an editing direction pointing to areas with lower image quality, because the generator is not well trained in these VF regions. Furthermore, the amount of sparse data available for sampling is extremely limited and, since sampling takes all that is available given the sparsity of this data, these samples could be close to the original separation limit.Therefore, they may fail to provide a strong enough signal for the separation boundary to move significantly.

[0070] Comparison with VF+ space editing. The results of [13, 10, 7] show that VF+ space brings localized changes. However, for attributes such as aging, editing involves large amounts of deformation of the original semantic regions; localized VF+ space edits might therefore fail to achieve the desired target effect, while interpolation in VF space is less prone to such failures as it modifies the image at a global level. The difference between spatial W+ interpolation and W interpolation for increasing the age of a baby was qualitatively demonstrated through a presentation of edited images (not shown). Both directions learned with the autocorrection framework applied in [7] and the original direction with VF+ interpolation do not suffer from eyeglass entanglement. However, the latter fails to create aging effects such as drooping cheeks and eyelid ptosis, with the changes being mostly limited to the initial semantic regions. Additional ablation studies

[0071] Additional results were prepared and observed (not shown) which showed the effectiveness of the self-correction framework on the disentanglement directions learned by

[19] by adding more self-corrected samples. For all attributes except lipstick, adding more data helps to significantly disentangle the original directions. For lipstick, the initial 100 self-corrected samples are sufficient to correct the skewed distribution. Application(s)

[0072] The resulting GAN models with disentangled editing directions are useful in a variety of applications. Image processing applications may include virtual reality applications, virtual try-on-suit (VTO) applications, or other virtual try-on applications that simulate an effect of a product or service, among other applications.

[0073] The editing directions may be associated with a user interface such as a GUI having user controls, for example to control semantic editing of the respective attributes using a GAN model associated with the user interface. An example is a face editing application for editing an input image of a face to produce an output image.

[0074] [Fig. 5] is a block diagram of a GUI 500, according to one embodiment. The GUI 500 may be presented via a display device (not shown). In one embodiment, a user may specify the target semantic attribute(s) and the desired quantity of the attribute. For example, +smile, -age may identify the attributes and a quantity such as a percentage may specify an amount to be present in the synthesized image when the generator is invoked. Since the interpolation process is continuous, image classifiers pre-trained (for example) can be used to quantify the amount of the semantic attribute (or amount of change) and find a desired manipulation strength.

[0075] The GUI 500 shows a plurality of input controls 502 provided to receive selective input for each target semantic attribute for which the GAN model (mon shown) has been configured and trained. In the present embodiment, each input control (e.g., 502A) shows a slider-like control for a single respective attribute and to identify granular input. Other types of input controls (radial controls, text boxes, buttons, etc.) may be used as for granular input or otherwise. The controls may receive a % value, a range selection, a scalar value, or otherwise. For example, an age attribute may be associated with a range of integers and accept granular inputs approximating age in years or decades, etc.Others may select relative ranges for the amount of attribute presence—small, medium, large—for example, or child, adolescent, middle-aged, old-aged, etc. These relative ranges may be associated with respective granular values ​​such as 15%, 50%, and 85%, or other values.

[0076] Via a selection of an "Apply" command 504, the semantic attribute input(s) are applied upon invocation of the generator. The output of the generator is controlled by semantic attribute commands derived from the respective auxiliary network semantic attribute classifiers. The semantic attribute input(s) are applied via these attribute commands to a source image x (e.g., 506) to generate a synthesized image x' (e.g., 508 as an output image). The synthesized image may be represented as x'=G(x), as controlled by the attribute commands. And the attribute commands respond to the semantic attribute input from interface commands 502.Although all available semantic attributes have a command, a user can choose not to change an attribute and thus the generator does not need to interpolate along a direction identified in an associated semantic command.

[0077] A source image 506 may be identified for use (e.g., uploaded, copied from a storage device, obtained via a camera input to obtain a photographic self-portrait, etc.) via a "source image" command 510. The resulting synthesized image x' 508 may be saved via a "save image" command 512.

[0078] Although individual semantic attribute commands are provided for each trained attribute in the illustrated embodiment, in one embodiment (not shown), fewer commands (e.g., for a single attribute or two attributes, etc.) are provided. While separate commands are provided for individual attributes, in one embodiment (not shown), a single command may be provided for combined attributes (e.g., age and gender). A multi-attribute command is calculated by vector arithmetic. To make a face with less smile, more glasses, older (more age), the GUI is configured to receive inputs for the respective directions, i.e., -smile, +glasses, +age. In accordance with one embodiment of operations of a computing device, the inputs are associated with respective gradient vectors (of associated semantic attribute commands) and the vectors are summed and normalized. The generator interpolates (linearly) along the calculated direction (i.e., combined directions) to produce the output image.

[0079] In one embodiment, the GUI is provided by a first computing device (not shown) and the generator is provided by another computing device (not shown) located remotely from the first computing device. The other computing device may provide the controllable GAN generator as a service. The first computing device providing the GUI is configured to communicate the input image and semantic attribute direction inputs (e.g., as percentage values ​​or other inputs) to the remotely located computing device providing (e.g., executing) the generator. Such a remotely located computing device may provide an application programming interface (API) or other interface for receiving the source image (or a selection) and the semantic attribute direction inputs.The remotely located computing device may calculate the direction vector and invoke the generator by applying the semantic attribute commands. In another embodiment, the computing device providing the GUI may be the same as the computing device providing the generator.

[0080] In one embodiment, in addition to the developer (e.g., used at training time) and target computing device (used at inference time) aspects, those skilled in the art will understand that computer program product aspects are disclosed, where instructions are stored in a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.), which, when executed, cause a computing device to perform any of the method aspects disclosed herein. In one embodiment, a computing device includes a processor (e.g., a microprocessor (e.g., a CPU, a GPU, a plurality of any of these), a microcontroller, etc.) that executes computer-readable instructions such as those stored in the storage device. In one embodiment, the computing device includes a circuitry (e.g., "specialized") that performs the functions of instructions, for example, without needing to read such instructions.

[0081] Related aspects of the e-commerce system are also shown and described. A user's computing device, in one embodiment, is configured as a client computing device with respect to the e-commerce system. The e-commerce system, for example, stores a computer program for such a client computing device. Thus, the e-commerce system includes, as a component thereof, a computer program product, which product stores instructions that, when executed by the client computing device (e.g., a processing unit thereof), configure such a client computing device. These and other aspects will be apparent.

[0082] [Fig. 6] is a block diagram of a computer system 600. In one embodiment, the computer system 600 includes a plurality of computing devices which, in one embodiment, include servers, development computers (PCs, laptops, etc.), and mobile devices such as smartphones and tablets, etc. Shown is a network model training environment 602 including hardware and software for defining and configuring, such as through conditioning, a GAN-based CNN architecture 601. The architecture 601 includes a GAN-based model 603 (e.g., including a generator G with a plurality of editing directions) and network-based classifiers 605 (e.g., a plurality of classifiers for determining the presence of particularly semantic attributes). Other types of classifiers may be used.In one embodiment, the classifiers 605 are useful for determining editing directions. Once the GAN model 603 is trained and the editing directions are disentangled as described herein using the self-correction framework, the GAN 603 may be useful for producing output edited images without needing to use the classifiers 605.

[0083] In one embodiment, the GAN model 603 and, optionally, if applicable, the attribute classifiers 605, (e.g., CNN 601, once trained, are provided for use on a target device such as one of the mobile devices 610, 612 of the system 600 or other devices such as 613. The mobile devices 610 are representative user computing devices, e.g., consumer users. It is understood that such users may use other forms of computing device such as a desktop computer, workstation, etc. The device 613 represents a computing device for performing training data generation (thus, a training data generation device). In this embodiment, such a computing device employs the generator (GAN model 601) to generate additional image data that can be easily labeled with semantic attributes (e.g., using attribute classifiers 605) and can be used to train a network model (e.g., in a supervised manner). The form factor of the device 613 may be a server, laptop, desktop, etc. and need not be a consumer-type mobile device such as a tablet, smartphone, etc.

[0084] In one embodiment, the network model training environment 602 employs, at least in part, a GAN model 603 that is pre-trained for the image task (e.g., face image generation). The generator is pre-trained such as by using an image dataset 614 stored at a data server 616. In one embodiment, the model 603 is a home-grown model. In one embodiment, the model 603 is publicly available, e.g., via an open-source license. The dataset 614 may be similarly developed and available (e.g., as an open-source dataset). Depending on the type of image task and the network architecture (e.g., supervised), the training is supervised and the dataset annotated accordingly for that training. In other scenarios, training is unsupervised and the data is set accordingly.

[0085] In one embodiment, the generator 603 and, optionally, as desired, the classifiers 605 are incorporated into an application 620, such as an augmented reality (AR) or virtual reality (VR) application, etc. Although not shown, in one embodiment, the application 620 is developed using an application developer computing device for particular target devices having particular hardware and software, particularly a target operating system configuration. In one embodiment, the application 620 is a native application configured to run in a specific native environment such as an environment defined for a particular operating system (and / or hardware). In one embodiment, the application 620 takes the form of a browser-based application, for example, configured to run in a browser environment of the target device.

[0086] In one embodiment, the application 620 is distributed (e.g., downloaded) by user devices such as mobile devices 610 and 612. Native applications are often distributed through an application distribution server 622 (e.g., a "Store" operated by a third-party service), although this is not required.

[0087] In one embodiment (not shown), the application 620 does not include the CNN architecture 601 per se (i.e., it does not include the generator and auxiliary network). Instead, the application 620 is configured with an interface to communicate with a remote device providing these components as a service (not shown), e.g., as a cloud service. Storing and running the generator and auxiliary network is resource intensive and may be too large / demanding for some computing devices. Other reasons may also factor into the paradigm of the application 620.

[0088] In one embodiment, the application 620 is configured to provide an augmented reality experience (e.g., via an interface) to a user. For example, an effect is provided to an image via processing by the generator 603. In one embodiment, one or more effects are provided via other operations, which may include other deep networks, effects pipelines, etc. In one embodiment, effects are provided to simulate the application of a product or service, as described in more detail below.

[0089] In one embodiment, the mobile device 610 or 612 includes a camera (not shown) for capturing an image (e.g., a still or video image, whether or not it is a photographic self-portrait image). In one embodiment, the effect is applied to the image, e.g., the video image, and displayed on a display (not shown) of the mobile device 610 or 612 to simulate an effect on a user image. In some embodiments, the effects are applied in (near) real-time. When the position of the camera is changed, the effect is applied in response to the image or images of the video as captured to simulate augmented reality. As will be appreciated, real-time operation is limited by processing resources. In one embodiment, an effect is not simulated in real-time but is delayed, which may impact the augmented reality experience.

[0090] In one embodiment, the computing devices are coupled for communication via one or more networks (e.g., 622), which include wireless or other networks, public or other networks, etc.

[0091] By way of example, but not limitation, an e-commerce system 624 is web-based and provides a browser-based AR application 620A as a component of an e-commerce service provided by the e-commerce system 624. The e-commerce system 624 includes a configured computing device and a data store 626 (e.g., a database or other configuration). The data store 626 stores data about products, services, and related information (e.g., techniques for applying a product). The data store 626 or other data storage devices (not shown) stores recommendation rules or other forms of product and / or service recommendations, etc. to assist a user in choosing from available products and services. The products and services are presented via a user experience interface displayed on a (mobile) computing device of the user. It will be appreciated that the e-commerce system 624 is simplified.

[0092] In one embodiment, the browser-based AR application 620A (or application 920) provides an augmented reality customer experience such as to simulate a product, technique, or service provided or facilitated by the e-commerce system 624. It will be understood that the application 920, in one embodiment, is also configured to provide e-commerce services such as via a connection to the e-commerce service 624.

[0093] By way of example, but not limitation, a product includes a cosmetic (e.g., makeup), anti-aging, or rejuvenating product, and a service includes a cosmetic, anti-aging, or rejuvenating service. A service includes a treatment or other procedure. The product or service relates to a portion of a human body such as the face, hair, or nails. In one embodiment, a computing device (such as mobile device 612) so configured therefore provides a facial effect unit 612A including processing circuitry configured to apply at least one facial effect to the source image and to generate one or more virtual instances of a source image with the effect applied (e.g., displayed) on an e-commerce interface of the computing device that is facilitated by the e-commerce system.In one embodiment, the facial effect unit 612A uses the generative adversarial network (GAN)-based generator (g) and optionally, if applicable, the classifiers 605 as described herein, to produce the effect-applied source image. In one embodiment, the computing device provides a user experience unit 612B including processing circuitry to determine at least one product or service from the data store 626 and to generate one or more virtual instances of a recommendation on an e-commerce interface for purchasing products or services. In one embodiment, the at least one product is associated with a respective facial effect and the facial effect unit applies the respective facial effect to provide a virtual try-on experience.

[0094] In one embodiment, the user experience unit 612B is configured to present a graphical user interface (e.g., browser-based or otherwise) for working with the computing device 612 and the e-commerce system 624. In one embodiment, the e-commerce system 624 is thus configured to provide the AR application for execution by client computing devices such as a mobile device (e.g., 612) and is cooperatively configured to provide e-commerce services to the client computing device to facilitate (product / service) recommendations for the AR simulation via the client computing device (e.g., 612) and facilitate purchases.

[0095] Thus, any of the computing devices, but in particular mobile devices, provides a computing device for translating an image from a first domain space to a second domain space. The computing device comprises a storage unit storing a generative adversarial network (GAN)-based model (e.g., the generator (g)), configured to generate images driven for semantic attributes.In one embodiment, the computing device includes a processing unit configured to (e.g., via the RA application 620): receive the source image; receive an input to identify at least one semantic attribute to be controlled, e.g., including an input to refine the semantic attribute (e.g., a percentage); provide the image to the generator g to obtain a synthesized (e.g., new) image in response to the semantic attribute input; and provide the new image for presentation.

[0096] In one embodiment, the generator is configured to synthesize a particular semantic attribute that includes one of: a facial feature including age, gender, smile, or other facial feature; a pose effect; a makeup effect; a hair effect; a nail effect; a cosmetic or dental surgery effect including one of rhinoplasty, rhytidectomy, blepharoplasty, implant, otoplasty, teeth whitening, teeth straightening, or other cosmetic or dental surgery effect; and an appliance effect including one of an ocular appliance, an oral appliance, a hearing aid, or other appliance effect.

[0097] In one embodiment, the synthesized output of the generator g (e.g., the synthesized image or new image) is provided to a second network model, for example for a face editing task. The second network model may include a second generator or feature detector and a simulator for applying an effect (e.g., producing another new image). The effect may be presented in an AR interface. In one embodiment, the processing unit is configured to provide the synthesized image (or another new image defined therefrom) in an augmented reality interface to simulate an effect applied to the image. In one embodiment, the effect includes any of a makeup effect, a hair effect, a nail effect, a cosmetic or dental surgery effect, appliance effect, or other simulation effect being applied. In one embodiment, the source image comprises an applicable portion (e.g., face, hair, nails, or body portion) of a subject such as a user of the device.

[0098] The network model training environment 602 provides a computing device configured to perform a method such as a configuration method by conditioning a GAN-based generator and classifiers (e.g., the classifiers and the GAN model may share the same latent space in one embodiment). It is understood that embodiments of the computing device aspect of the network model training environment 602 and any related embodiments, e.g., the generator or model, apply to the training method aspect, with appropriate adaptation.

[0099] Figure 7 is a flowchart of operations 700 in accordance with one embodiment hereof. Operations 700 may be performed, for example, by computing device 602, in one embodiment. The computing device is provided at step 702 with a GAN model (e.g., 601) having a generator G and an entangled latent space IV for editing attributes ae A to which an editing direction ffl (w) is given, where the editing direction is learned from the latent codes w in accordance, for example, with a conventionally known technique as described herein. At 704, operations obtain a plurality of latent code samples {w} (e.g., a subset of IV space) of the original latent space IV. In one embodiment, the sampling applied respective attribute classifiers to a large image set comprising 500K images to obtain confidence scores for each attribute.The sampling is responsive to the scores, for example, by selecting latent codes corresponding to those with the highest confidence scores for a particular attribute or the lowest scores. The sampling in one embodiment is further responsive to one or more other attributes such as for balancing purposes, for example, by selecting a balanced subset that takes into account gender - 50 / 50 splitting of the samples.

[0100] At 706 latent interpolation is performed on the latent codes {w} in the IV+ space following the expressions of Table 1 in lines 4 to 6 for a target attribute (e.g., d). The resulting latent codes {W+'} correspond to images with localized changes for the target attribute a.

[0101] In 708 the operations obtain self-correcting latent codes { w'} in the IV space through latent optimization, by applying inversion, according to the expression in line 7 of Table 1.

[0102] At 710, the operations relearn the edit direction / fl for attribute a with the latent codes {w} and the self-correcting latent codes {w'} (e.g., as merged). Safeguard declarations

[0103] In accordance with the embodiments herein, various aspects will be apparent to one skilled in the art, including those set forth in the following numbered statements.

[0104] Statement 1: A method of disentangling a target entangled semantic attribute editing direction from a plurality of semantic attribute editing directions learned from a first latent space of a generative adversarial network (GAN) model, the target entangled semantic attribute editing direction learned for generating edited images with respect to a target semantic attribute, the method comprising: retraining the target entangled semantic attribute editing direction from a merged latent space for the target semantic attribute, the merged latent space comprising self-correcting latent code samples generated from interpolations along the target entangled semantic attribute editing direction to obtain edited image samples having localized changes corresponding to the target semantic attribute,and wherein the target semantic attribute comprises a minority attribute in a training dataset used to obtain the first latent space. ,

[0105] Statement 2: A method of disentangling a target entangled semantic attribute editing direction from a plurality of semantic attribute editing directions, each of the semantic attribute editing directions being learned from a first latent space of a generative adversarial network (GAN) model for generating edited images and the target entangled semantic attribute editing direction being learned for generating edited images with respect to a target semantic attribute, the method comprising:

[0106] generating self-correcting latent code samples, each self-correcting latent code sample being generated by interpolating a latent code sample from the first latent space along the target entangled editing direction to obtain an edited image sample having localized changes corresponding to the target semantic attribute, wherein the target semantic attribute comprises a minority attribute as represented in the first latent space;

[0107] using the self-corrected latent code samples to define a merged latent space for retraining the target entangled semantic attribute editing direction for the target semantic attribute; and

[0108] relearning the target entangled semantic attribute editing direction using the fused latent space.

[0109] Statement 3: The method of Statement 2, wherein the first latent space comprises an intermediate latent space determined for controlling the generation of the edited images using at least a portion of the plurality of editing directions.

[0110] Statement 4: A method according to Statement 3, wherein the GAN model comprises a model defined in accordance with a StyleGAN technique and the intermediate latent space comprises an IV space in accordance with the StyleGAN technique for generating the edited images.

[0111] Statement 5: A method according to Statement 1, wherein: the GAN model maps latent codes z from a known distribution Z çRd to an image space X çRH^W^S with a mapping function g : ZX; and the first latent space comprises an intermediate latent space IV ç Rd such that the edited images are driven by a function w = M(z) where M is a multi-layer perceptron network, the latent code samples w are in the intermediate latent space IV and each latent code sample w is repeated to provide a latent code sample w+ and fed to the generator layers of a generator G of the GAN model at different resolutions to generate the edited images according to G(w+).

[0112] Statement 6: A method according to Statement 2 comprising inverting each edited image sample to determine the self-correcting latent code samples in IV space.

[0113] Statement 7: A method according to Statement 2 comprising learning the plurality of semantic attribute editing directions including the target semantic attribute editing direction.

[0114] Statement 8: The method of Statement 2, wherein the training dataset comprises images in which the minority attribute is present with a second attribute such that the minority attribute and the second attribute are entangled in the first latent space.

[0115] Statement 9: An apparatus comprising: a generative adversarial network (GAN) model comprising at least a retrained first semantic attribute editing direction in which a first semantic attribute editing direction associated with a first semantic attribute is disentangled from a second semantic attribute editing direction associated with a second semantic attribute, the first semantic attribute editing direction being retrained using a fused latent space comprising self-corrected latent code samples, the self-corrected latent code samples being generated to balance a data distribution representing the first semantic attribute; and an interface for operating the GAN model to to generate edited images using at least the first relearned editing direction.

[0116] Statement 10: Apparatus according to Statement 9, wherein the self-correcting latent code samples are generated from interpolations along the first editing direction as entangled to obtain edited image samples with localized changes corresponding to a minority attribute as represented in a first latent space from which the first editing direction is first learned.

[0117] Statement 11: Apparatus according to statement 10, wherein the at least one relearned first semantic attribute editing direction, the first semantic attribute editing direction and the second semantic attribute editing direction all comprise one of i) state vector machine (SVM) based editing directions or ii) gradient based editing directions.

[0118] Statement 12: Apparatus according to Statement 9, wherein the interface for operating the GAN model is configured to receive an input image for generating an output image, the output image being edited along one or more of a plurality of editing directions, the plurality of editing directions comprising the at least the first learned editing direction.

[0119] Statement 13: Apparatus according to Statement 12 comprising a user interface for receiving the input image and presenting the output image.

[0120] Statement 14: Apparatus according to Statement 12 comprising an effects simulation engine for applying one or more effects to one or both of the input images to the GAN model or to the output image of the GAN model.

[0121] Statement 15: Apparatus according to Statement 14 wherein: the simulation engine comprises a virtual try-on (VTO) engine that simulates an effect of a product or service; and the apparatus comprises one or both of i) an e-commerce interface for purchasing the product or service; and ii) a recommendation engine for recommending the product or service.

[0122] Statement 16: Apparatus according to Statement 15, wherein the product comprises a beauty product.

[0123] Statement 17: A method according to Statement 2 wherein the at least one relearned first semantic attribute editing direction, the first semantic attribute editing direction and the second semantic attribute editing direction all comprise one of i) state vector machine-based (SVM-based) editing directions or ii) gradient-based editing directions.

[0124] Statement 18: A computing device comprising circuitry configured to provide a virtual try-on experience (VTO), the circuitry being configured to provide:

[0125] an editing interface to a generative adversarial network (GAN) model having a plurality of semantic attribute editing directions configured to edit semantic attributes in an input image to produce an edited output image, the plurality including a first semantic attribute editing direction, associated with a first attribute, that is disentangled from a second semantic attribute editing direction,the first semantic attribute editing direction being disentangled by retraining the first semantic attribute editing direction using a fused latent space comprising i) self-correcting latent codes generated to balance a distribution of latent codes associated with the first semantic attribute in the fused latent space and ii) uncorrected latent codes associated with the first semantic attribute used to learn the first semantic attribute editing direction as entangled; ,

[0126] an effects interface to an effects engine configured to simulate an effect on the input image or the edited output image from the GAN model, the effect being associated with a product, a service, or both a product and a service; and

[0127] an e-commerce interface for purchasing products, services, or both products and services that are available for simulation through the VTO experience.

[0128] Statement 19: The computing device of claim 18, wherein the product comprises any one of a makeup product; a hair coloring product; a nail coloring product or a skin, hair or nail treatment product.

[0129] Statement 20: A computing device, system or apparatus comprising one or more processors configured to perform a method in accordance with any of Statements 1 to 8 and 17.

[0130] Statement 21: A computer program product comprising a non-transitory storage device that stores computer-readable instructions for execution by a processor to cause a computing device to perform any of the Method Statements herein.

[0131] Any of the Dependent Declarations may be multi-dependent unless the context requires otherwise. 5. Conclusion

[0132] We study the entanglement problem in the VF space of StyleGAN2 and propose SC2GAN, a simple but effective method that generates self-corrected samples in low-density regions to obtain disentangled orders. With these self-corrected samples added to the original VF distribution, we learn decoupled separation boundaries that enable a disentangled editing. The framework shows strong ability to disentangle attributes with similar separation boundaries and salient channels in the original latent space, and requires only small amounts of data to correct the distribution (e.g., 30 to 1000 codes).

[0133] A practical implementation may include some or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, program products, and other ways, combining the features described herein. A number of embodiments have been described. Nevertheless, it is understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided, or steps may be eliminated, from the described process, and other components may be added to or removed from the described systems. Accordingly, other embodiments fall within the scope of the claims that follow.

[0134] Throughout the description and claims of this specification, the terms "include" and "contain" and variations thereof mean "including but not limited to" and are not intended to (and do not) exclude other components, integers, or steps. Throughout this specification, the singular includes the plural, unless the context otherwise requires. In particular, when the indefinite article is used, this specification is to be understood as contemplating plurality as well as singularity, unless the context otherwise requires.

[0135] Features, integers, characteristics or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood as being applicable to any other aspect, embodiment or example, unless inconsistent therewith. Any features disclosed herein (including the claims, the abstract and the accompanying drawings), and / or any steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of these features and / or steps are mutually exclusive. The invention is not limited to the details of the foregoing examples or embodiments.The invention extends to any new feature, or any new combination, of the features disclosed in this specification (including any accompanying claim, abstract and drawing) or to any new feature, or any new combination, of the steps of any disclosed method or process. REFERENCES .

[0136] (each document below is cited here)

[0137] [1] Rameen Abdal, Yipeng Qin and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 4432-4441, 2019.

[0138] [2] Rameen Abdal, Yipeng Qin and Peter Wonka. Image2stylegan++: How to edit the embedded images? In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 8296-8305, 2020.

[0139] [3] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman et Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. Dans Proceedings of the International Conférence on Learning Représentations (ICLR), 2019.

[0140] [4] David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou et Antonio Torralba. Seeing what a gan cannot generate. Dans Proceedings of the IEEE / CVF International Conférence on Computer Vision, pages 4502- 4511, 2019.

[0141] [5] Volker Blanz et Thomas Vetter. A morphable model for the synthesis of 3d faces. Dans Proceedings of the 26th annual conférence on Computer graphies and interactive techniques, pages 187-194, 1999.

[0142] [6] Andrew Brock, Jeff Donahue, et Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. Dans International Conférence on Learning Représentations, 2019.

[0143] [7] Zikun Chen, Ruowei Jiang, Brendan Duke, Han Zhao and Parham Aarabi. Exploring gradient-based multi-directional controls in gans. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, 23-27 October 2022, Proceedings, Part XXIII, pages 104-119. Springer, 2022.

[0144] [8] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrastive learning. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 5154-5163, 2020.

[0145] [9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.

[0146]

[10] Erik Hârkônen, Aaron Hertzmann, Jaakko Lehtinen and Sylvain Paris. Ganspace: Discovering interprétable gan Controls. Advances in Neural Information Processing Systems, 33:9841-9850, 2020.

[0147]

[11] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou et Alexei A. Efros. Image-to-image translation with conditional adversarial networks. Dans Proceedings of the IEEE conférence on computer vision and pattern récognition (CVPR), juillet 2017.

[0148]

[12] Tero Karras, Müka Aittala, Samuli Laine, Erik Hârkônen, Janne Hellsten, Jaakko Lehtinen et Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852-863, 2021.

[0149]

[13] Tero Karras, Samuli Laine, et Timo Aila. A style-based generator architecture for generative adversarial networks. Dans Proceedings of the IEEE / CVF conférence on computer vision and pattern récognition, pages 4401-4410, 2019.

[0150]

[14] Tero Karras, Samuli Laine, Müka Aittala, Janne Hellsten, Jaakko Lehtinen et Timo Aila. Analyzing and improving the image quality of stylegan. Dans Proceedings of the IEEE / CVF conférence on computer vision and pattern récognition, pages 8110-8119, 2020.

[0151]

[15] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz et Philip Torr. Controllable text- to-image génération. Advances in Neural Information Processing Systems, 32, 2019.

[0152]

[16] Mehdi Mirza et Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 1411.1784, 2014.

[0153]

[17] mrmartin. ffhq-features-dataset. URL: github.com / DCGM / ffhq-features-dataset, 2019.

[0154]

[18] Augustus Odena, Christopher Olah et Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. Dans International Conférence on Machine Learning, pages 2642-2651. PMLR, 2017.

[0155]

[19] Yujun Shen, Ceyuan Yang, Xiaoou Tang, et Bolei Zhou. Interfacegan: Interpreting the disentangled face représentation leamed by gans. IEEE transactions on pattern analysis and machine intelligence, 2020.

[0156]

[20] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans- Peter Seidel, Patrick Perez, Michael Zollhofer, et Christian Theobalt. Stylerig: Rigging style-gan for 3d control over portrait images. Dans Proceedings of the IEEE / CVF Conférence on Computer Vision and Pattern Récognition, pages 6142-6151, 2020.

[0157]

[21] Orner Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik et Daniel Cohen- Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4):1-14, 2021.

[0158]

[22] Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang et Qifeng Chen. High- fidelity gan inversion for image attribute editing. Dans Proceedings of the IEEE / CVF Conférence on Computer Vision and Pattern Récognition, pages 11379-11388, 2022.

[0159]

[23] Zongze Wu, Dani Lischinski, et Eli Shechtman. Stylespace analysis: Disentangled Controls for stylegan image génération. Dans Proceedings of the IEEE / CVF Conférence on Computer Vision and Pattern Récognition, pages 12863-12872, 2021.

[0160]

[24] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiao-gang Wang, Xiaolei Huang et Dimitris N Metaxas. Stack-gan: Text to photo-realistic image synthesis with stacked generative adversarial networks. Dans Proceedings of the IEEE international conférence on computer vision, pages 5907-5915, 2017.

[0161]

[25] Jun-Yan Zhu, Taesung Park, Phillip Isola et Alexei A Efros. Unpaired image-to- image translation using cycle-consistent adversarial networks. Dans Proceedings of the IEEE international conférence on computer vision, pages 2223-2232, 2017.

[0162]

[26] Peihao Zhu, Rameen Abdal, Yipeng Qin, John Femiani et Peter Wonka. Improved stylegan embedding: Where are the good latents? arXiv preprint arXiv:2012.09036, 2020.

Claims

Claims

1. A method for disentangling a target entangled semantic attribute editing direction from a plurality of semantic attribute editing directions, each of the semantic attribute editing directions being learned from a first latent space of a generative adversarial network (GAN) model for generating edited images and the target entangled semantic attribute editing direction being learned for generating edited images with respect to a target semantic attribute, the method comprising: generating self-correcting latent code samples, each self-correcting latent code sample being generated by interpolating a latent code sample from the first latent space along the target entangled editing direction to obtain an edited image sample having localized changes corresponding to the target semantic attribute,wherein the target semantic attribute comprises a minority attribute as represented in the first latent space; merging the self-corrected latent code samples to define a merged latent space for retraining the target entangled semantic attribute editing direction for the target semantic attribute; and retraining the target entangled semantic attribute editing direction using the merged latent space.,

2. The method of claim 1, wherein the first latent space comprises an intermediate latent space determined for controlling the generation of the edited images using at least a portion of the plurality of editing directions.

3. The method of claim 2, wherein the GAN model comprises a model defined in accordance with a StyleGAN technique and the intermediate latent space comprises an IV space in accordance with the StyleGAN technique for generating the edited images.

4. The method of claim 1, wherein: the GAN model maps latent codes z from a known distribution Z ç to an image space X ç with a mapping function g : ZX; and the first latent space comprises an intermediate latent space IV ç Rd such that the edited images are driven by a function w = M(z) where M is a multi-layer perceptron network, the latent code samples w are in the intermediate latent space W and each latent code sample w is repeated to provide a latent code sample w+ and fed to generator layers of a generator G of the GAN model at different resolutions to generate the edited images according to G(w+).

5. The method of claim 1, comprising inverting each edited image sample to determine the self-correcting latent code samples in W space.

6. The method of claim 1, comprising learning the plurality of semantic attribute editing directions including the target semantic attribute editing direction.

7. The method of claim 1, wherein the training dataset comprises images in which the minority attribute is present with a second attribute such that the minority attribute and the second attribute are entangled in the first latent space.

8. The method of claim 1, wherein the at least one relearned first semantic attribute editing direction, the first semantic attribute editing direction, and the second semantic attribute editing direction all comprise one of i) state vector machine (SVM)-based editing directions or ii) gradient-based editing directions.

Images