A method and apparatus for determining and using controllable directions in a GAN space

Abstract

The methods, apparatus, and techniques herein relate to determining directions in a GAN latent space and obtaining untangled control over GAN output semantics, and for enabling use in the generation of synthetic images, such as for use in learning another model or creating augmented reality. The methods, apparatus, and techniques herein utilize, according to embodiments, the gradient directions of an auxiliary network to control semantics within a GAN latent code. It has been shown that about 60 samples can be used as the minimum amount of labeled data, which can be quickly obtained by a human teacher. Also, herein, according to embodiments, it has been shown that more untangled control is achieved by using a mask to select important latent code channels during operation.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims priority to U.S. Provisional Application No. 63 / 392,905, filed on July 28, 2022, the entire content of which is incorporated herein by reference. This application also claims priority to French Patent Application No. FR2210849, filed on October 20, 2022, the entire content of which is incorporated herein by reference.

[0002] The present disclosure relates to image processing using deep neural networks and artificial intelligence, such as for creating augmented reality, including adversarial generative networks (GANs). More particularly, the present disclosure relates to methods and apparatuses for determining and using controllable directions in GAN space.

Background Art

[0003] GANs have been successfully applied to various tasks such as simulating makeup effects on human faces and changing hair colors to provide realistic augmented reality (AR) technology in the beauty industry. Nevertheless, conditional generation models that control output semantics typically require large amounts of labeled training data, which can be costly and time - consuming to obtain. Furthermore, artifacts and entanglements of attributes may appear in current face - editing outputs, which can lead to a reduction in the overall user experience.

[0004] It is desirable to control native GAN output semantics for the purpose of improving the user experience by removing artifacts and disentangling attributes when editing human faces or generating large amounts of synthetic labeled data for learning conditional models.

Summary of the Invention

[0005] The methods, apparatus, and techniques herein relate to determining directions in a GAN latent space and obtaining untangled control over GAN output semantics to enable use in the generation of synthetic images, such as for use in learning another model or creating augmented reality. The methods, apparatus, and techniques herein utilize gradient directions of an auxiliary network to control semantics in GAN latent codes, according to embodiments. It has been shown that about 60 samples can be used as the minimum amount of labeled data, which can be quickly obtained by a human teacher. Also herein, it has been shown that, according to embodiments, using a mask to select important latent code channels during operation results in more untangled control. As will be described in the following description according to embodiments, various aspects are shown and described herein. These and other aspects will be apparent to those of ordinary skill in the art considering the present application as a whole.

[0006] Statement 1: A computing device including a processor and a memory device. The memory device stores instructions that, when executed by the processor, cause the computer device to perform the following. The instructions cause the computer device to provide a generator and an auxiliary network that share a latent space. The generator is configured to generate a synthetic image that exhibits human-interpretable semantic attributes, and the auxiliary network includes a plurality of semantic attribute classifiers including a semantic attribute classifier for each semantic attribute that is controlled to generate a synthetic image from an original image, and each semantic attribute classifier is configured to classify the presence of one of the semantic attributes in the image and provide a significant direction for controlling the one of the semantic attributes in the synthetic image of the generator. The instructions cause the computer device to generate the synthetic image from the original image by applying a respective semantic attribute control for controlling each semantic attribute in the synthetic image. Each of the respective semantic attribute controls corresponds to the significant direction provided by the classifier associated with the semantic attribute.

[0007] Statement 2: The significant direction includes a gradient direction, and the instructions cause the computing device of Statement 1 to calculate a respective semantic attribute control from each gradient direction of each classifier associated with the semantic attribute.

[0008] Statement 3: The instructions cause the computing device of Statement 2 to combine each gradient direction in each classifier associated with the two or more semantic attributes to control the two or more semantic attributes.

[0009] Statement 4: The command causes the computing device of Statement 1 or 2 to unwind each of the semantic attribute operation units for application to the generator.

[0010] Statement 5: Each semantic attribute operation unit includes a respective gradient direction vector calculated from the parameters of a respective semantic attribute classifier associated with each of the semantic attribute operation units, and unwinding one of each of the semantic attribute operation units from another of each of the semantic attribute operation units includes removing important data dimensions of the gradient direction vector associated with the one of each of the semantic attribute operation units from the gradient direction vector associated with the other of each of the semantic attribute operation units, for the computing device of Statement 4.

[0011] Statement 6: The important data dimensions to be removed correspond to a threshold value that identifies those data dimensions having an absolute value greater than or equal to the threshold value, for the computing device of Statement 5.

[0012] Statement 7: If the i-th data dimension for the gradient direction vector associated with the one of each of the semantic attribute operation units is identified by the threshold value, the corresponding i-th data dimension for the gradient direction vector associated with the other of each of the semantic attribute operation units is set to zero, for the computing device of Statement 6.

[0013] Statement 8: The command causes the computing device of any one of Statements 1 to 7 to receive an input identifying at least one of each of the semantic attributes controlled for the original image.

[0014] Statement 9: The input that identifies at least one of each of the semantic attributes includes a granular input for determining the amount of the semantic attributes applied when generating the composite image, for the computing device of Statement 8.

[0015] Statement 10: The instructions cause the computing device of any one of Statements 1 to 9 to cause the generator and the auxiliary network to generate the composite image as a service, provide an e-commerce interface for purchasing a product or service, provide a recommendation interface for recommending a product or service, and provide an augmented reality interface using the composite image to provide an augmented reality experience, any one or more of which.

[0016] Statement 11: A specific semantic attribute among the plurality of semantic attributes includes one of facial features such as age, gender, smile, etc., a pose effect, a makeup effect, a hair effect, a nail effect, a rhinoplasty, a lift, a blepharoplasty, an implant, an otoplasty, a teeth whitening, a teeth correction, etc., a beauty surgery or a teeth effect, and an effect of a fixture including one of an eye fixture, a mouth fixture, an ear fixture, etc., for the computing device of any one of Statements 1 to 10.

[0017] Statement 12: The generator is a GAN-based generator, each classifier is a neural network-based binary classifier, and the generator and each neural network-based binary classifier are co-trained using training images showing semantic attribute labels to define the significant directions in their respective classifiers, on the computing device of any one of Statements 1 to 11.

[0018] Statement 13: A method including learning a generator g of a generative adversarial network (GAN)-based to generate a synthetic image from a source image in which at least one semantic attribute of a defined set of semantic attributes is selectively controlled, the generator including a model that maps a latent code (z) in a latent space (Z) to an image (x = g(z)) in an image space (X) where there are human-interpretable semantics, the learning including co-training the generator g and an auxiliary network including respective classifiers for each semantic attribute of the defined set, each classifier providing a significant data direction used to control its respective semantic attribute when generating an updated image, and further the learning including providing the generator g and the auxiliary network to generate a synthetic image.

[0019] Statement 14: The method of Statement 13 includes calling the generator g using at least one control calculated from the parameters of the auxiliary network to generate a plurality of synthetic images having semantic attributes selected from the defined set, the auxiliary network providing the significant data direction for each of the at least one control.

[0020] Statement 15: The method of Statement 13 or 14, including learning a further network model using at least some of the plurality of synthetic images.

[0021] Statement 16: A method comprising generating a synthetic image g(z’) from an original image using a generator g having a latent code z, wherein the generator manipulates a target semantic attribute k in the synthetic image, and said generating is in a significant direction identified from an auxiliary network that classifies each semantic attribute including the target semantic attribute k at z, discovering a significant direction at z with respect to the target semantic attribute k, defining z’ by optimizing z according to the significant direction with respect to the target semantic attribute k, and outputting the synthetic image.

[0022] Statement 17: The method of Statement 16, wherein the auxiliary network includes a set of binary classifiers, one for each semantic attribute manipulable by the generator, and the auxiliary network is co-trained to share the generator g and the latent code space Z.

[0023] Statement 18: Each of the semantically significant data directions includes a respective dimensional data vector obtained from each of the individual classifiers, each vector including the direction and rate of the fastest increase in the respective individual classifier, of the method of Statement 16 or 17.

[0024] Statement 19: The method of any of Statements 16 to 18 includes unraveling the data direction by filtering out important dimensions of the semantically significant data directions of each other semantic attribute (m, m≠k) associated with the target semantic attribute from the dimensions of the semantically significant data direction of the target semantic attribute, where a particular dimension is greatly affected by the magnitude of its absolute gradient.

[0025] Statement 20: The filtering includes evaluating each dimension of each semantically significant data direction unwound from the target semantically significant data direction, and setting the value of the corresponding dimension in the target semantically significant data direction to a zero value when a specific dimension exceeds a threshold, the method of Statement 19.

[0026] Statement 21: The latent code z is optimized as z’ = z + αn z k' where n z k is a vector representing the target semantically significant data direction before filtering, and n z k' is a vector representing the target semantically significant data direction after filtering, and α is a hyperparameter controlling the interpolation direction and step size, the method of Statement 20.

[0027] Statement 22: The method includes repeating the discovery, definition, and optimization operations with respect to the latent code z’ in order to further manipulate the semantic attribute k, the method of any of Statements 16 to 21.

[0028] Statement 23: The method includes generating a synthetic image to manipulate a plurality of target semantic attributes, discovering each respective significant direction in z for each of the target semantic attributes as identified from the auxiliary network that classifies each of the target semantic attributes in z, and optimizing z according to each of the significant directions, the method of any of Statements 16 to 22.

[0029] Statement 24: The method includes learning another network model using the synthetic image, the method of any of Statements 16 to 23.

[0030] Statement 25: The method according to any one of statements 16 to 24, comprising receiving an input for identifying the target semantic attribute controlled for the original image.

[0031] Statement 26: The method according to statement 25, wherein the input includes a detailed input for determining the amount of the target semantic attribute applied when generating the composite image.

[0032] Statement 27: The method according to any one of statements 16 to 26, including any one or more of: providing the generator and the auxiliary network to generate the composite image as a service; providing an e-commerce interface for purchasing a product or service; providing a recommendation interface for recommending a product or service; and providing an augmented reality interface using the composite image to provide an augmented reality experience.

[0033] Statement 28: The target semantic attribute includes one of: facial features such as age, gender, smile, etc.; pose effect; makeup effect; hair effect; nail effect; cosmetic surgery or dental effect including one of rhinoplasty, lift, blepharoplasty, implant, otoplasty, tooth whitening, orthodontics, etc.; and appliance effect including one of eye appliance, mouth appliance, ear appliance, etc., according to any one of statements 16 to 27.

[0034] Statement 29: A method including steps of providing an augmented reality (AR) interface to provide an AR experience. The AR interface is configured to generate a synthetic image from a received image using a generator by applying respective semantic attribute manipulation units for controlling respective semantic attributes in the synthetic image, and each of the respective semantic attribute manipulation units corresponds to a significant direction provided by a classifier associated with the semantic attribute. The method includes steps of receiving the received image and providing the synthetic image for the AR experience.

[0035] Statement 30: The method includes processing the synthetic image using an effect pipeline to simulate an effect, and providing the synthetic image having the simulated effect for presentation on the AR interface, of the method of Statement 29. The method of Statement 29 or 30 can be combined with, for example, any one of the subjects of Statements 1 to 13 or 16 to 28 (subjects adapted as applicable).

[0036] A computing device including at least one processor and at least one non-transitory storage device storing computer-readable instructions for execution by the at least one processor, the instructions causing the computing device to perform any one method of Statements 13 to 30.

[0037] A computer program product including at least one non-transitory storage device storing computer-readable instructions for execution by at least one processor of a computing device, the execution of the instructions causing the computing device to perform any one method of Statements 13 to 30.

Brief Description of the Drawings

[0038]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] This concept is best described through its specific embodiments described herein with reference to the accompanying drawings, where like reference numerals throughout refer to like features. The term "the present invention," as used herein, is intended to mean the inventive concept underlying the embodiments described below and is not meant to refer simply to the embodiments themselves. Further, the general inventive concept is not limited to the exemplary embodiments described below, and it should be understood that the following description should be read from such a perspective.

DETAILED DESCRIPTION OF THE INVENTION

[0049] GAN models can generate highly realistic images when given random inputs from their latent spaces, but the generation process is typically a black box, so the output semantics cannot be directly controlled. Nevertheless, previous studies [VB20, VB20, SZ21, HHLP20, SYTZ20] have shown that significant directions and channels exist within the GAN latent space, and linearly interpolating in these directions or varying individual channel values can lead to interpretable transformations, such as adding glasses or a smile to a human face. The aims of the methods, apparatuses, and techniques herein relate to determining (e.g., learning) directions in the GAN latent space and obtaining disentangled control over the GAN output semantics, enabling, for example, using such learned directions to create augmented reality or generating synthetic images for model learning. Herein, the focus is on human face editing tasks useful in the beauty industry and other industries. For this purpose, the methods, apparatuses, and techniques herein, according to embodiments, utilize the gradient directions of auxiliary networks to control the semantics in the GAN latent code. It has been shown that about 60 samples can be used as the minimum amount of labeled data, which can be quickly obtained by human teachers. Also, according to embodiments, it is shown herein that using a Grad-CAM-based mask during operation to select important latent code channels can lead to more disentangled control. In the following sections, prior research in related fields is reviewed to contextualize the teachings herein.

[0050] [GPAM +The Generative Adversarial Network (GAN) introduced in

[14] currently dominates the field of generative modeling due to its powerful ability to synthesize photo-realistic images. Generally, a GAN consists of two networks: a generator that learns the mapping from its latent space to the image space, and a discriminator that distinguishes GAN-synthesized images from real images. These two networks are jointly trained in an adversarial manner. To improve the fidelity of the output and the stability of learning, various variants of GANs have been proposed [ACB17, BDS18, CCK + 18, KLA18, KLA + 20], and there is a growing interest in studying the latent spaces of these native GAN models that can control the generation process more finely.

[0051] The potential semantic operations of GANs have been studied by multiple prior works [VB20, VB20, SZ21, HHLP20, SYTZ20, PBH20], and it has been shown that there are significant directions and channels in the GAN latent space / GAN feature space. One of a series of studies has discovered the control of the output semantics of unconditional GANs with explicit teachers [SYTZ20, WLS21]. For example, InterFaceGAN [SYZT20] assumes that for any binary semantics (e.g., male vs. female), there exists a hyperplane in the latent space that functions as a separating boundary, and its normal vector represents a significant direction. A classifier pre-trained on the CelebA dataset [LLWT15] is used to generate pseudo-labels for GAN-synthesized images, and then the separating boundary is learned using state vector machines (SVMs) learned with paired data of GAN latent codes and corresponding semantic labels. [WLS21] discovers channels that control the positioning effect in the GAN activation space with the guidance from a pre-trained semantic discrimination network. However, these teacher-aided methods require a large amount of labeled data and pre-trained neural networks, and the learned controls sometimes become intertwined, so the scope of control found by these teacher-aided methods is usually limited.

[0052] Another series of studies has been discovering such controls in a self-supervised or unsupervised manner [HHLP20, CVB21, VB20]. For example, GANSpace [HHLP20] identifies important latent directions by applying principal component analysis (PCA) to vectors within the GAN latent space or feature space, and [VB20] discovers interpretable directions within the GAN latent space by jointly optimizing a direction and a reconstructor that restores these directions and manipulation strengths from the images generated based on the manipulated GAN latent code. These methods do not require labeled data, but they typically require extensive manual inspection of various manipulation directions and the identification of significant controls. More importantly, unlike the techniques taught according to the embodiments herein, control over the target semantics is not necessarily guaranteed.

[0053] Gradient-based knowledge has been utilized to improve the interpretability [ZKL + 16, SCD + 17] or learning stability [WDB + 19] of neural networks. For example, [SCD + 17] obtains a location-specific map, which is visual evidence within an image, by utilizing gradient information from the classification output. [WDB + 19] proposes enhancing the stability of the learning behavior of a GAN by adding an additional step before jointly optimizing the generator and the discriminator. In this additional step, first, the latent code is initially optimized towards regions that are regarded as more real by the discriminator, and the above direction is obtained by calculating the gradient with respect to the latent code.

[0054] [WDB +In contrast to

[19] , according to the embodiments of this specification, although not seen in the literature so far, the auxiliary binary classifier gradient is utilized to obtain control over the GAN output semantics. Thus, according to this embodiment, methods, apparatuses, and techniques for obtaining relaxed control in the GAN latent space are described in this specification. These methods, apparatuses, and techniques first find the semantically significant directions by calculating the gradients of a binary classifier that scores various semantics given a latent code input, and then select important dimensions within the latent code for attribute relaxation during operation. [Determination of Semantically Significant Directions]

[0055] FIG. 1A is an image display of a computing device 100 in one embodiment, including a CNN architecture 102 and an image output (e.g., image 104 including 104A and 104B) stored in a storage device 106 of the device 100. FIG. 1B is an image display of a part of the storage device 106 that stores the dimensions of respective gradients 132, 130 in semantic attributes k and m in one embodiment. This image shows that the dimensions of the gradient of attribute m are uncoupled from the corresponding dimensions in the gradient of attribute k. FIG. 1C is a flowchart of the operation 150 of one-step optimization, which will be further described below in this specification.

[0056] The CNN architecture 102 includes a controllable GAN g 108 configured to generate output images g(z) 104A and g(z’) 104B from respective latent codes z and z’ (e.g., each part of the latent space Z of the GAN g 108). The images g(z) 104A and g(z’) 104B show face images with various semantic attributes. In the illustrated example, the face of the individual in the image g(z) 104A has no smile, and the face of the same individual in the image g(z’) 104B has a smile.

[0057] The CNN architecture 102 further includes a plurality of auxiliary networks 110, one for each semantic attribute controlled by the GAN g 108. Each of the plurality of auxiliary networks 110 includes a respective mapping function f, and these auxiliary networks 110 are learned together with the GAN g 108 to discover the significant directions for each of the controlled semantic attributes. In one embodiment, the mapping function f includes a binary classifier. FIG. 1A shows, for simplicity, only the details within the associated box 112 for one of each of the classifiers from the auxiliary network 110, namely the mapping function f for the semantic attribute k. k In FIG. 1A, each of the GAN g 108 and the auxiliary network 110 is pre-trained with training images (e.g., 60 images for each semantic attribute) for each semantic attribute. As shown in box 112, a mapping function f that maps the GAN latent code (e.g., z) to the semantic score for k is used to manipulate the semantic attribute k (e.g., a smiling face) with respect to the GAN latent code z. k The gradient direction n z k from 114 is used as an input to derive a modified z (e.g., z’) of the GAN g 108. The equation for deriving the modified z will be further described below.

[0058] Next, refer to FIG. 1B. According to one embodiment, the semantic attributes are untangled. The important dimensions for manipulating the semantic attribute m are removed with respect to the attribute k. According to one embodiment, the important dimension is the dimension with a large gradient magnitude at 130 for the semantic attribute m. Such a dimension has its gradient direction n z m masked (e.g., set or defined) to a value of 0 at 112 for the semantic attribute k, and the gradient direction n z k 132 is the gradient direction n after masking (e.g., after setting the important attribute to zero). z k' z k ​112 is shown. The dimensional mask will be further described below, particularly for those with a large gradient magnitude.

[0059] Next, referring to FIG. 1C, operation 150 shows a pipeline for one-step optimization. The operation starts at 152 using a latent code z such as a face image. At 154, a significant direction in z with respect to the semantic attribute k is discovered (e.g., the gradient is calculated from the parameters of each learned mapping function f k ). The determination at 156 evaluates whether there is an entanglement with any semantic m (where m represents a semantic attribute different from k controlled by the GAN g). If so, branching to 158 via "Yes", a significant direction in z with respect to the semantic attribute m is discovered (e.g., calculated from the parameters of each learned mapping function f m ). At 160, as described with reference to FIG. 1B and further described below in this specification, the operation performs disentanglement. In one embodiment, the determination of entanglement as in 156 is identified experimentally. Entanglement was observed by editing each attribute using a small meta-validation set of 30 images following the original gradient direction of k and then determining which attributes are entangled. The associations between the target attribute and any entangled attributes were tabulated. The important attributes from each of the identified attributes (i.e., attributes m≠k) are excluded by the mask. In one embodiment, for disentanglement, the combination of the top channels of the attributes (e.g., 100 from those with a large gradient magnitude) is determined. And these dimensions are set to 0 in the target direction (e.g., to generate n z k' ).

[0060] In one embodiment, for example, without prior observational experiments, directly, depending on the magnitude of the value of the i-th dimension in the gradient n z m z kThe value of the i-th dimension in [context] can be excluded by a mask. In the first case, when there are entanglements, excluding those dimensions by a mask helps in disentangling. In the second case, when there are no entanglements and the important dimensions of m are not important in k, excluding them by a mask does not change the direction significantly.

[0061] In 162, the operation optimizes the latent code z. The operation can be repeated (e.g., in a loop to step 154) using the next instance of the latent code z until the desired operation result is achieved. Referring back to step 156, in one embodiment, when there are no entanglements respectively, the operation proceeds to step 162 via the "no" branch and optimizes the latent code z. As described above, in one embodiment, the operation may exclude the important dimensions of any other attribute by a mask even when there are no entanglements. For example, after determining which attributes of m are entangled, it may be easier (e.g., programmatically) to simply disentangle the important dimensions from all m≠k rather than disentangling the important dimensions from the gradients of k.

[0062] FIG. 2 is a table 200 of images including rows 202, 204 of a table to show the disentangling of a smiling face from glasses. The first (left) column 206 in both rows 202, 204 is the original image. The second (center) column 208 is edited with a smaller interpolation distance, and the third (right) column 210 is edited with a larger interpolation distance. The difference is that in the second row when the distance is larger (i.e., when more smiling faces are added compared to column 208), glasses are not added. That is, disentangling is applied to the second row. Thus, row 202 shows the respective face images of the same subject in each image. In order from left to right, the subject is i) without a smiling face and glasses, ii) with a smiling face but without glasses, iii) with a smiling face and glasses. In row 204, the images show the same subject as shown in row 202. The images in row 204 show the application of disentangling that is not applied in row 202. In order from left to right, the subject is i) without a smiling face and glasses, ii) with a smiling face but without glasses, iii) with a smiling face but without glasses.

[0063] During the one-step optimization operation, as described with reference to FIGS. 1B and 1C, the important dimensions in the gradient from f 眼鏡 are masked to 0 at n z 笑顔 . |n z k | i The term indicates the absolute value of the gradient vector in dimension i. Contrary to the original multiclass classification setting of Grad-CAM where ReLU is selected as the activation, here the absolute value is used as in the binary classification setting. Dimensions that have an adverse effect on the semantic score also contain significant information.

[0064] More specifically, according to one embodiment, a method for determining (e.g., learning) a significant direction in the GAN latent space to manipulate the semantics in the GAN output is provided. A well-trained GAN model learns a mapping g that maps a d-dimensional latent code z (in the latent space Z of the GAN, e.g., z ∈ Z, Z ⊆ R d ) to an image x = g(z) in the image space X. Human-interpretable semantics exist within the image space X and include, for example, age, gender, glasses, smile, the pose of a person's head, or other semantic observations. Given a series of scoring functions s1···s k for K semantics, K mappings from the latent space Z to the semantic spaces C1···C k ⊆ R are obtained respectively. Here,

Number

[0065] s1···s k Given the accuracy and independence of, when the k-th semantic attribute of the output image changes, its score c z k will change accordingly, but it is expected that the other semantic scores will remain approximately the same. There is a hypothesis that useful information is embedded in the mapping from the GAN latent space to the semantic space, and this can be utilized to find the semantically significant direction. In particular, to control c z k it is proposed to interpolate the latent code according to the gradient direction of such a mapping function. To simplify the calculation, each mapping function s k (g(z)) is parameterized using a neural network f k learned on pairs of samples of the GAN latent code and the corresponding semantic labels generated by the original s k (g(z)) (for example, the scoring function of the ground truth / human recognition). The direction is calculated as follows.

Equation

[0066] [Relaxation of the attribute during operation] According to one embodiment, a method and / or technique for minimizing the entanglement of semantic attributes is provided. Entanglement of attributes may occur (e.g., sometimes) during the interpolation of the latent code following the direction found as described above. As used herein, as would be understood by those skilled in the art, two or more semantic attributes are entangled when, during interpolation, the manipulation of one semantic attribute affects one or more others in two or more attributes. It has been observed that non-target semantics may change due to interpolation along the original direction, but such effects can sometimes be removed by randomly excluding dimensions of the direction vector used for interpolation. Thus, among the discovered d-dimensional direction vectors, it is assumed that only some dimensions cause changes in the target semantic attributes, and the other dimensions exhibit biases learned from the training data. For example, in the direction of increasing a person's age, glasses appear during the operation because glasses and age are correlated in the data.

[0067] Grad-CAM[SCD + 17] is a class-discriminative localization technique that uses gradient information to provide visual explanations for CNN-based models. According to the methods and / or techniques herein, dimensions are removed by filtering based on the magnitude of the gradients from the semantic scoring function. This results in more relaxed control. In particular, Grad-CAM measures the importance of neurons as follows.

Equation

Equation

[0068] According to an embodiment, nz k is regarded as the only activation map, and the importance of the i-th dimension is calculated as follows.

Number

[0069] According to the definition of the gradient, the value of n in the i-th dimension is caused by a small change in z in the same dimension. z k of c. Intuitively, a large n z k where L is large i k has a greater impact on c, and a small n z k where L is small z k has a lower relevance. However, such irrelevant dimensions may have a large gradient magnitude when calculating the control for another semantic attribute n i k (m ≠ k). Therefore, when these dimensions change even slightly when optimizing the k-th semantics, it may affect c z k z m z m i k and cause problems with the entanglement of attributes. Therefore, according to one embodiment, dimensions in the gradient for any k-th semantic attribute with an L not less than a specific threshold are regarded as important, while any important dimensions when predicting semantic attributes that appear to be combined with the target from the target direction are excluded. In one embodiment, when k is entangled with m and they share important dimensions, even important dimensions of k can be excluded by a mask. i k Figure 3 illustrates, in the form of pseudo-code, the operations for calculating a new direction in which attributes are unraveled in one embodiment. In line 1, the target semantics is the direction n

[0070] Figure 3 illustrates, in the form of pseudo-code, the operations for calculating a new direction in which attributes are unraveled in one embodiment. In line 1, the target semantics is the direction nz k is described as being associated with another semantics. In line 2, a scoring function f for the associated semantics m m and a threshold of t are described as being established to identify important gradients (i.e., those with large gradient magnitudes). In line 3, the gradient of the associated semantics m (n z k ) is determined from their respective functions. In line 4, an investigation is made to compare each of the i-dimensional elements L i m with t. Here, the element L i m = |n z m | i . In line 5, or, for the gradients identified by the threshold, the magnitude of the corresponding gradient in the dimension of k is set to 0. That is, n z k [i] = 0 in i ∈ E, and, in line 6, the final form of n z k is returned (e.g., provided).

[0071] At each step, to increment the semantic score c z k by one through the unraveling of the m-th semantics, z is updated according to the following equation in which n z k' is recomputed according to operation 300 of the pseudo-code algorithm in FIG. 3. [Equation] For example, it will be understood that the operation can loop and repeat such that z’ is calculated more than once to further interpolate along the direction of the target semantic attribute until an image with desired characteristics (e.g., g(z’)104B) is generated.

[0072] [Details of Embodiment] According to one embodiment, the GAN part (e.g., GAN g 108) of the network architecture structure 102 adapts the structure of StyleGAN2 [KLA + 20] as pre-trained, for example, on the FFHQ dataset [KLA + 20]. The optimization is performed in the W space of StyleGAN2 [KLA k 20]. 400 images are generated by StyleGAN2, and 30 images are manually selected from them as positive / negative samples for each target semantic, and 10% of the selected pairs are used for evaluation. Multiple binary classifiers (e.g., each instance of 110) are trained in a multi-label learning manner simultaneously for different target semantics, minimizing the sum of all binary cross-entropy losses. According to one embodiment, each classifier (e.g., f z m ) includes two fully connected layers with hidden layers of size equal to 16, ReLU activation, and sigmoid activation at its output neuron. Regarding the filtering threshold t, it has been confirmed by experiments etc. that it actually works well in the channel where the magnitude of the gradient in n

[0073] is the 100th largest. For the α value, 0.4 was used in all experiments.

[0073] Then, in one embodiment, to remove the dimensions important for the semantic attribute m from the gradient vector n z k , m is combined with k, and the gradient vector n z mis determined. Each dimension of the vector is evaluated (e.g., a sorting format can be executed) using the absolute value of the dimension to determine the 100th largest value. This 100th largest value becomes the threshold t according to m during filtering. If each i-th dimension in the vector of m is greater than or equal to the 100th largest value, the corresponding i-th dimension is set to 0 in the vector of k. For each semantic attribute m associated with k, the gradient vector of m is determined and evaluated against t in a similar manner, and then each dimension in the gradient vector of k is masked.

[0074] [Results and Discussion] The evaluation results are presented for the disclosed methods, systems, and techniques (e.g., according to the described embodiments, on the FFHQ dataset [KLA18] of the network architecture 102 pre-trained on StyleGAN2 [KLA + 20]). Figure 4 is a table of images showing the manipulation results 400 for five attributes: age, pose, gender, glasses, and smile. These qualitative examples are edited using the learned directions learned with 60 samples for each attribute. Figure 5 is a table of images 500 showing the disentanglement of attributes. Figures 6 and 7 are tables of images 600 and 700 showing the comparison results with other studies [SYTZ20, WLS21, HHLP20], respectively.

[0075] Table 400 shows the manipulation results for five different attributes, i.e., the manipulation results for a single attribute. For each group of three samples within a row (e.g., 402), the central image is the original image synthesized by StyleGAN2, and the left / right images correspond to the synthesis results based on the original latent code edited in the negative / positive directions found by the method or technique disclosed herein (i.e., as the output from the network architecture 102 via (GAN g 108)).

[0076] This approach works well in both negative and positive directions for all attributes. Specifically, for the age attribute, it has been found that not only can fine aging effects such as wrinkles and acne be created / removed, but the face shape can also be changed at a high level while maintaining the person's identity well. This approach can also add glasses with little change to unrelated semantics. This means that many realistic image samples with glasses can be created despite the absence (i.e., lack) of such semantic attributes in the original FFHQ training data.

[0077] Table 500 shows the results of disentangling the attributes of Figure 5. For each group of three images in each row (e.g., 502, 504, 506, and 508), the first image on the left (e.g., 502A, 502D, 504A, and 504D) is the original image synthesized by StyleGAN2. The adjacent left and right images (502B / 502C, 502E / 502F, 504B / 504C, and 504E / 504F) correspond to the synthesis results (via GAN g 108) based on the original latent code edited according to the negative / positive directions found by the method or technique of this specification (e.g., using relatively small and large interpolation distances). The first two rows 502 and 504 show that the aging direction is intertwined with the glasses direction. By excluding the important dimensions of the glasses classification from the aging direction during optimization, the two attributes are successfully disentangled (e.g., row 504), and similar aging results are achieved. Similarly, for the last two rows 506 and 508, the approach of excluding important dimensions as disclosed in this specification has been successful in finding a new direction where the gender direction is less intertwined with the smiling face.

[0078] The controls found by the methods or techniques disclosed in this specification are generally independent, but there are still failure cases where manipulating one semantic affects another semantic. It was noticed that the most common entanglements are glasses with age and smiles with gender. This result shows that by excluding important dimensions that cause changes in the logits of entangled attributes during optimization, a slightly different direction can be found where irrelevant semantics are less affected and the target semantics are still successfully manipulated.

[0079] [Comparison] The results from the methods and / or techniques taught in this application can be compared with InterFaceGAN [SYTZ20] and GANSpace [HHLP20], for example, for three semantic attributes (smile, gender, and age). Attributes not supported by some of the known methods and implementations are not shown. In [SYTZ20], the SVM boundary was used with the same training data used to learn the gan g and the auxiliary network in Figure 1. In [HHLP20], the officially published directions were used. Table 600 in Figure 6 visualizes the directions found by each of the three implementations, the first two being known, and the bottom row showing the results according to the embodiments of this specification (e.g., using the network architecture 102 with GAN g 108 and auxiliary network 110). For each group of three samples within a row, the central image is the original image synthesized by StyleGAN2, and the left and right images correspond to the synthesis results based on the original latent code edited by the directions found by each method to decrease / increase the target semantic score.

[0080] Overall, the method disclosed herein is superior to the teacherless GANSpace method that generates less realistic images with non-obvious changes in the target semantics and finds a direction similar to InterFaceGAN. Nevertheless, the method disclosed herein is superior in unraveling attributes by channel filtering inspired by Grad-CAM compared to the conditional operation technique proposed in [SYTZ20] that adjusts the target direction by subtracting its projection vector in the direction for editing intertwined semantics. Some examples are shown in FIG. 7.

[0081] Table 700 (FIG. 7) shows a comparison of the attribute unraveling results for the method disclosed herein and only one of the conventional known methods (InterFaceGAN [SYTZ20]). The method disclosed herein better unravels the attributes during operation. In the example of the gender-smile entanglement in the first two rows, in the second of such rows (the method disclosed herein), the smile attribute is better preserved when editing the gender. In the example of the age-glasses entanglement in the bottom two rows, the method disclosed herein finds a better unraveled control of preserving irrelevant semantics such as, for example, the expression and hairstyle of a person's face.

[0082] An example of the use of GANS is shown and described in U.S. Patent Application No. 16 / 683,398, filed on November 14, 2019, entitled "System and Method for Augmented Reality by translating an image using Conditional cycle-consistent Generative Adversarial Networks (ccGans)", which is incorporated herein by reference.

[0083] In one embodiment, the disclosed techniques and methods include developer-related methods and systems that (optionally, such as by conditioning) define a CNN architecture that includes a GAN g and an auxiliary classifier f for classifying semantic attributes. The GAN is controllable, by learning, to interpolate semantic attributes. Also shown are user-related methods and systems such that a learned generator model (e.g., generator g(x)) is used at run-time to process an original image x for image-to-image conversion to obtain a synthetic image x' = g(x) that has (or does not have) a particular semantic attribute.

[0084] FIG. 8 is a block diagram of a GUI 800 in one embodiment. The GUI 800 can be displayed via a display device (not shown). In one embodiment, a user can specify a target semantic attribute and an amount of the desired attribute. For example, + smiling face, - age can identify the attribute, and an amount such as a percentage can specify the amount to be given to the synthetic image when the generator is called. Since the interpolation process is continuous, a pre-trained image classifier can be used to quantify the amount (or amount of change) of the semantic attribute and find the desired operation intensity.

[0085] The GUI 800 shows a plurality of input operation units 802 provided to receive selection inputs for each target semantic attribute in which the CNN architecture 102 is configured and learned. In the present embodiment, each input operation unit (for example, 802A) shows a slider type operation unit for each single attribute and discriminates fine inputs. Other types of input operation units (such as radial operation units, text boxes, buttons, etc.) may be used for fine inputs or other inputs. The operation unit can receive % values, range selections, scalars or other values. For example, the age attribute is associated with an integer range and can receive fine inputs approximating ages such as in years or decades. Others can select a relative range of the abundance of the attribute (for example, small, medium, large). These relative ranges can be associated with fine values such as 15%, 50% and 85% and other values.

[0086] By the selection of the "Apply" operation unit 804, the semantic attribute input is applied when calling the generator. The output of the generator is controlled by the semantic attribute operation unit derived from the semantic attribute classifier of each auxiliary network. The semantic attribute input is applied to the original image x (806) through these attribute operation units to generate a synthesized image x' 808 (for example, an output image). The synthesized image is controlled by the attribute operation unit and can be expressed as x' = g(x). And the attribute operation unit corresponds to the semantic attribute input from the interface operation unit 802. All available semantic attributes have operation units, but the user can choose not to change the attributes, so the generator does not need to interpolate along the direction identified within the relevant semantic operation unit.

[0087] The original image 806 can be specified for use (for example, uploaded, copied from storage, obtained through camera input to obtain a self-taken photo, etc.) through the "Original Image" operation unit 810. The obtained synthesized image x' 808 can be saved through the "Image Save" operation unit 812.

[0088] In one embodiment, before applying the manipulation unit when the generator is called, the attributes are loosened so that the composite image is generated with minimal entanglement. In one embodiment, the loosening can be enabled or disabled via one or more manipulation units (not shown). For example, in one embodiment, age and glasses can be selectively loosened, or smile and gender can be selectively loosened, or both such entanglements can be loosened.

[0089] In the illustrated embodiment, a separate semantic attribute manipulation unit is provided for each learned attribute, but in one embodiment (not shown), fewer manipulation units (e.g., for only a single attribute or only two attributes, etc.) are provided. Separate manipulation units are provided for individual attributes, but in one embodiment (not shown), a single manipulation unit can be provided for combined attributes (e.g., age and gender). The multi-attribute manipulation unit is calculated by vector operations. To create a face with fewer smiles, more glasses, and an older (higher age) appearance, the GUI is configured to receive inputs in each direction, namely, -smile, +glasses, +age. According to one embodiment of the operation of the computing device, an input is associated with each gradient vector (of the relevant semantic attribute manipulation unit), and the vectors are added and normalized. The generator interpolates (linearly) along the calculated direction (i.e., the combined direction) to generate an output image.

[0090] In one embodiment, the GUI is provided by a first computing device, and the generator is provided by another computing device remotely located with respect to the first computing device. Such another computing device can provide a controllable GAN generator as a service. The first computing device that provides the GUI is configured to communicate an input image and a semantic attribute direction input (e.g., a percentage value or other input, etc.) to the remotely located computing device that provides (e.g., executes) the generator. Such a remotely located computing device can provide an application programming interface (API) or other interface for receiving the original image (or selection) and the semantic attribute direction input. The remotely located computing device can calculate a direction vector and apply a semantic attribute operation unit to call the generator. In an alternative embodiment, the computing device that provides the GUI can be the same as the computing device that provides the generator.

[0091] In one embodiment, in addition to the aspects of the computing device of the developer (e.g., used for learning time) and the target (used for inference time), when instructions stored in a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.) are executed, those skilled in the art will understand that an aspect of a computer program product is disclosed that causes a computing device to perform any of the method aspects disclosed herein. In one embodiment, the computing device includes a processor (e.g., a microprocessor (e.g., a CPU, a GPU, any plurality of the same), a microcontroller, etc.) that executes computer-readable instructions such as those stored in the storage device. In one embodiment, the computing device includes a circuit (e.g., "purpose built") that executes the functions of the instructions without, for example, the need to read such instructions.

[0092] Aspects related to an e-commerce system are also shown and described. In one embodiment, a user's computing device is configured as a client computing device with respect to the e-commerce system. The e-commerce system stores, for example, a computer program for such a client computing device. Thus, the e-commerce system has, as a component thereof, a computer program product that stores instructions that, when executed by a client computing device (e.g., its processing unit), configure such a client computing device. These and other aspects will be apparent.

[0093] FIG. 9 is a block diagram of a computer system 900. In one embodiment, the computer system 900 includes a plurality of computing devices, and in one embodiment, the plurality of computing devices include servers, developer computers (PCs, laptops, etc.), mobile devices such as smartphones and tablets, and the like. A network model learning environment 902 is shown that includes hardware and software for defining and configuring a GAN-based CNN architecture 102 by conditioning and the like. The architecture 102 includes a GAN-based generator 108 (e.g., generator g) and an auxiliary network 110 of classifiers (e.g., a plurality of mapping functions f). In one embodiment, the GAN and the mapping function are jointly conditioned, and the latent space z of the GAN is updated based on the results of the semantic function as described above.

[0094] In one embodiment, the CNN 102 that combines the generator 108 and the auxiliary network 110, once trained, is provided for use on a target device such as one of the mobile devices 910, 912, or 913 of the system 900, or other devices. The mobile device 910 is, for example, a representative user computing device of a consumer user. It is understood that such a user may also use other forms of computing devices such as desktop computers, workstations, etc. The device 913 represents a computing device (and thus a learning data generation device) for performing learning data generation in advance. In this embodiment, such a computing device employs the generator 108 to generate additional image data. This additional image data can be easily labeled with semantic attributes and can be used (e.g., in a supervised manner) to train the network model. The form factor of the device 913 can be a server, laptop, desktop, etc., and does not need to be a consumer-type mobile device such as a tablet, smartphone, etc.

[0095] In one embodiment, the network model learning environment 902 employs, at least in part, a GAN model (generator 108) pre-trained for an image task (e.g., face image generation). The generator 108 is pre-trained, for example, by using an image dataset 914 stored in a data server 916. In one embodiment, the generator 108 is a model "developed in-house". In one embodiment, the generator 108 is publicly available, such as via an open source license. The dataset is similarly developed or available. Depending on the image task and the type of (e.g., supervised) network architecture, the learning is supervised, and the dataset is annotated according to such learning. In other scenarios, the learning is unsupervised, and the data is defined accordingly. In one embodiment, the GAN model (generator 108) is further conditioned in the form shown in FIG. 1A by using, for example, face images labeled for semantic attributes.

[0096] In one embodiment, the generator 108 and the auxiliary network 110 are incorporated into an augmented reality (AR) application 920. Although not shown, in one embodiment, the application is developed using a computing device for application developers for a specific target device having specific hardware and software, particularly an operating system configuration. In one embodiment, the AR application 920 is a native application configured for execution in a specific native environment, such as one defined for a specific operating system (and / or hardware). In one embodiment, the AR application 920 takes the form of a browser-based application configured to execute, for example, in the browser environment of the target device.

[0097] In one embodiment, the AR application 920 is distributed (e.g., downloaded on the user device) to user devices such as mobile devices 910 and 912. Native applications are often distributed via an application delivery server 922 (e.g., a "store" operated by a third-party service), but this is not essential.

[0098] In one embodiment (not shown), the AR application 920 does not include the CNN architecture 102 itself (including the generator and the auxiliary network). Rather, the application is configured with an interface for communicating with a remote device that provides these components as a service (not shown) (e.g., as a cloud-based service). Storing and executing the generator and the auxiliary network requires a large amount of resources and may be too large / demanding for some computing devices. Other reasons may also affect the paradigm of the AR application.

[0099] In one embodiment, the AR application 920 is configured to provide an augmented reality experience to the user (e.g., via an interface). For example, the image is effected by the processing of the generator 108. The mobile device includes a camera (not shown) for taking an image (e.g., a still image or a moving image, whether a self-shot image or not). In one embodiment, the effect is applied to the image (e.g., a moving image) in real time (and displayed on the display device of the mobile device) to simulate the effect on the user during video shooting. When the position of the camera is changed, the effect is applied according to the image of the video being shot, simulating augmented reality. As understood, real-time operation is restricted by processing resources. In one embodiment, the effect is delayed instead of being simulated in real time, which may affect the augmented reality experience.

[0100] In one embodiment, the computing device is coupled to communicate via one or more networks (e.g., 922) including wireless networks and other public networks and the like.

[0101] By way of example and not limitation, the e-commerce system 924 is web-based and provides a browser-based AR application 920A as a component of the e-commerce services provided by the e-commerce system 924. The e-commerce system 924 includes a configured computing device and a data store 8926 (e.g., a database or other configuration). The data store 926 stores data regarding products, services, and related information (e.g., techniques for applying the products). The data store 926 or other data storage device (not shown) stores recommendation rules or other product and / or service recommendations in the form of other products to assist the user in selecting from the available products and services. The products and services are presented via a user experience interface displayed on the user's (mobile) computing device. It will be understood that the e-commerce system 924 is simplified.

[0102] In one embodiment, the browser-based AR application 920A (or AR application 920) provides an augmented reality customer experience such as simulating a product, technique, or service provided or facilitated by the e-commerce system 924. In this embodiment, it will be understood that the AR application 920 is also configured to provide e-commerce services, such as via a connection to the e-commerce service 924.

[0103] By way of example and not limitation, products include beauty (e.g., makeup) products, anti-aging or rejuvenation products, and services include beauty, anti-aging or rejuvenation services. Services include treatments or other procedures. The products or services relate to a part of the human body such as the face, hair or nails. In one embodiment, a computing device (such as mobile device 912) configured as such provides a face effect unit 912A including a processing circuit. This processing circuit is configured to apply at least one face effect to the original image to generate one or more virtual instances in the original image with the effect applied on the e-commerce interface of the computing device facilitated by the e-commerce system. In one embodiment, the face effect unit 912A utilizes an adversarial generation network (GAN)-based generator (g) and an auxiliary network 110 as described herein to generate what the original image with the effect applied. In one embodiment, the computing device provides a user experience unit 912B including a processing circuit. This processing circuit determines at least one product or service from the data store 926 and generates one or more virtual instances recommended on the e-commerce interface for purchasing the product or service. In one embodiment, at least one product is associated with each face effect, and the face effect unit applies each face effect to provide a virtual trial experience.

[0104] In one embodiment, the user experience unit 912B is configured to display a graphical user interface (e.g., browser-based or otherwise) and cooperate with the computing device 912 and the e-commerce system 924. In one embodiment, the e-commerce system 924 is thus configured to provide an AR application for execution by a client computing device such as a mobile device (e.g., 912), and cooperate to provide an e-commerce service to the client computing device to facilitate recommendations for (products / services) for AR simulation via the client computing device (e.g., 8912) to promote purchases.

[0105] Accordingly, any computing device, particularly a mobile device, provides a computing device for converting an image from a first domain space to a second domain space. This computing device includes a storage unit that stores an adversarial generation network (GAN)-based generator (g), and the generator (g) is configured to generate an image controlled with respect to semantic attributes. In one embodiment, the computing device receives an original image and an input for identifying at least one semantic attribute to be controlled (e.g., including an input for refining the semantic attribute (e.g., in percentage)), and provides the image to the generator g to obtain a synthetic (e.g., new) image according to the semantic attribute input, and provides a new image for display, and includes a processing unit configured to perform the above operations (e.g., via an AR application 920).

[0106] In one embodiment, the generator is configured to synthesize a specific semantic attribute including one of face features such as age, gender, smile, etc., a pose effect, a makeup effect, a hair effect, a nail effect, a cosmetic surgery or dental effect including one of rhinoplasty, lift, blepharoplasty, implant, otoplasty, tooth whitening, orthodontics, etc., and an accessory effect including one of an eye accessory, a mouth accessory, an ear accessory, etc.

[0107] In one embodiment, the synthetic output (e.g., synthetic image or new image) of the generator g is provided to a second network model (e.g., as a component not shown in AR application 920 or 920A) for, e.g., a face editing task. The second network model can include a second generator or a feature detector and a simulator for applying an effect (e.g., for generating a further new image). This effect can be displayed on an AR interface. In one embodiment, the processing unit is configured to provide a synthetic image (or a further new image defined therefrom) in an augmented reality interface to simulate an effect applied to the image. In one embodiment, the effect includes any of a makeup effect, a hair effect, a nail effect, a cosmetic surgery or dental effect, an appliance effect, or other simulation effects applied. In one embodiment, the original image includes an applicable portion of a subject such as the user of the device (e.g., face, hair, nails, or body part). For example, the input image can be processed by the generator to simulate an age change, and the effect pipeline can process the synthetic image showing the age change to apply a makeup effect and / or a hair effect to the synthetic image.

[0108] The network model learning environment 902 provides a computing device configured to perform a method, such as a method of configuring by conditioning a GAN-based generator and an auxiliary network of a classifier sharing the same latent space. Embodiments in the context of the computing device of the network model learning environment 902, or any related embodiments in, e.g., the generator or model, will be understood to apply to aspects of the learning method with appropriate adaptation.

[0109] FIG. 10 is a flowchart of operation 1000 in one embodiment of the present specification. In one embodiment, this operation provides a method such as learning (e.g., conditional). In step 1002, the operation is configured through conditioning by a generator g 108 based on an adversarial generative network (GAN) along with the learning of the auxiliary network 110. The learning updates the latent space Z. In one embodiment, the generator g 108 is learned with labeled images having respective semantic attribute labels along with the learning of the auxiliary network 110. The co-learning uses the latent space Z to learn the auxiliary network 110, and the respective gradients of the auxiliary network 110 (e.g., those when each semantic attribute has a corresponding binary classifier) are useful for controlling the operation of the GAN-based generator g when generating an image.

[0110] In step 1004, the operation provides the generator g 108 and the auxiliary network 110 for use in a computing device to generate an image. Related aspects of the computed device and computer program product will be apparent as in other aspects.

[0111] FIG. 11 is a flowchart of operation 1100 according to one embodiment of the present specification. Operation 1100 is performed by a computing device such as devices 910, 912, 913, or 924.

[0112] The operation at 1102 provides a generator and an auxiliary network that share a latent space. The generator is configured to generate a synthetic image showing human-interpretable semantic attributes. The auxiliary network includes a plurality of semantic attribute classifiers including a semantic attribute classifier for each semantic attribute that is controlled (by the generator) when generating a synthetic image from the original image. Each semantic attribute classifier is configured to classify the presence of one of the (relevant) semantic attributes in the image and provide a significant direction for controlling one of the semantic attributes in the synthetic image of the generator.

[0113] When the generator is called to generate a composite image from the source image, the operation at 1104 applies each semantic attribute operation unit to control each semantic attribute in the composite image generated from the source image. Each semantic attribute operation unit corresponds to the significant direction provided by the classifier associated with the semantic attribute.

[0114] A practical application of the generator g includes generating and providing more usable (large amounts of) labeled composite data for use in training (or further training) one or more network models. For example, such data helps improve the results of face editing models. For example, in the case of a hair segmentation task, the segmentation model may first be trained with unbalanced data having unevenly distributed hair colors, which results in unsatisfactory segmentation results for minority hair color groups. To address this problem and create a more balanced dataset, the generator g and / or the techniques herein can be used to edit the hair color of existing labeled data and generate rich data for minority groups. Thus, the use of the generator 108 (and network 110) can include generating labeled composite images for training and for the model, and can further include training the model using the labeled composite images.

[0115] Practical implementations can include any or all of the features described herein. These and other aspects, features and various combinations can be expressed as methods, apparatus, systems, means for performing functions, program products, other methods by combining the features described herein. Some embodiments have been described. Nevertheless, it will be understood that various changes can be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided, or steps may be removed from the described process, and other components may be added to or removed from the described system. Accordingly, other aspects are within the scope of the following claims.

[0116] Throughout the description and claims of this specification, the words "comprise" and "contain" and their variations mean "include but are not limited to" and are not intended to (and do not) exclude other components, integers or steps. Throughout this specification, unless the context requires otherwise, the singular form includes the plural. In particular, where the indefinite article is used, the specification is to be understood as contemplating both the singular and the plural unless the context requires otherwise.

[0117] It should be understood that any feature, integer, characteristic, or group described in connection with a particular aspect, embodiment, or example of the invention is applicable to any other aspect, embodiment, or example, except where inconsistent therewith. All features disclosed in this specification (including any accompanying claims, abstract, and drawings), and / or all steps of any method or process so disclosed, may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of any foregoing examples or embodiments. The invention extends to any novel one or any novel combination of features disclosed in this specification (including any accompanying claims, abstract, and drawings), or any novel one or any novel combination of steps of any method or process disclosed. [References] [ACB17] Martin Arjovsky, Soumith Chintala, and Leon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017. [BDS18] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018. [CCK +18] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recog-nition, pages 8789-8797, 2018. [CVB21] Anton Cherepkov, Andrey Voynov, and Artem Babenko. Navigating the gan parameter space for semantic image editing. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 3671-3680, 2021. [GPAM + 14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014. [HHLP20] Erik Harkonen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Dis-covering interpretable gan controls. arXiv preprint arXiv:2004.02546, 2020. [HRU +17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equi-librium. Advances in neural information processing systems, 30, 2017. [KLA18] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.04948, 2018. [KLA + 20] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 8110-8119, 2020. [LLWT15] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015. [PBH20] Antoine Plumerault, Herve Le Borgne, and Celine Hudelot. Controlling generative models with continuous factors of variations. arXiv preprint arXiv:2001.10238, 2020. [SCD + 17] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017. [SYTZ20] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, 2020. [SZ21] Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 1532-1540, 2021. [VB20] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In International Conference on Machine Learning, pages 9786- 9796. PMLR, 2020. [WDB + 19] Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisation for generative adversarial networks. arXiv preprint arXiv:1912.00953, 2019. [WLS21] Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled con-trols for stylegan image generation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 12863-12872, 2021. [ZKL + 16] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.

Claims

1. A method comprising generating a synthetic image (g(z')) from an original image using a generator (g) having a latent code (z), wherein the generator manipulates a target semantic attribute (k) within the synthetic image, said generating comprising: finding a significant direction identified from an auxiliary network that classifies each semantic attribute including the target semantic attribute k at z, said significant direction at z with respect to the target semantic attribute k; defining z' by optimizing z according to the significant direction with respect to the target semantic attribute k; outputting the synthetic image.

2. The method according to claim 1, wherein the auxiliary network includes a set of binary classifiers, one for each semantic attribute manipulable by the generator, and the auxiliary network is co-trained to share the generator g and the latent code space Z.

3. The method according to claim 1 or 2, wherein each of the semantic significant data directions includes a respective dimensional data vector obtained from each individual classifier, each vector including the direction and rate of fastest increase in the individual classifier.

4. The method according to any one of claims 1 to 3, comprising unraveling the data directions by filtering out significant dimensions of the respective semantic significant data directions for each other semantic attribute (m, m≠k) associated with the target semantic attribute from the dimensions of the semantic significant data direction of the target semantic attribute.

5. The method according to claim 4, wherein said filtering comprises evaluating each dimension of the respective semantic significant data direction unraveled from the target semantic significant data direction and setting the value of the corresponding dimension in the target semantic significant data direction to a zero value if a particular dimension exceeds a threshold.

6. The potential code z is optimized as z' = z + αn z k' and Here, n z k is a vector representing the target semantic significant data direction before filtering, n z k' is a vector representing the target semantic significant data direction after filtering, The method according to claim 5, wherein α is a hyperparameter that controls the interpolation direction and step size.

7. The method according to any one of claims 1 to 6, comprising repeating the operations of finding, defining, and optimizing with respect to the latent code z' to further manipulate the semantic attribute k.

8. generating a synthetic image for manipulating a plurality of target semantic attributes, Discovering, for each of the target semantic attributes, a respective significant direction in z as identified from the auxiliary network that classifies each of the target semantic attributes in z; Optimizing z according to each of the significant directions, the method according to any one of claims 1 to 7. **Claim 9** The method according to any one of claims 1 to 8, including learning another network model using the synthetic image. **Claim 10** The method according to any one of claims 1 to 9, including receiving an input for identifying at least one of the respective semantic attributes controlled for the original image. **Claim 11** Providing the generator and the auxiliary network to generate the synthetic image as a service; Providing an e-commerce interface for purchasing products or services; Providing a recommendation interface for recommending products or services; Providing an augmented reality interface using the synthetic image to provide an augmented reality experience; The method according to claim 1, including any one or more of the above. **Claim 12** The target semantic attribute includes one of face features such as age, gender, smile, etc., a pose effect, a makeup effect, a hair effect, a nail effect, a cosmetic surgery or dental effect including one of rhinoplasty, lift, blepharoplasty, implant, otoplasty, teeth whitening, teeth correction, etc., and an appliance effect including one of an eye appliance, a mouth appliance, an ear appliance, etc., the method according to any one of claims 1 to 11. **Claim 13** A method including providing an augmented reality (AR) interface to provide an AR experience, wherein the AR interface is configured to generate the synthetic image from a received image using a generator by applying respective semantic attribute operation units for controlling respective semantic attributes in the synthetic image, and each of the respective semantic attribute operation units corresponds to a significant direction provided by a classifier associated with the semantic attribute; the method including receiving the received image and providing the synthetic image for the AR experience. **Claim 14** processing the composite image using an effect pipeline to simulate an effect; and providing the composite image having the simulated effect for presentation on the AR interface, the method of claim 13 **Claim 15** A computing device comprising at least one processor and at least one non-transitory storage device storing computer-readable instructions for execution by the at least one processor, the instructions causing the computing device to perform the method of any one of claims 1 to 14

Images