Method and apparatus for determining and using controllable direction of GAN space

Abstract

Methods, devices, and techniques herein relate to determining directions in a GAN potential space and obtaining unwrapping control of GAN output semantics, e.g., to be able to generate composite images using these methods, devices, and techniques, such as for training another model or creating augmented reality. In accordance with an embodiment, the methods, devices, and techniques herein utilize a gradient direction of an auxiliary network to control semantics in GAN potential code. It has been shown that a minimum amount of labeled data with a sample size of as small as 60 can be used, and these data can be quickly obtained under manual supervision. It is also shown herein, according to embodiments, to select important potential code channels using masks during manipulation, thereby achieving more unwrapping control.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 392,905 filed on July 28, 2022, the entire contents of which are incorporated herein by reference. This application claims priority to French Patent Application No. FR 2210849 filed on October 20, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to image processing using artificial intelligence and deep neural networks, including generative adversarial networks (GANs), such as for creating augmented reality. More specifically, the present disclosure relates to methods and devices for determining and using controllable directions of GAN space. Background Art

[0004] GANs have been successfully applied to solve a variety of tasks, such as simulating makeup effects on a person's face or changing hair color to provide realistic augmented reality (AR) technology in the beauty industry. However, conditional generative models that control the semantics of the output usually require a large amount of labeled training data, which can be expensive and time-consuming to obtain. In addition, current facial editing outputs sometimes have artifacts and attribute entanglement, which can reduce the overall user experience.

[0005] The goal is to gain control over the semantics of native GAN outputs, improve the user experience by removing artifacts or disentangled attributes when editing faces, and generate large amounts of synthetic labeled data for training conditional models. Summary of the invention

[0006] The methods, devices and techniques herein relate to determining directions in a GAN latent space and obtaining disentangled control over the semantics of the GAN output, for example, enabling the use of these methods, devices and techniques to generate synthetic images, such as for training another model or creating augmented reality. According to an embodiment, the methods, devices and techniques herein use the gradient direction of an auxiliary network to control the semantics in the GAN latent code. It has been shown that a minimum amount of labeled data with a sample size as small as 60 can be used, which can be quickly obtained under human supervision. It is also shown herein that, according to an embodiment, a mask is used to select important latent code channels during manipulation, thereby achieving more disentangled control. Various aspects are shown and described herein, as set forth in the following statements according to embodiments. These and other aspects will be apparent to those of ordinary skill in the art after considering the present application as a whole.

[0007] Statement 1: A computing device is provided, comprising: a processor and a storage device, the storage device storing instructions that, when executed by the processor, cause the computing device to: provide a generator and an auxiliary network that share a latent space, the generator being configured to generate a synthetic image exhibiting human-interpretable semantic attributes, and the auxiliary network comprising a plurality of semantic attribute classifiers, including a semantic attribute classifier for each semantic attribute to be controlled for generating a synthetic image from a source image, each semantic attribute classifier being configured to classify the presence of one of the semantic attributes in the image and provide a meaningful direction for controlling one of the semantic attributes in the synthetic image of the generator; and generate a synthetic image from the source image using the generator by controlling the corresponding semantic attribute in the synthetic image by applying a corresponding semantic attribute control, the corresponding semantic attribute control being responsive to the meaningful direction provided by the classifier associated with the semantic attribute.

[0008] Statement 2: The computing device of Statement 1, wherein the meaningful direction comprises a gradient direction, and wherein the instructions cause the computing device to compute a corresponding semantic attribute control based on a corresponding gradient direction of a classifier associated with the semantic attribute.

[0009] Statement 3: The computing device of Statement 2, wherein, to control two or more semantic attributes, the instructions cause the computing device to combine the respective gradient directions of each classifier associated with the two or more semantic attributes to be controlled.

[0010] Statement 4: The computing device of Statement 1 or 2, wherein the instructions cause the computing device to unwrap the corresponding semantic attribute controls for application to the generator.

[0011] Statement 5: A computing device according to Statement 4, wherein: each corresponding semantic attribute control includes a corresponding gradient direction vector calculated based on parameters of a corresponding semantic attribute classifier associated with the corresponding semantic attribute control; and disentangling one corresponding semantic attribute control from another corresponding semantic attribute control includes: removing important data dimensions of the gradient direction vector associated with one corresponding semantic attribute control from the gradient direction vector associated with another corresponding semantic attribute control.

[0012] Statement 6. The computing device of Statement 5, wherein the significant data dimensions are removed in response to a threshold value identifying those data dimensions having an absolute value greater than or equal to the threshold value.

[0013] Statement 7: A computing device according to Statement 6, wherein, if the i-th data dimension of the gradient direction vector associated with one corresponding semantic attribute control is identified by a threshold, the corresponding i-th data dimension of the gradient direction vector associated with another corresponding semantic attribute control is set to zero.

[0014] Statement 8. The computing device of any of Statements 1 to 7, wherein the instructions cause the computing device to receive input identifying at least one of the respective semantic attributes to be controlled relative to the source image.

[0015] Statement 9: The computing device of Statement 8, wherein the input identifying at least one of the respective semantic attributes comprises a granularity input to determine an amount of the semantic attribute to apply when generating the composite image.

[0016] Statement 10: A computing device according to any one of statements 1 to 9, wherein the instructions cause the computing device to perform any one or more of the following: providing a generator and an auxiliary network to generate composite images as a service; providing an e-commerce interface to purchase products or services; providing a recommendation interface to recommend products or services; and providing an augmented reality interface using the composite images to provide an augmented reality experience.

[0017] Statement 11: A computing device according to any of statements 1 to 10, wherein a particular semantic attribute among the plurality of semantic attributes comprises one of: facial features, including age, gender, smile, or other facial features; posture effects; makeup effects; hair effects; nail effects; cosmetic or dental effects, including one of rhinoplasty, lift, blepharoplasty, implants, otoplasty, teeth whitening, orthodontics, or other cosmetic or dental effects; or an appliance effect, including one of an eye appliance, an oral appliance, an ear appliance, or other appliance effects.

[0018] Statement 12: A computing device according to any one of statements 1 to 11, wherein the generator is a GAN-based generator and each classifier is a neural network-based binary classifier, and the generator and each neural network-based binary classifier are co-trained using training images that exhibit semantic attributes of the labels to define meaningful directions in the corresponding classifiers.

[0019] Statement 13: A method is provided, comprising: training a controllable generative adversarial network (GAN)-based generator g to generate a synthetic image from a source image, wherein at least one semantic attribute in a defined set of semantic attributes is selectively controlled, wherein the generator comprises a model that maps a latent code (z) in its latent space (Z) to an image (x=g(z)) in an image space (X) where human-interpretable semantics exist, the training comprising: collaboratively training the generator g and an auxiliary network, the auxiliary network comprising a corresponding classifier for each semantic attribute in the defined group, each corresponding classifier providing meaningful data direction for controlling the corresponding semantic attribute when generating an updated image; and providing the generator g and the auxiliary network for generating the synthetic image.

[0020] Statement 14: The method of Statement 13, comprising: calling a generator g using at least one control calculated from parameters of an auxiliary network, the auxiliary network providing meaningful data directions for each of the at least one control to generate a plurality of synthetic images having semantic properties selected from a defined group.

[0021] Statement 15: The method of Statement 13 or 14, comprising: using at least some of the plurality of synthetic images to train a further network model.

[0022] Statement 16: A method comprising: generating a synthetic image (g(z′)) from a source image using a generator (g) having a latent code (z), the generator manipulating a target semantic attribute (k) in the synthetic image, comprising: discovering a meaningful direction at z for the target semantic attribute k, the meaningful direction being identified by classifying corresponding semantic attributes at z through an auxiliary network, including the target semantic attribute k; defining z′ by optimizing z in response to the meaningful direction of the target semantic attribute k; and outputting the synthetic image.

[0023] Statement 17: The method of Statement 16, wherein the auxiliary network comprises a set of binary classifiers, one for each semantic attribute that the generator can manipulate, and the auxiliary network is co-trained with the generator g to share the latent code space Z.

[0024] Statement 18: The method of Statement 16 or 17, wherein the respective semantically meaningful data directions comprise respective dimensional data vectors obtained from each individual classifier, each vector comprising the fastest increasing direction and rate in the individual classifier.

[0025] Statement 19: A method according to any one of Statements 16 to 18, comprising: disentangling data directions by filtering out from the dimensions of the semantically meaningful data directions of the target semantic attribute those important dimensions in the corresponding semantically meaningful data directions of each other semantic attribute (m, m≠k) that is entangled with the target semantic attribute, wherein the importance of a particular dimension is based on its absolute gradient magnitude.

[0026] Statement 20: The method of Statement 19, wherein filtering comprises: evaluating each dimension of each semantically meaningful data direction to be disentangled from the target semantically meaningful data direction; if a particular dimension exceeds a threshold, setting the value of the corresponding dimension in the target semantically meaningful data direction to a zero value.

[0027] Statement 21: The method according to Statement 20, wherein the latent code z is optimized as: in: is a vector representing the direction of the target semantically meaningful data before filtering; is a vector representing the direction of the data that is semantically meaningful after filtering; and α is a hyperparameter that controls the interpolation direction and step size.

[0028] Statement 22: The method of any one of Statements 16 to 21, comprising: repeating the operations of discovering, defining and optimizing the latent code z' to further manipulate the semantic attribute k.

[0029] Statement 23: A method according to any one of Statements 16 to 22, comprising: generating a synthetic image to manipulate multiple target semantic attributes; finding a corresponding meaningful direction at z for each of the target semantic attributes, the meaningful directions being identified by classifying each of the target semantic attributes at z through an auxiliary network; and optimizing z in response to each of the meaningful directions.

[0030] Statement 24: A method according to any one of Statements 16 to 23, comprising: training another network model using the synthetic images.

[0031] Statement 25: The method of any of Statements 16 to 24, comprising: receiving an input identifying a target semantic attribute to be controlled relative to a source image.

[0032] Statement 26: The method of Statement 25, wherein the input comprises a granularity input to determine an amount of a target semantic attribute to apply when generating the composite image.

[0033] Statement 27: A method according to any of Statements 16 to 26, comprising any one or more of the following: providing a generator and an auxiliary network to generate composite images as a service; providing an e-commerce interface to purchase products or services; providing a recommendation interface to recommend products or services; or providing an augmented reality interface using the composite images to provide an augmented reality experience.

[0034] Statement 28: A method according to any of Statements 16 to 27, wherein the target semantic attribute includes one of: facial features, including age, gender, smile, or other facial features; posture effects; makeup effects; hair effects; nail effects; cosmetic surgery or dental effects, including one of rhinoplasty, lift, blepharoplasty, implants, otoplasty, teeth whitening, orthodontics, or other cosmetic surgery or dental effects; or appliance effects, including one of eye appliances, mouth appliances, ear appliances, or other appliance effects.

[0035] Statement 29: A method comprising the steps of: providing an augmented reality (AR) interface to provide an AR experience, the AR interface being configured to control corresponding semantic attributes in a composite image by applying corresponding semantic attribute controls, generating a composite image from a received image using a generator, the corresponding semantic attribute controls being responsive to meaningful directions provided by a classifier associated with the semantic attributes; and receiving the received image and providing the composite image for the AR experience.

[0036] Statement 30: The method according to statement 29, comprising: processing the composite image using an effects pipeline to simulate an effect, and providing the composite image with the simulated effect for presentation in an AR interface. For example, the method according to statement 29 or 30 can be combined with any of the subject matters of statements 1 to 13 or 16 to 28 (adjusted as appropriate).

[0037] Statement 31: A computing device comprising at least one processor and at least one non-transitory storage device storing computer readable instructions for execution by the at least one processor, the instructions causing the computing device to perform a method according to any of Statements 13 to 30.

[0038] Statement 32: A computer program product comprising at least one non-transitory storage device storing computer readable instructions for execution by at least one processor of a computing device, wherein execution of the instructions causes the computing device to perform a method according to any of Statements 13 to 30. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1A , Figure 1B and Figure 1C An overview of the training and / or architecture-related aspects disclosed herein according to corresponding embodiments is shown. Figure 1A is a graphical representation of a computing device providing a convolutional neural network (CNN) according to an embodiment. Figure 1B is a graphical representation of the dimensions of the corresponding gradients of semantic attributes k and m according to an embodiment. Figure 1C is a flow chart of a step optimization operation according to an implementation scheme.

[0040] Figure 2 is a table of images illustrating the disentanglement of semantic attributes in a facial image (ie, disentanglement of a smile from glasses), according to an embodiment.

[0041] Figure 3 is a pseudocode listing operation according to an implementation.

[0042] Figure 4is a table of images showing results of semantic attribute manipulation using a controllable neural network, according to an embodiment.

[0043] Figure 5 is a table of images showing semantic attribute disentanglement results using a controllable neural network, according to an embodiment.

[0044] Figure 6 is a table of images visualizing the orientations of three controllable neural networks (ie, two previously known controllable neural networks and a controllable neural network according to an embodiment of the present invention).

[0045] Figure 7 is a graphical table showing a comparison of attribute disentanglement results obtained by a previously known controllable neural network and a controllable neural network according to embodiments of the present invention.

[0046] Figure 8 is a block diagram of a graphical user interface (GUI) for generating a synthetic image from a source image using a controllable GAN, according to an embodiment.

[0047] Fig. 9 is a block diagram of a computer system according to an embodiment.

[0048] Fig.10 and Fig.11 is a flow chart illustrating operations according to corresponding embodiments herein.

[0049] The present concept is best described by certain embodiments thereof, which are described herein with reference to the accompanying drawings, wherein like reference numerals refer to like features throughout the specification. It should be understood that the term "invention" as used herein is intended to refer to the inventive concept underlying the embodiments described below, and not just the embodiments themselves. It should also be understood that the overall inventive concept is not limited to the illustrative embodiments described below, and the following description should be understood on this basis. DETAILED DESCRIPTION

[0050] Although GAN models are able to generate highly realistic images based on random inputs from their latent space, the generation process is usually a black box, thus lacking direct control over the semantics of the output. However, previous studies [VB20, VB20, SZ21, HHLP20, SYTZ20] have shown that there are meaningful directions and channels in the GAN latent space, and that linear interpolation in these directions or changing individual channel values ​​produces interpretable transformations, such as adding glasses or a smile to a face. The goal of the methods, devices, and techniques of this paper is to determine (e.g., learn) directions in the GAN latent space and obtain disentangled control over the semantics of the GAN output, for example, so that these learned directions can be used to create augmented reality, generate synthetic images for training models, etc. The focus of this paper is on the task of face editing, which is useful in the beauty industry as well as other industries. To this end, according to an embodiment, the methods, devices, and techniques of this paper use the gradient directions of an auxiliary network to control the semantics in the GAN latent code. It has been shown that a minimal amount of labeled data with a sample size as small as 60 can be used, which can be quickly obtained under human supervision. It is also shown herein that, according to embodiments, Grad-CAM based masks are used to select important potential code channels during manipulation, thereby achieving more disentangled control. The following section reviews prior work in related areas to contextualize the teachings of this article.

[0051] By [GPAM + 14], the Generative Adversarial Network (GAN), currently dominates the field of generative modeling due to its powerful ability to synthesize photo-realistic images. Typically, a GAN consists of two networks: a generator that learns a mapping from its latent space to the image space; and a discriminator that distinguishes images synthesized by the GAN from real images, and the two networks are jointly trained in an adversarial manner. Different GAN variants have been proposed to improve output fidelity and training stability [ACB17, BDS18, CCK + 18. KLA18. KLA + 20], and there is increasing research interest in studying the latent space of these native GAN models, which provides more control over the generation process.

[0052] Latent semantic manipulation of GANs has been studied by multiple previous studies [VB20, VB20, SZ21, HHLP20, SYTZ20, PBH20], and it has been shown that there are meaningful directions and channels in the latent space / feature space of GANs. Some studies have found that the output semantics of unconditional GANs can be controlled through explicit supervision [SYTZ20, WLS21]. For example, InterFaceGAN [SYTZ20] assumes that for any binary semantics (e.g., male vs. female), there exists a hyperplane in the latent space as a separating boundary, and its normal vector represents a meaningful direction. A classifier pre-trained on the CelebA dataset [LLWT15] is used to generate pseudo labels for GAN synthesized images, and then a state vector machine (SVM) trained on paired data with GAN latent codes and corresponding semantic labels is used to learn the separating boundaries. [WLS21] discovered channels that control local effects in the activation space of GANs under the guidance of a pre-trained semantic segmentation network. However, the range of controls found by these supervised methods is usually limited because they require large amounts of labeled data or pre-trained neural networks, and the learned controls are sometimes entangled.

[0053] Other studies have discovered such controls in a self-supervised or unsupervised manner [HHLP20, CVB21, VB20]. For example, GANSpace [HHLP20] identifies important latent directions by applying principal component analysis (PCA) to vectors in the GAN latent space or feature space, while [VB20] discovers interpretable directions in the GAN latent space by jointly optimizing directions and a reconstructor that recovers these directions and manipulation strengths from images generated from the manipulated GAN latent code. While these methods do not require labeled data, they typically involve extensive manual inspection of different manipulation directions and identification of meaningful controls. More importantly and unlike the techniques taught in accordance with the embodiments of this article, control of the target semantics cannot always be guaranteed.

[0054] Gradient-based knowledge has been used to improve the interpretability of neural networks [ZKL + 16. SCD + 17] or training stability [WDB + 1 9]. For example, [SCD + 1 7]] utilizes the gradient information from the classification output to obtain a localization map that provides visual evidence in the image. [WDB + 19] proposed to improve the stability of the GAN training dynamics by adding an extra step before jointly optimizing the generator and the discriminator, where the latent code is first optimized to a region that is considered more realistic by the discriminator, and then the direction is obtained by computing the gradient with respect to the latent code.

[0055] With [WDB + 19] In contrast, and in accordance with embodiments herein, previously unseen in the literature, auxiliary binary classifier gradients are used to gain control over the semantics of the GAN output. Thus, in accordance with embodiments, methods, devices, and techniques are described herein to gain control over disentanglement in the GAN latent space by first finding semantically meaningful directions by computing the gradients of a binary classifier that scores different semantics based on a given latent code input, and then selecting important dimensions in the latent code for attribute disentanglement during manipulation.

[0056] Determine semantically meaningful directions

[0057] Figure 1A is a graphical representation of a computing device 100 according to an embodiment, the computing device including a CNN architecture 102 such as stored in a storage device 106 of the device 100 and an image output (e.g., image 104 including 104A and 104B). Figure 1B is a graphical representation of a portion of a storage device 106 storing dimensions of respective gradients 132 and 130 for semantic attributes k and m according to an embodiment. The figure shows the disentanglement of the gradient dimensions of attribute m from the corresponding gradient dimensions of attribute k. Figure 1C is a flow chart of an optimization operation 150 according to an embodiment and as further described below.

[0058] CNN architecture 102 includes a controllable GAN g 108 configured to generate output images g(z) 104A and g(z′) 104B from respective latent codes z and z′ (e.g., respective portions of the latent space Z of GAN g 108). Images g(z) 104A and g(z′) 104B show facial images exhibiting different semantic attributes. In the example shown, the face of the individual in image g(z) 104A is not smiling, while the face of the same individual in image g(z′) 104B is smiling.

[0059] The CNN architecture 102 also includes a plurality of corresponding auxiliary networks 110, one for each semantic attribute controlled by the GAN g 108. The plurality of auxiliary networks 110 each include a corresponding mapping function f, and the auxiliary networks 110 are trained together with the GAN g 108 to discover a meaningful direction for each semantic attribute to be controlled. In an embodiment, the mapping function f includes a binary classifier. For simplicity, Figure 1A Only details related to one of the corresponding classifiers from the auxiliary network 110 are shown in box 112, namely the mapping function f for the semantic attribute k. k110A. Figure 1A In , each of the GAN g 108 and the auxiliary network 110 is previously trained using training images for each of the semantic attributes (e.g., 60 images for each semantic attribute). As shown in block 112, to manipulate the semantic attribute k (e.g., smile) of the GAN latent code z, a mapping function f from mapping the GAN latent code (e.g., z) to the semantic score of k is obtained. k The gradient direction 114 is used as input to derive the modified z (i.e., z′) of GAN g 108. The equations for deriving the modified z are further described below.

[0060] Next reference Figure 1B According to an embodiment, the semantic attributes are disentangled. For attribute k, the important dimensions used to manipulate the semantic attribute m are removed. According to an embodiment, the important dimensions are the semantic attributes m in 130 has a larger gradient magnitude. For the semantic attribute k, such dimensions have a larger gradient magnitude in the gradient direction. 112 is masked (eg, set or defined) to a value of 0, wherein the gradient direction 132 represents the gradient direction after masking (i.e., after setting important attributes to zero) 112. Dimensionality masking is described further below, particularly with respect to what are large gradient magnitudes.

[0061] Next reference Figure 1C , where operation 150 shows a pipeline for a single-step optimization. Operations begin at 152 using a latent code z, such as a latent code for a face image. At 154, a meaningful direction of a semantic attribute k at z is found (e.g., according to a trained corresponding mapping function f k The decision at 156 evaluates whether there is entanglement with any semantic m (where m represents a semantic attribute different from k that is controlled by GAN g). If so, the “yes” branch to 158 finds a meaningful direction for the semantic attribute m at z (e.g., according to the corresponding mapping function f after training). m At 160, an operation performs unwrapping, such as referring to Figure 1BAs described and further described below. In an embodiment, entanglement is experimentally identified, such as at 156. Entanglement is observed by editing each attribute in the original gradient direction of k using a small meta-validation set of 30 images, and then deciding which attributes are entangled. The association between the target attribute and any entangled attributes is tabulated. The important attributes from each of the identified attributes (i.e., those attributes where m≠k) are masked out. In an embodiment, the union of the first few channels (e.g., the 100 largest gradient magnitudes) of the attribute to be disentangled is determined. These dimensions are then zero in the target direction (e.g., producing ).

[0062] In embodiments, for example, without prior observation experiments, the gradient can be responded to The size of the i-th dimension value in directly masks the gradient In the first case, when there is entanglement, masking out these dimensions helps disentanglement; in the second case, there is no entanglement and the important dimensions of m are negligible in k, so masking out these dimensions does not change the direction significantly.

[0063] At 162, the operation optimizes the potential code z. The operation can be repeated for the next instance of the potential code z (e.g., looping to step 154) until the desired manipulation result is achieved. Referring again to step 156, in an embodiment, if there is no corresponding entanglement, the operation proceeds to step 162 through the "no" branch to perform optimization of the potential code z. As previously described, in an embodiment, even if there is no entanglement, the operation can mask out important dimensions of any other attributes. For example, it may be simpler (e.g., programmatically) to determine which m attributes are entangled and then disentangle the important dimensions from the gradient of k, but only disentangle the important dimensions from all m≠k.

[0064] Figure 22 is an image table 200 including table rows 202 and 204 for illustrating the disentanglement of smiles from glasses. The first (left) column 206 in both rows 202 and 204 is the original image. The second (middle) column 208 and the third (right) column 210 are edited using smaller and larger interpolation distances. The difference is that in the second row, when the distance is larger (i.e., more smiles are added relative to column 208), no glasses are added. That is, the second row applies the disentanglement. Therefore, row 202 shows the corresponding facial images of the same subject in each image. From left to right, the subject has: i) no smile and no glasses, ii) smile and no glasses, and iii) smile and glasses. In row 204, the image shows the same subject as shown in row 202. The image of row 204 shows the application of the disentanglement not applied in row 202. From left to right, the subject has: i) no smile and no glasses, ii) smile and no glasses, and iii) smile and no glasses.

[0065] During a one-step optimization operation, as shown in Figure 1B and Figure 1C As discussed, 眼镜 The important dimension in the gradient of is masked to 0. denotes the absolute value of the gradient vector at dimension i. In contrast to the original multi-class classification setting of Grad-CAM (where ReLU is chosen as the activation), the absolute value is used here, as in the binary classification setting. Dimensions that have a negative impact on the semantic score also contain meaningful information.

[0066] In more detail, according to an embodiment, a method for determining (e.g., learning) meaningful directions in a GAN latent space to manipulate semantics in a GAN output is provided. A well-trained GAN model learns a mapping g that transforms a d-dimensional latent code z (located in the GAN latent space Z (e.g., z∈Z, ) is mapped to an image x=g(z) in image space X. Human-interpretable semantics exist in image space X and include, for example, age, gender, glasses, smile, head pose of a person, or other semantic observations. Given a series of scoring functions s for K semantics 1 ...s k , respectively, from the latent space Z to the semantic space K mappings, where:

[0067]

[0068] And among them, represents the kth semantic score in the image generated based on the latent code z. In an embodiment, each of the semantics is separate. For example, instead of treating the attribute "smile and teeth" as one attribute, "smile" and "teeth" are treated as separate attributes. By combining the control of smile and teeth (i.e., adding the direction vectors), multiple semantics (smile and teeth) can be controlled simultaneously.

[0069] Assumptions 1 ...s k With accuracy and independence, it is expected that when the kth semantic attribute in the output image changes, its score changes accordingly, while the other semantic scores remain roughly unchanged. Assuming that there is useful information in the mapping from the GAN latent space to the semantic space, this information can be used to find semantically meaningful directions. Specifically, in order to control It is proposed to interpolate the potential code according to the gradient direction of this mapping function. To simplify the calculation, each mapping function s k (g(z)) are all parameterized, where the neural network f k The GAN is trained on paired samples of latent codes and corresponding semantic labels, which are composed of the original s k (g(z)) (e.g., ground truth scoring function / human perception) is generated. The direction is calculated as follows:

[0070]

[0071] Among them, the vector Indicates that at z The direction and rate of fastest growth of . The potential optimization technique used to optimize the latent code during training (depending on the embodiment) follows the assumption that the direction pointed by the gradient from the GAN discriminator D indicates a better sample. The GAN discriminator can be viewed as a scoring function for "realness". This idea is extended to multiple (e.g., K) semantics. Intuitively, for each target semantic, interpolating the latent code according to the gradient from the corresponding semantic attribute scoring function changes the semantic score and thus manipulates the semantic attribute. To manipulate the semantic attribute k once, z is updated using:

[0072]

[0073] where α is a hyperparameter that controls the interpolation direction and step size. Note that, in contrast to previous studies [SYTZ20, HHLP20] that found linear semantically meaningful directions applicable to all data points in the GAN latent space, the present method and / or technique finds a unique direction for each data point: Due to the nonlinear nature of Equation 1, as z changes, will also change at the same time, and at each step of the optimization a new direction is calculated for the updated z.

[0074] Attribute unwrapping during manipulation

[0075] According to an embodiment, a method and / or technique for minimizing semantic attribute entanglement is provided. Attribute entanglement may (e.g., sometimes) occur during latent code interpolation in the above-mentioned direction. As used herein and as can be understood by a person of ordinary skill in the art, two or more semantic attributes become entangled when manipulation of one semantic attribute during interpolation affects another or other attributes of the two or more attributes. It has been observed that interpolation along the original direction may change non-target semantics, but this effect can sometimes be eliminated by randomly excluding dimensions in the direction vector used for interpolation. Therefore, it is assumed that within the d-dimensional direction vector found, only some dimensions are responsible for changes in the target semantic attributes, while other dimensions represent deviations learned from the training data. For example, in the direction of increasing the age of a person, because glasses are data-related to age, glasses appear during manipulation.

[0076] Grad-CAM[SCD + 17] is a class-discriminative localization technique that uses gradient information to provide visual explanations for CNN-based models. According to the methods and / or techniques herein, dimensions are filtered out based on the gradient magnitude from a semantic scoring function. This results in a more disentangled control. Specifically, Grad-CAM measures the importance of neurons by:

[0077]

[0078] Among them, the score y of class c c The gradients with respect to the activation map A in all width and height dimensions are globally average pooled. The final heat map is calculated as follows:

[0079]

[0080] According to an embodiment, is considered as the only activation map, and the importance of the i-th dimension is calculated as follows:

[0081]

[0082] By the definition of gradient, The value of indicates the difference caused by a small change in z in the same dimension. Intuitively, Has a larger The dimension of has a larger impact, while dimensions with smaller values ​​are less relevant. However, when calculating the effect of When m≠k is controlled, these irrelevant dimensions may have higher gradient magnitudes. Therefore, when optimizing for the kth semantics, even small changes in these dimensions may affect This leads to the problem of attribute entanglement. Therefore, according to the implementation method, for For any k-th semantic attribute that is not less than a certain threshold, the dimensions in its gradient are considered important dimensions, while any important dimensions that appear to be coupled with the target from the target direction when predicting the semantic attribute are excluded. In an embodiment, if k is entangled with m and they share important dimensions, then even the important dimensions of k can be masked out.

[0083] Figure 3 The operation 300 of calculating a new direction in the case of attribute disentanglement according to an embodiment is described in pseudo code form. At line 1, it should be noted that the target semantics is in the direction In line 2, it should be noted that for the entangled semantics m and the threshold t, a scoring function f is established. m , to identify important gradients (i.e., large gradient magnitudes). At line 3, the gradient of the entangled semantic m Determined by its corresponding function. At line 4, the i-dimensional element is checked relative to t Each of which, the element At line 5, or the gradient identified by the threshold, the corresponding gradient magnitude in the dimension for k is set to 0: for i∈E, and returns at line 6 (for example provided) final form.

[0084] At each step, in order to convert the semantic score Increase once and untangle the mth semantics, and update z using the following formula:

[0085]

[0086] in, according to Figure 3 Operation 300 of the pseudocode algorithm is calculated. It should be understood that the operation can be repeated cyclically so that z′ is calculated more than once to further interpolate in the direction of the target semantic attribute, for example, until an image having the desired characteristics is produced (e.g., g(z′) 104B).

[0087] Implementation details

[0088] According to an embodiment, the GAN part of the network architecture structure 102 (e.g., GAN g 108) adopts StyleGAN2 [KLA + 20], for example, pre-trained on the FFHQ dataset [KLA18]. + 20] is optimized in the W space. 400 images are generated using StyleGAN2, 30 images are manually selected as positive / negative samples for each target semantic, and 10% of the pairs are selected for evaluation. Multiple binary classifiers (e.g., corresponding instances of 110) are trained simultaneously for different target semantics in a multi-label learning manner to minimize the sum of all binary cross entropy losses. According to an embodiment, each classifier (e.g., f k ) includes two fully connected layers, where the size of the hidden layer is equal to 16, and ReLU activation and Sigmoid activation are located at the output neurons of the hidden layer. For the filtering threshold t, such as determined by experiment, from The 100th maximum gradient magnitude of each channel works well in practice. For the value of α, 0.4 is used in all experiments.

[0089] In an embodiment, in order to obtain the gradient vector Remove the dimension that is important to the semantic attribute m, so m is entangled with k, and determine the gradient vector of m Each dimension of this vector is evaluated (e.g., some form of sorting may be performed) to determine the 100th largest value using the absolute value of that dimension. This 100th largest value becomes the threshold t relative to m when filtering. For each i-th dimension in m's vector that is equal to or greater than the 100th largest value, in k's vector, the corresponding i-th dimension is set to 0. For each semantic attribute m that is entangled with k, the gradient vector of m is similarly determined and evaluated with respect to t, and the corresponding dimension is then masked in k's gradient vector.

[0090] Results and discussion

[0091] Evaluation results are presented for the disclosed methods, systems, and techniques (e.g., on StyleGAN2 [KLA + 20]). Figure 4 An image table 400 presenting manipulation results for 5 attributes (age, pose, gender, glasses and smile) is shown. These qualitative examples were compiled using learning directions trained on 60 samples per attribute. Figure 5 An image table 500 showing attribute disentanglement is shown. Figure 6 and Figure 7Corresponding graphical tables 600 and 700 are shown, which present comparative results with other studies [SYTZ20, WLS21, HHLP20].

[0092] Table 400 shows the manipulation results for 5 different attributes, i.e., single attribute manipulation results. For each set of three samples in a row (e.g., 402), the image in the middle is the original StyleGAN2 synthesized image, and the left / right images correspond to the synthesized results (i.e., as output from the network architecture 102 (via GAN g 108)) based on the original latent code edited in the negative / positive direction found according to the method or technique disclosed herein.

[0093] The proposed method performs well in both negative and positive directions for all attributes. Specifically, for the age attribute, it is noteworthy that the proposed method is not only able to produce / eliminate subtle aging effects (such as wrinkles or acne), but also able to change the face shape to a high level while preserving the identity of the person very well. The method is also able to add glasses while keeping the irrelevant semantics roughly unchanged, which means that many realistic image samples with glasses can be created despite the absence (i.e., lack) of such semantic attributes in the original FFHQ training data.

[0094] Table 500 shows Figure 5 Attribute disentanglement results in . For each set of three images at each row (e.g., 502, 504, 506, and 508), the first image on the left (e.g., 502A, 502D, 504A, and 504D) is the original StyleGAN2 synthesized image. The adjacent left and right images (502B / 502C, 502E / 502F, 504B / 504C, and 504E / 504F) correspond to the synthesized results (via GAN g 108) of the original latent code based on the edits according to the negative / positive directions (e.g., using relatively small and large interpolation distances) found by the methods or techniques of this article. The first two rows 502 and 504 illustrate that the aging direction is entangled with the glasses direction. By excluding the important dimension of glasses classification from the aging direction during optimization, the two attributes (e.g., row 504) are successfully disentangled while achieving similar aging results. Similarly, for the last two rows 506 and 508, the method of excluding important dimensions as disclosed herein successfully finds a new direction where the gender direction is less entangled with the smile.

[0095] It is important to note that although the controls found by the methods or techniques disclosed herein are generally independent, there are still cases of failure where manipulating one semantic affects another, and the most common entanglements are the entanglement of age with glasses and the entanglement of gender with smile. The results show that by excluding important dimensions responsible for the changes in the logits of the entangled attributes during optimization, slightly different directions can be found where irrelevant semantics are less affected, while the target semantics can still be successfully manipulated.

[0096] Compare

[0097] Results from the methods and / or techniques taught in this application can be compared with InterFaceGAN [SYTZ20] and GANSpace [HHLP20], for example on three semantic attributes: smile, gender, and age. Attributes not supported by some previously known methods and implementations are not shown. For [SYTZ20], the SVM bounds use the same training data as used to train the GAN g and auxiliary network in Figure 1. For [HHLP20], the official published directions are used. Figure 6 Table 600 in FIG. 6 visualizes the directions found by each of the three implementations, where the first two are previously known, and the bottom row shows the results according to embodiments of the present invention (e.g., using the network architecture 102 with GAN g 108 and auxiliary network 110). For each set of three samples in a row, the image in the middle is the original StyleGAN2 synthesized image, and the left and right images correspond to the synthesized results based on the edited original latent code, where the directions found by each method are used to reduce / increase the target semantic score.

[0098] Overall, our method outperforms the unsupervised GANSpace method (which generates less realistic images and does not change the target semantics significantly), and finds similar directions as InterFaceGAN. Nevertheless, it is important to note that our method performs better in disentangling attributes using Grad-CAM-inspired channel filtering than the conditional manipulation technique proposed in [SYTZ20], which adjusts the target direction by subtracting its projection vector on the direction used to edit the entangled semantics, and Figure 7 Some examples are shown in .

[0099] Form 700( Figure 7) shows a comparison of attribute disentanglement results for the method disclosed herein with only one previously known method, InterFaceGAN [SYTZ20]. The method disclosed herein can better disentangle attributes during manipulation. In the gender-smile entanglement example in the first two rows, the smile attribute is better preserved when the gender is edited in the second of these rows (i.e., the method disclosed herein). In the age-glasses entanglement example in the bottom two rows, the method disclosed herein finds disentangled controls that better preserve unrelated semantics (e.g., the facial expression and hairstyle of the person).

[0100] In U.S. patent application Ser. No. 16 / 683,398, filed by the applicant on November 14, 2019 and entitled “System and Method for Augmented Reality by translating an image using Conditional cycle-consistent Generative Adversarial Networks (ccGANs)”, examples of the use of GANs are shown and described, and the U.S. patent application is incorporated herein by reference.

[0101] In an embodiment, the disclosed techniques and methods include developer-related methods and systems to define (such as by adjusting) a CNN architecture including a GAN g and an auxiliary classifier f for classifying semantic attributes. The GAN can be controlled to interpolate semantic attributes according to training. User-related methods and systems are also shown, such as using a trained generator model (e.g., generator g(x)) to process a source image x at runtime to perform an image-to-image conversion to obtain a synthetic image x′=g(x) with (or without) a specific semantic attribute (or multiple attributes).

[0102] Figure 8 800 according to an embodiment. The GUI 800 may be presented via a display device (not shown). In an embodiment, a user may specify one or more target semantic attributes and a desired amount of the attribute. For example, +smile, -age may identify the attribute, and an amount (such as a percentage) may specify the amount that will be present in the synthesized image when the generator is called. Since the interpolation process is continuous, a pre-trained image classifier may be used to quantify the amount (or amount of change) of the semantic attribute and find the desired manipulation strength.

[0103] GUI 800 shows a plurality of input controls 802, which are configured to receive selective input for each target semantic attribute configured and trained for CNN architecture 102. In the present embodiment, each input control (e.g., 802A) shows a slider type control for a corresponding single attribute and is used to identify a granularity input. Other types of input controls (radial controls, text boxes, buttons, etc.) can be used, such as for granular input or other input. The control can receive a % value, a range selection, a scalar, or other value. For example, an age attribute may be associated with an integer range and accept a granular input of age in years or decades, etc. Others can select a relative range of attribute presence, such as small, medium, and large. These relative ranges can be associated with corresponding granularity values, such as 15%, 50%, and 85%, or other values.

[0104] By selecting the "Apply" control 804, the semantic attribute input or multiple semantic attribute inputs are applied when the generator is called. The output of the generator is controlled by the semantic attribute controls from the corresponding auxiliary network semantic attribute classifier. One or more semantic attribute inputs are applied to the source image x (806) through these attribute controls to generate a composite image x'808 (e.g., an output image). The composite image can be represented as x'=g(x), as controlled by the attribute controls. And the attribute controls are responsive to the semantic attribute inputs from the interface controls 802. Although all available semantic attributes have controls, the user can choose not to change the attribute, and therefore the generator does not need to interpolate along the direction identified within the associated semantic control.

[0105] The source image 806 may be identified for use (eg, uploaded, copied from storage, self-portrait obtained via camera input, etc.) via a “source image” control 810 . The resulting composite image x′ 808 may be saved via a “save image” control 812 .

[0106] In an embodiment, the attributes are untangled before the controls are applied when the generator is called, so that a composite image with minimized entanglement is generated. In an embodiment, untangling can be enabled or disabled by one or more controls (not shown). For example, in an embodiment, age and glasses can be selectively untangled, or smile and gender can be selectively untangled, or both can be untangled.

[0107] Although in the illustrated embodiment a separate semantic attribute control is provided for each trained attribute, in an embodiment (not shown), fewer controls are provided (e.g., only for a single attribute or two attributes, etc.). Although separate controls are provided for each attribute, in an embodiment (not shown), a single control may be provided for a combined attribute (e.g., age and gender). The multi-attribute controls are calculated by a vector algorithm. In order to make the face have less smile, more glasses, and older (older age), the GUI is configured to receive inputs for the corresponding directions, i.e., -smile, +glasses, +age. According to an embodiment of the operation of the computing device, the inputs are associated with the corresponding gradient vectors (of the associated semantic attribute controls), and these vectors are added and normalized. The generator performs (linear) interpolation along the calculated direction (i.e., the combined direction) to generate an output image.

[0108] In an embodiment, the GUI is provided by a first computing device, and the generator is provided by another computing device remotely located relative to the first computing device. Another computing device may provide a controllable GAN generator as a service. The first computing device providing the GUI is configured to transmit an input image and a semantic attribute direction input (e.g., as a percentage value or other input) to a remotely located computing device, which provides (e.g., executes) the generator. Such a remotely located computing device may provide an application programming interface (API) or other interface to receive a source image (or selection) and a semantic attribute direction input. The remotely located computing device may calculate a direction vector and call a generator that applies a semantic attribute control. In an alternative embodiment, the computing device providing the GUI may be the same as the computing device providing the generator.

[0109] In embodiments, in addition to developer (e.g., used at training) and target (used at inference) computing device aspects, a person of ordinary skill will appreciate that a computer program product aspect is disclosed, wherein instructions are stored in a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, optical disk, etc.), which, when executed, causes the computing device to perform any method aspect disclosed herein. In embodiments, the computing device includes a processor (e.g., a microprocessor (e.g., a CPU, a GPU, a plurality of any of them), a microcontroller, etc.), which executes computer-readable instructions, such as computer-readable instructions stored in a storage device. In embodiments, the computing device includes (e.g., "dedicated") circuitry, which, for example, performs the functions of the instructions without reading those instructions.

[0110] Aspects related to the e-commerce system are also shown and described. In embodiments, a user's computing device is configured as a client computing device associated with the e-commerce system. For example, the e-commerce system stores a computer program for such a client computing device. Thus, the e-commerce system has as a component thereof a computer program product that stores instructions that, when executed by a client computing device (e.g., a processing unit thereof), configures such a client computing device. These and other aspects will be apparent.

[0111] Fig. 9 is a block diagram of a computer system 900. In an embodiment, the computer system 900 includes a plurality of computing devices, which in an embodiment include servers, developer computers (PCs, laptops, etc.), and mobile devices (such as smartphones and tablets, etc.). A network model training environment 902 is shown, which includes hardware and software for defining and configuring (such as by adjusting) a GAN-based CNN architecture 102. The architecture 102 includes a GAN-based generator 108 (e.g., generator g) and an auxiliary network 110 of a classifier (e.g., multiple mapping functions f). In an embodiment, the GAN and the mapping function are adjusted in coordination, and the latent space z of the GAN is updated based on the results of the indicated semantic function.

[0112] In an embodiment, once the generator 108 and auxiliary network 110, collectively referred to as CNN 102, are trained, they can be set up for use on a target device, such as one of the mobile devices 910, 912 of the system 900 or other devices (such as 913). Mobile device 910 is a representative user computing device, for example, a consumer user. It should be understood that these users can use other forms of computing devices, such as desktop computers, workstations, etc. Device 913 represents a computing device for performing training data generation (and therefore a training data generation device). In this embodiment, such a computing device uses the generator 108 to generate additional image data, which can be easily labeled with semantic attributes and can be used to train the network model (e.g., in a supervised manner). The form factor of device 913 can be a server, laptop, desktop computer, etc., and is not necessarily a consumer-type mobile device, such as a tablet computer, smartphone, etc.

[0113] In an embodiment, the network model training environment 902 employs, at least in part, a GAN model (generator 108) pre-trained for an image task (e.g., facial image generation). The generator 108 is pre-trained, such as by using an image dataset 914 stored at a data server 916. In an embodiment, the generator 108 is a model developed "in-house". In an embodiment, the generator 108 is publicly available, such as through an open source license. Datasets may also be developed and provided in a similar manner. Depending on the type of image task and the network architecture (e.g., supervised), the training is supervised and the dataset is annotated based on such training. In other scenarios, the training is unsupervised and the data is defined accordingly. In an embodiment, the GAN model (generator 108) is pre-trained with its Figure 1A The presented form is further conditioned, for example using labeled face images for semantic attributes.

[0114] In an embodiment, the generator 108 and the auxiliary network 110 are incorporated into an augmented reality (AR) application 920. Although not shown, in an embodiment, the application is developed by an application developer computing device for a specific target device having specific hardware and software (specifically, an operating system configuration). In an embodiment, the AR application 920 is a native application configured to execute in a specific native environment, such as an environment defined for a specific operating system (and / or hardware). In an embodiment, the AR application 920 takes the form of a browser-based application, for example, configured to execute in a browser environment of the target device.

[0115] In an embodiment, AR application 920 is distributed (e.g., downloaded) through user devices such as mobile devices 910 and 912. Native applications are typically distributed through an application distribution server 922 (e.g., a "store" operated by a third-party service), but this is not required.

[0116] In an embodiment (not shown), the AR application 920 itself does not include the CNN architecture 102 (excluding the generator and auxiliary networks). Instead, the application is configured with an interface for communicating with a remote device that provides these components as a service (not shown) (e.g., as a cloud-based service). The storage and execution of the generator and auxiliary networks are resource intensive and may be too large or demanding for some computing devices. Other reasons may also factor into the paradigm of the AR application.

[0117] In an embodiment, the AR application 920 is configured to provide an augmented reality experience to the user (e.g., via an interface). For example, an effect is added to an image by processing by the generator 108. The mobile device includes a camera (not shown) to capture an image (e.g., a still image or a video image, whether or not it is a selfie image). In an embodiment, the effect is applied to the image in real time, for example, to a video image (and displayed on a display device of the mobile device) to simulate the effect on the user when the video is captured. When the position of the camera changes, the effect is applied in response to the image (multiple images) of the captured video, thereby simulating augmented reality. As will be understood, real-time operation is limited by processing resources. In an embodiment, the effect is not simulated in real time, but is delayed, which may affect the augmented reality experience.

[0118] In an embodiment, the computing device is coupled for communication via one or more networks (eg, 922), including wireless networks or otherwise, public networks or otherwise, and the like.

[0119] By way of example and not limitation, the e-commerce system 924 is web-based and provides a browser-based AR application 920A as a component of the e-commerce service provided by the e-commerce system 924. The e-commerce system 924 includes a configured computing device and a data storage device 8926 (e.g., a database or other configuration). The data storage device 926 stores data about products, services, and related information (e.g., technology for applying products). The data storage device 926 or other data storage devices (not shown) stores recommendation rules or other forms of product and / or service recommendations, etc., to help users choose from available products and services. Products and services are presented via a user experience interface displayed on the user's (mobile) computing device. It will be understood that the e-commerce system 924 is simplified.

[0120] In an embodiment, the browser-based AR application 920A (or AR application 920) provides an augmented reality customer experience, such as to simulate products, technologies, or services provided or facilitated by the e-commerce system 924. It will be understood that in this embodiment, the AR application 920 is also configured to provide e-commerce services, such as via a connection with the e-commerce service 924.

[0121] By way of example and not limitation, products include beauty (e.g., makeup) products, anti-aging or rejuvenation products, and services include beauty, anti-aging or rejuvenation services. Services include treatments or other procedures. Products or services involve a part of the human body, such as the face, hair, or nails. In an embodiment, therefore, a computing device (such as a mobile device 912) so configured provides a facial effect unit 912A including a processing circuit, which is configured to apply at least one facial effect to a source image and generate (e.g., displayed) one or more virtual instances of the source image with the applied effect on an e-commerce interface of the computing device facilitated by the e-commerce system. In an embodiment, the facial effect unit 912A utilizes a generator (g) and an auxiliary network 110 based on a generative adversarial network (GAN) such as described herein to generate the source image with the applied effect. In an embodiment, the computing device provides a user experience unit 912B including a processing circuit, which is used to determine at least one product or service from a data storage device 926 and generate one or more virtual instances of recommendations on an e-commerce interface to purchase the product or service. In an embodiment, at least one product is associated with a corresponding facial effect, and the facial effect unit applies the corresponding facial effect to provide a virtual try-on experience.

[0122] In an embodiment, the user experience unit 912B is configured to present a graphical user interface (e.g., a browser-based interface or other form) to work in conjunction with the computing device 912 and the e-commerce system 924. In an embodiment, the e-commerce system 924 is therefore configured to provide an AR application for execution by a client computing device such as a mobile device (e.g., 912), and is cooperatively configured to provide e-commerce services to the client computing device to facilitate AR-simulated (product / service) recommendations and purchases via the client computing device (e.g., 912).

[0123] Therefore, any computing device (particularly a mobile device) provides a computing device to convert an image from a first domain space to a second domain space. The computing device includes a storage unit storing a generator (g) based on a generative adversarial network (GAN), which is configured to generate an image controlled for a semantic attribute. In an embodiment, the computing device includes a processing unit configured to (e.g., via an AR application 920): receive a source image; receive input to identify at least one semantic attribute to be controlled, for example, including input to refine the semantic attribute (e.g., percentage); in response to the semantic attribute input, provide the image to the generator g to obtain a synthesized (e.g., new) image; and provide the new image for presentation.

[0124] In an embodiment, the generator is configured to synthesize specific semantic attributes including one of: facial features, including age, gender, smile, or other facial features; posture effects; makeup effects; hairstyle effects; nail effects; cosmetic surgery or dental effects, including one of rhinoplasty, lift, blepharoplasty, implants, otoplasty, teeth whitening, orthodontics, or other cosmetic surgery or dental effects; and appliance effects, including one of eye appliances, mouth appliances, ear appliances, or other appliance effects.

[0125] In an embodiment, the synthesized output of the generator g (e.g., a synthesized image or a new image) is provided to a second network model (e.g., as an unshown component of the AR application 920 or 920A), for example, for a facial editing task. The second network model may include a second generator or feature detector and simulator to apply an effect (e.g., to generate another new image). The effect may be presented in an AR interface. In an embodiment, the processing unit is configured to provide a synthesized image (or another new image defined therefrom) in an augmented reality interface to simulate an effect applied to the image. In an embodiment, the effect includes any of a makeup effect, a hair effect, a nail effect, a cosmetic surgery or dental effect, an appliance effect, or other simulated effect applied. In an embodiment, the source image includes an applicable portion (e.g., a face, hair, nails, or a body part) of an object (such as a user of the device). For example, an input image may be processed by a generator to simulate age changes, and the effect pipeline may process a synthesized image showing age changes to apply makeup and / or hair effects to the synthesized image.

[0126] The network model training environment 902 provides a computing device configured to perform a method, such as a method configured by adjusting a GAN-based generator and an auxiliary network of a classifier sharing the same latent space. It will be understood that the implementation of the computing device aspect of the network model training environment 902 and any related implementations such as a generator or model are applied to the training method aspect by appropriate adaptation.

[0127] Fig.101000 is a flow chart of operation 1000 according to an embodiment of the present invention. In an embodiment, the operation provides a method such as for training (e.g., by adjustment). At step 1002, the operation is configured by adjusting the generator g108 and the training auxiliary network 110 based on a generative adversarial network (GAN). The training updates the latent space Z. In an embodiment, the generator g 108 and the training auxiliary network 110 are trained using labeled images with corresponding semantic attribute labels. The collaborative training uses the latent space Z to train the auxiliary network 110, and the corresponding gradient of the auxiliary network 110 (e.g., where each semantic attribute has a corresponding binary classifier) ​​is very useful for controlling the operation of the GAN-based generator when generating images.

[0128] At step 1004, the operation provides the generator g 108 and the auxiliary network 110 for use by the computing device to generate an image. Related computing device and computer program product aspects, as well as other aspects, will be apparent.

[0129] Fig.11 is a flow diagram of operation 1100 according to embodiments herein. Operation 1100 is performed by a computing device, such as device 910, 912, 913, or 924.

[0130] At 1102, operations provide a generator and an auxiliary network that share a latent space. The generator is configured to generate a synthetic image that exhibits human-interpretable semantic attributes. The auxiliary network includes a plurality of semantic attribute classifiers, including a semantic attribute classifier for each semantic attribute to be controlled (by the generator) when generating a synthetic image from a source image. Each semantic attribute classifier is configured to classify the presence of (associated) one of the semantic attributes in the image and provide meaningful directions for controlling one of the semantic attributes in the synthetic image of the generator.

[0131] When the generator is called to generate a composite image from a source image, the operation at 1104 applies a corresponding semantic attribute control to control a corresponding semantic attribute in the composite image generated from the source image. The corresponding semantic attribute control is responsive to a meaningful direction provided by a classifier associated with the semantic attribute.

[0132] Practical applications of the generator g include: generating and providing (large) amounts of labeled synthetic data so as to be more suitable for training (or further training) one or more network models. For example, these data can be used to improve the effect of a facial editing model. For example, for a hair segmentation task, the segmentation model may initially be trained on unbalanced data with uneven distribution of hair colors, which results in unsatisfactory segmentation results for minority hair color groups. In order to solve this problem and create a more balanced dataset, the generator g and / or techniques in this article can be used to edit the hair color of existing labeled data and generate a large amount of data for minority groups. Therefore, using the generator 108 (and the network 110) includes: generating labeled synthetic images for training and models, and may also include: using labeled synthetic images to train models.

[0133] Actual implementations may include any or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, devices, systems, means for performing functions, program products, and otherwise combining the features described herein. A number of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided, or steps may be deleted from the described processes, and other components may be added to or removed from the described systems. Therefore, other implementations are within the scope of the following claims.

[0134] Throughout the description and claims of this specification, the words "comprises" and "comprising" and their variations mean "including but not limited to", and they are not intended to (and do not) exclude other components, integers or steps. Throughout this specification, the singular encompasses the plural, unless the context requires otherwise. In particular, where the indefinite article is used, this specification should be understood as contemplating the plural as well as the singular, unless the context requires otherwise.

[0135] Features, integers, characteristics or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood to be applicable to any other aspect, embodiment or example unless incompatible therewith. All features disclosed herein (including any accompanying claims, abstracts and drawings) and / or all steps of any method or process so disclosed may be combined in any combination, except for at least some mutually exclusive combinations of such features and / or steps. The invention is not limited to the details of any foregoing examples or embodiments. The invention extends to any novel feature or any novel combination of features disclosed in this specification (including any accompanying claims, abstracts and drawings), or to any novel step or any novel combination of steps of any method or process disclosed.

[0136] References

[0137] [ACB17]Martin Arjovsky,Soumith Chintala,and Léon Bottou.Wassersteingenerative adversarial networks.In International conference on machinelearning,pages 214-223.PMLR,2017.

[0138] [BDS18]Andrew Brock,Jeff Donahue,and Karen Simonyan.Large scale gantraining for high fidelity natural image synthesis.arXiv preprint arXiv:1809.11096,2018.

[0139] [CCK + 18]Yunjey Choi,Minje Choi,Munyoung Kim,Jung-Woo Ha,Sunghun Kim,and Jaegul Choo.Stargan:Unified generative adversarial networks for multi-domain image-to-image translation.In Proceedings of the IEEE conference oncomputer vision and pattern recognition,pages 8789-8797,2018.[CVB21]AntonCherepkov,Andrey Voynov,and Artem Babenko.Navigating the gan parameter spacefor semantic image editing.In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition,pages 3671-3680,2021.

[0140] [GPAM +14]Ian Goodfellow,Jean Pouget-Abadie,Mehdi Mirza,Bing Xu,DavidWarde-Farley,Sherjil Ozair,Aaron Courville,and Yoshua Bengio.Generativeadversarial nets.Advances in neural information processing systems,27,2014.

[0141] [HHLP20]Erik Aaron Hertzmann,Jaakko Lehtinen,and SylvainParis.Ganspace:Discovering interpretable gan controls.arXiv preprint arXiv:2004.02546,2020.

[0142] [HRU + 17]Martin Heusel,Hubert Ramsauer,Thomas Unterthiner,BernhardNessler,and Sepp Hochreiter.Gans trained by a two time-scale update ruleconverge to a local nash equilibrium.Advances in neural informationprocessing systems,30,2017.

[0143] [KLA18]Tero Karras,Samuli Laine,and Timo Aila.A style-based generatorarchitecture for generative adversarial networks.arXiv preprint arXiv:1812.04948,2018.

[0144] [KLA +20]Tero Karras,Samuli Laine,Miika Aittala,Janne Hellsten,JaakkoLehtinen,and Timo Aila.Analyzing and improving the image quality ofstylegan.In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition,pages 8110-8119,2020.

[0145] [LLWT15]Ziwei Liu,Ping Luo,Xiaogang Wang,and Xiaoou Tang.Deeplearning face attributes in the wild.In Proceedings of the IEEE internationalconference on computer vision,pages 3730-3738,2015.

[0146] [PBH20]Antoine Plumerault,HervéLe Borgne,and CélineHudelot.Controlling generative models with continuous factors ofvariations.arXiv preprint arXiv:2001.10238,2020.

[0147] [SCD + 17]Ramprasaath R Selvaraju,Michael Cogswell,Abhishek Das,Ramakrishna Vedantam,Devi Parikh,and Dhruv Batra.Grad-cam:Visual explanationsfrom deep networks via gradient-based localization.In Proceedings of the IEEEinternational conference on computer vision,pages618-626,2017.

[0148] [SYTZ20]Yujun Shen,Ceyuan Yang,Xiaoou Tang,and BoleiZhou.Interfacegan:Interpreting the disentangled face representation learnedby gans.IEEE transactions on pattern analysis and machine intelligence,2020.

[0149] [SZ21]Yujun Shen and Bolei Zhou.Closed-form factorization of latentsemantics in gans.In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition,pages 1532-1540,2021.

[0150] [VB20]Andrey Voynov and Artem Babenko.Unsupervised discovery ofinterpretable directions in the gan latent space.In International Conferenceon Machine Learning,pages 9786-9796.PMLR,2020.

[0151] [WDB + 19]Yan Wu,Jeff Donahue,David Balduzzi,Karen Simonyan,and TimothyLillicrap.Logan:Latent optimisation for generative adversarial networks.arXivpreprint arXiv:1912.00953,2019.

[0152] [WLS21]Zongze Wu,Dani Lischinski,and Eli Shechtman.Stylespaceanalysis:Disentangled controls for stylegan image generation.In Proceedingsof the IEEE / CVF Conference on Computer Vision and Pattern Recognition,pages12863-12872,2021.

[0153] [ZKL + 16]Bolei Zhou,Aditya Khosla,Agata Lapedriza,Aude Oliva,andAntonio Torralba.Learning deep features for discriminative localization.InProceedings of the IEEE conference on computer vision and patternrecognition,pages 2921-2929,2016.

Claims

1. A method comprising: Generating a composite image (g(z')) from a source image using a generator (g) having a latent code (z), the generator manipulating a target semantic attribute (k) in the composite image, wherein the generating comprises: Finding a meaningful direction at z for the target semantic attribute k, the meaningful direction is identified by classifying corresponding semantic attributes at z through an auxiliary network, the corresponding semantic attributes including the target semantic attribute k; defining z' by optimizing z in response to the meaningful direction of the target semantic attribute k; and The composite image is output.

2. The method according to claim 1, wherein: The auxiliary network includes a set of binary classifiers, one for each semantic attribute that the generator can manipulate, and the auxiliary network is trained jointly with the generator g to share the latent code space Z.

3. The method according to claim 1 or claim 2, wherein: The corresponding semantically meaningful data directions include corresponding dimensional data vectors obtained from each individual classifier, each vector including the fastest increasing direction and rate in the individual classifier.

4. The method according to any one of claims 1 to 3, comprising: disentangle the data direction by filtering out from the dimensions of the semantically meaningful data direction of the target semantic attribute those important dimensions of the corresponding semantically meaningful data direction of each other semantic attribute (m, m≠k) entangled with the target semantic attribute; The importance of a particular dimension is based on the absolute gradient magnitude of the particular dimension.

5. The method according to claim 4, wherein: Filtering includes: evaluating each dimension of each semantically meaningful data direction to be disentangled from the target semantically meaningful data direction; if a particular dimension exceeds a threshold, setting the value of the corresponding dimension in the target semantically meaningful data direction to a zero value.

6. The method according to claim 5, wherein: The potential code z is optimized to: in: is a vector representing the direction of the semantically meaningful data of the target before filtering; is a vector representing the direction of the data that is semantically meaningful for the target after filtering; and α is a hyperparameter that controls the interpolation direction and step size.

7. The method according to any one of claims 1 to 6, comprising: Repeat about potential code z ′ The discovery, definition, and optimization of operations to further manipulate the semantic attribute k.

8. The method according to any one of claims 1 to 7, comprising: Generate synthetic images to manipulate multiple target semantic attributes; For each of the target semantic attributes, a corresponding meaningful direction at z is found, the meaningful direction being identified by the auxiliary network classifying each of the target semantic attributes at z; and z is optimized in response to each of the meaningful directions.

9. The method according to any one of claims 1 to 8, comprising: Another network model is trained using the synthetic images.

10. The method according to any one of claims 1 to 9, comprising: An input is received, the input identifying at least one of the corresponding semantic properties to be controlled relative to the source image.

11. The method according to claim 1, comprising any one or more of the following: Providing the generator and the auxiliary network to generate the synthetic image as a service; Provide an e-commerce interface to purchase products or services; Provide a recommendation interface to recommend products or services; or An augmented reality interface is provided using the composite image to provide an augmented reality experience.

12. The method according to any one of claims 1 to 11, wherein: The target semantic attribute includes one of the following: facial features, including age, gender, smile, or other facial features; a pose effect; a makeup effect; a hair effect; a nail effect; a cosmetic or dental effect, including one of rhinoplasty, lift, blepharoplasty, implants, otoplasty, teeth whitening, orthodontics, or other cosmetic or dental effects; or an appliance effect, including one of an eye appliance, mouth appliance, ear appliance, or other appliance effect.

13. A method comprising the steps of: providing an augmented reality (AR) interface to provide an AR experience, the AR interface being configured to generate a synthetic image from a received image using a generator by applying a corresponding semantic attribute control responsive to a meaningful direction provided by a classifier associated with the semantic attribute; as well as The received image is received, and the composite image is provided for use in the AR experience.

14. The method according to claim 13, comprising: The composite image is processed using an effects pipeline to simulate an effect, and the composite image with the simulated effect is provided for presentation in the AR interface.

15. A computing device comprising at least one processor and at least one non-transitory storage device storing computer-readable instructions for execution by the at least one processor, the instructions causing the computing device to perform the method according to any one of claims 1 to 14.

Images