METHODS AND APPARATUS FOR DETERMINING AND USING CONTROLLABLE DIRECTIONS OF GAN SPACE

Abstract

METHODS AND APPARATUS FOR DETERMINING AND USING CONTROLLABLE DIRECTIONS IN GAN SPACE. Methods, apparatus, and techniques relating to determining directions in a latent GAN space and obtaining disentangled controls over the GAN output semantics, for example, to enable its use in generating synthesized images such as for training another model or creating augmented reality. The present methods, apparatus, and techniques, according to the embodiments, use gradient directions from auxiliary networks to control the semantics in latent GAN codes. It is demonstrated that it is possible to use minimal amounts of labeled data with sizes as small as 60 samples, which data can be obtained rapidly under human supervision.Also shown here, according to embodiments, is the selection of important latent code channels with masks during manipulation, resulting in more complex controls. Figure for the abstract: none.

Description

Title of the invention: METHODS AND APPARATUS FOR DETERMINING AND USING CONTROLLABLE DIRECTIONS OF GAN SPACE

[0001] DOMAIN

[0002] This disclosure relates to image processing using artificial intelligence and deep neural networks, including generative adversarial networks (GANs), such as for creating augmented reality. More specifically, this disclosure relates to methods and devices for determining and using controllable spatial directions of GANs.

[0003] CONTEXT

[0004] GANs have been successfully applied to address various tasks such as simulating makeup effects on the human face or changing hair color to provide realistic augmented reality (AR) technology in the beauty industry. However, conditional generative models that offer controls over the output semantics generally require large amounts of labeled training data, which can be expensive and time-consuming to obtain. Furthermore, artifacts and attribute entanglements sometimes appear in current face retouching results, which can lead to a reduction in the overall user experience.

[0005] It is desirable to obtain controls over the output semantics of the original GAN, with a view to improving the user experience by removing artifacts or disentangling attributes when retouching human faces and generating large amounts of labeled synthetic data to form conditional models.

[0006] SUMMARY

[0007] The present methods, apparatus, and techniques relate to determining directions in a GAN latent space and obtaining disentangled controls over the GAN output semantics, for example, to enable its use in generating synthesized images such as for training another model or creating augmented reality. According to certain embodiments, the present methods, apparatus, and techniques use gradient directions of auxiliary networks to control the semantics in GAN latent codes. It is shown that it is possible to use minimal amounts of labeled data with sizes as small as 60 samples, which data can be obtained rapidly under human supervision. Also shown here, according to certain embodiments, is the selection of channels from latent codes. wearing masks during handling, resulting in more complex checks.

[0008] The invention relates to a computer device comprising:

[0009] a processor and a storage device, the storage device containing instructions which, when executed by the processor, cause the computing device to:

[0010] providing a generator and an auxiliary network sharing a latent space, the generator being configured to generate synthesized images exhibiting human-interpretable semantic attributes and the auxiliary network comprising a plurality of semantic attribute classifiers, including a semantic attribute classifier for each semantic attribute to be controlled when generating a synthesized image from a source image, each semantic attribute classifier being configured to classify the presence of one of the semantic attributes in images and provide a meaningful direction for controlling one of the semantic attributes in the generator's synthesized images; and

[0011] when requesting the generator to generate the synthesized image from the source image, apply a respective semantic attribute control to control a respective semantic attribute in the synthesized image, the respective semantic attribute control responding to the meaningful direction provided by the classifier associated with the semantic attribute.

[0012] The invention may further have one or more of the following features:

[0013] - the significant direction includes a gradient direction, and the instructions lead the computer system to calculate a respective semantic attribute control from a respective gradient direction of the classifier associated with the semantic attribute;

[0014] - to control two or more semantic attributes, the instructions bring the computer system to combine each respective gradient direction of each classifier associated with the two or more semantic attributes to be ordered;

[0015] - the instructions cause the computer device to disentangle the attribute control respective semantics before application to the generator;

[0016] - each respective semantic attribute control includes a direction vector of respective gradient calculated from parameters of the respective semantic attribute classifier associated with the respective semantic attribute control; and

[0017] disentangling a respective semantic attribute control from another respective semantic attribute control consists of removing important data dimensions from the gradient direction vector associated with one respective semantic attribute control from the gradient direction vector associated with the other respective semantic attribute control;

[0018] - the dimensions of important data removed meet an identifying threshold these data dimensions having an absolute value greater than or equal to the threshold;

[0019] - if the z-th dimension of data for the gradient direction vector associated with the the respective semantic attribute control is identified by the threshold, the corresponding z-th data dimension for the gradient direction vector associated with the other respective semantic attribute control is reset to zero;

[0020] - the instructions cause the computer device to receive an identifying input at least one of the respective semantic attributes to be controlled in relation to the source image;

[0021] - the entry identifying at least one of the respective semantic attributes includes a granular input to determine a quantity of semantic attribute to apply when generating the synthesized image;

[0022] - the instructions cause the computer device to perform any one or several of the actions consisting of:

[0023] provide the generator and auxiliary network to generate the synthesized images as a service;

[0024] provide an e-commerce interface to purchase a product or service;

[0025] provide a recommendation interface to recommend a product or a service; and

[0026] provide an augmented reality interface using the synthesized image to provide an augmented reality experience. Brief description of the drawings

[0027] [Fig. IA] [Fig. 1B][Fig. 1C] Figures IA, IA, and IA illustrate an overview of training and / or architecture aspects disclosed in this document, according to their respective embodiments. Fig. IA is a graphical representation of a computer device providing a convolutional neural network (CNN) according to one embodiment. Fig. IA is a graphical representation of the respective gradient dimensions for semantic attributes k and m according to one embodiment. Fig. 1C is a flowchart of operations for a one-step optimization, according to one embodiment.

[0028] [Fig.2] The [Fig.2] is a table of images to illustrate the disentanglement of semantic attributes, namely, the smile of the glasses, on facial images in accordance with one embodiment.

[0029] [Fig.3] The [Fig.3] is a pseudocode listing operations, according to an embodiment.

[0030] [Fig.4] Fig.4 is an array of images to illustrate results of manipulating semantic attributes using a controllable neural network, in accordance with a method of implementation.

[0031] [Fig.5] The [Fig.5] is an array of images to illustrate results of semantic attribute disentanglement using a controllable neural network, according to one embodiment.

[0032] [Fig.6] The [Fig.6] is an array of images visualizing directions found to compare three controllable neural networks, namely two previously known controllable neural networks and one controllable neural network according to a present embodiment.

[0033] [Fig.7] The [Fig.7] is an array of images showing a comparison of attribute untangling results obtained by a previously known controllable neural network and a controllable neural network according to a present embodiment.

[0034] [Fig.8] The [Fig.8] is a functional diagram of a graphical user interface (GUI) for producing a synthesized image from a source image using a controllable GAN, according to one embodiment.

[0035] [Fig.9] The [Fig.9] is a functional diagram of a computer system, according to one embodiment.

[0036] [Fig.10][Fig.11] Figs. 10 and 11 are flowcharts showing operations, in accordance with the respective present embodiments.

[0037] The present concept is best described by means of some of its embodiments, which are described herein with reference to the accompanying drawings, on which similar numerals refer to similar features throughout. It should be understood that the term "invention," when used herein, is intended to designate the inventive concept underlying the embodiments described below and not merely the embodiments themselves. It should also be understood that the general inventive concept is not limited to the illustrative embodiments described below and that the following descriptions should be read from this perspective. DETAILED DESCRIPTION

[0038] Although GAN models are capable of generating highly realistic images due to random inputs to their latent space, the generation process is usually a black box, hence the lack of direct control over the output semantics. Nevertheless, previous work [VB20, VB20, SZ21, HHLP20, SYTZ20] has shown that meaningful directions and channels exist in the GAN latent space, and linear interpolation in these directions or modification of individual channel values ​​leads to interpretable transformations, e.g., adding glasses or a smile to a person's face. One objective of the methods, apparatus, and techniques described herein is to determine (e.g., learn) directions in the GAN latent space and to obtain disentangled controls over the semantics. GAN output mantic analysis, for example, to enable the use of such learned directions to create augmented realities, generate synthesized images to train a model, etc. The focus here is on the task of retouching human faces, useful in the beauty industry as well as others. To this end, the present methods, apparatus, and techniques, according to embodiments, use the gradient directions of auxiliary networks to control the semantics in latent GAN codes. It is shown that it is possible to use minimal amounts of labeled data with sizes as small as 60 samples, which data can be obtained rapidly under human supervision. Also shown here, according to embodiments, is the selection of important latent code channels with Grad-CAM-based masks during manipulation, resulting in more disentangled controls.The following section reviews previous work in relevant fields to contextualize the present teachings.

[0039] Generative adversarial networks (GANs), introduced by [GPAM+ 14], currently dominate the field of generative modeling with their strong ability to synthesize photorealistic images. In general, GANs consist of two networks, including a generator that learns a mapping from its latent space onto the image space, and a discriminator that distinguishes GAN-synthesized images from real images. The two networks are jointly trained in an adversarial manner. Different GAN variants have been proposed to improve production fidelity and training stability [ACB17, BDS18, CCK+18, KLA18, KLA+20], and there is a growing research interest in studying the latent space of these original GAN ​​models, which offers more control over the generation process.

[0040] Latent semantic manipulation for GANs has been investigated by several previous works [VB20, VB20, SZ21, HHLP20, SYTZ20, PBH20], and it has been shown that meaningful directions and channels exist in the latent space / feature space of GANs. One line of work discovers controls over the output semantics for unconditional GANs with explicit supervision [SYTZ20, WLS21]. For example, InterFaceGAN [SYTZ20] assumes that for any binary semantics (e.g., male vs. female), there exists a hyperplane serving as a separation boundary in the latent space, and its normal vector represents the meaningful direction.Using pre-trained classifiers on the CelebA dataset [LLWT15] to generate pseudo-labels of GAN-synthesized images, separation boundaries are then learned using state vector machines (SVMs) trained on data paired with GAN latent codes and corresponding semantic labels. [WLS21] discovers channels that control localized effects in a GAN activation space with the help of semantic segmentation networks. pre-trained. However, the range of controls found by these supervised methods is normally limited, as they require large amounts of labeled data or pre-trained neural networks, and the learned controls are sometimes entangled.

[0041] Another line of work discovers these controls in a self-supervised or unsupervised manner [HHLP20, CVB21, VB20]. For example, GANSpace [HHLP20] identifies important latent directions by applying principal component analysis (PCA) to vectors in the latent space or in the feature space of GAN, and [VB20] discovers interpretable directions in the latent space of GAN by jointly optimizing directions and a reconstructor that recovers these directions and manipulation forces from images generated on the basis of manipulated GAN latent codes. Although these methods do not require labeled data, they normally involve extensive manual examinations of different manipulation directions and the identification of significant controls. More importantly, and distinguishing themselves from techniques, etc.When taught in accordance with these embodiments, control of a target semantics is not always guaranteed.

[0042] Gradient-based knowledge has been exploited to improve the interpretability [ZKL+ 16, SCD+17] or the training stability [WDB+ 19] of neural networks. For example, [SCD+ 17] uses gradient information from classification outputs to obtain location maps that show visual evidence in images. [WDB+19] proposes increasing the stability of GAN training dynamics by adding an extra step before jointly optimizing the generator and discriminator, in which the latent code is first optimized towards regions considered more real by the discriminator, and the direction is obtained by computing the gradient with respect to the latent codes.

[0043] Contrary to [WDB+ 19] and in accordance with the present embodiments, which are hitherto unpublished in the literature, gradients of auxiliary binary classifiers are used to obtain controls over the output semantics of GANs. Thus, in accordance with the embodiments, methods, apparatus, and techniques are described here that allow obtaining disentangled controls in the latent space of GANs, which first find semantically significant directions by computing gradients of binary classifiers that denote different semantics as a function of the latent code inputs, and then select important dimensions in the latent code for attribute disentanglement during manipulation.

[0044] DETERMINATION OF SEMANTICALLY SIGNIFICANT DIRECTIONS

[0045] Fig. IA is a graphical representation of a computer device 100, according to one embodiment, comprising a CNN architecture 102 and a Image output (e.g., images 104 comprising 104A and 104B) as stored in a storage device 106 of device 100. Fig. 1B is a graphical representation of a portion of the storage device 106 storing the respective gradient dimensions 132 and 130 for semantic attributes k and , according to one embodiment. The graphic shows the untangling of the dimensions of a gradient for an attribute m with respect to the corresponding dimensions of a gradient for an attribute k. Fig. 1C is a flowchart of operations 150 of a one-step optimization, according to one embodiment, and as described in more detail below.

[0046] The CNN 102 architecture includes a controllable GAN S 108 configured to generate output images g(z) 104A and g(z) 104B from respective latent codes z and < (e.g., respective parts of the latent space Z of the GAN S 108). The images g(z) 104A and g(z') 104B show a facial image exhibiting different semantic attributes. In the illustrated example, the face of the individual in image g(z) 104A is not smiling, while the face of the same individual in image g(z) 104B is smiling.

[0047] The CNN architecture 102 further comprises a plurality of respective auxiliary networks 110, one for each semantic attribute to be controlled by the GAN 108. The plurality of auxiliary networks 110 comprise respective mapping functions f, which auxiliary networks 110 are formed with the GAN 108 to discover meaningful directions for each semantic attribute to be controlled. In one embodiment, the mapping functions f comprise binary classifiers. Fig. 1A shows only the details of box 112 relating to one of the respective classifiers of the auxiliary network 110 for simplicity, namely the mapping function fk 110A for the semantic attribute k. In Fig. 1 A, each among the GAN & 108 and the auxiliary networks 110 is pre-trained with training images for each of the semantic attributes (e.g. 60 images per semantic attribute).As shown in box 112, to manipulate a semantic attribute k, e.g., a smile, for a latent GAN code, the gradient direction nk 114 of the mapping function fk that maps latent GAN codes (e.g. ") onto semantic notes for k is used as input to derive a modified (i.e., £) for the GAN 8 108. The equation for deriving the modified '• is described in more detail below.

[0048] Reference is then made to Fig. IB. According to one embodiment, the semantic attributes are disentangled. The dimensions important for the manipulation of the semantic attribute m are suppressed with respect to the attribute k. According to one embodiment, the important dimensions are those with Large gradient amplitudes are present in nf 130 for the semantic attribute m. These dimensions are masked (e.g., set or defined) to a value of 0 in the gradient direction nk 112 for the semantic attribute k, where the gradient direction nk 132 refers to the gradient direction nk 112 after masking (e.g., after resetting important attributes to zero). The masking of dimensions is described in more detail below, particularly with respect to what constitutes a large gradient amplitude.

[0049] Reference is then made to Fig. IC where operations 150 show a pipeline for a one-step optimization. The operations begin at 152 with latent codes 4, as for facial images. In step 154, meaningful directions at c for the semantic attribute k are discovered (e.g., the gradient is calculated from the parameters of the corresponding mapping function formed / J). A decision in step 156 assesses whether there is entanglement with any semantic attribute m (where m represents a semantic attribute other than k that must be checked by GAN 8). If so, via the "Yes" branch up to step 158, meaningful directions at z for the semantic attribute m are discovered (e.g., calculated from the parameters of the corresponding mapping function formed f). In step 160, the operations perform an untangling, as described with reference to Fig. IB and in detail below.In one embodiment, the determination of an entanglement such as in 156 was identified experimentally. An entanglement was observed by editing each attribute using a small set of 30-image meta-validations along the original gradient direction of k, and then deciding which attributes are entangled. An association between the target attribute and any entangled attribute was tabulated. Important attributes arising from each of the identified attributes (i.e., those w * k) are masked. In one embodiment, a union of higher channels (e.g., the 100 largest gradient amplitudes) for the attributes to be disentangled is determined. These dimensions are then set to 0 in the target direction (e.g., to produce nk).

[0050] According to one embodiment, for example, without prior observation experience, it is possible to directly mask the z-th dimension value in the gradient nk in response to an amplitude of the z-th dimension value in the gradient nk1. In the first case, when there is entanglement, masking these dimensions helps with untangling; in the second case, there is no entanglement, and the dimensions important for m are insignificant in k, so that masking them does not change the direction too much.

[0051] In step 162, the operations optimize the latent codes S. The operations can be repeated (e.g., loop in step 154) with the next case of latent codes z until the desired manipulation result is obtained. Referring again to step 156, according to one embodiment, if there is no respective entanglement, the operations They proceed through the "No" branch up to step 162 to perform roptimization of the latent codes z. As noted, in one embodiment, the operations can mask the important dimensions of any other attribute even if there is no entanglement. For example, instead of determining which m attributes are entangled and then untangling the important dimensions of the gradient for k, it may be simpler (e.g., programmatically) to simply untangle the important dimensions of all m k.

[0052] Figure [Fig. 2] is a 200-image array comprising rows 202 and 204, for This illustrates the disentanglement of the smile from the glasses. The first column (left) of rows 202 and 204 corresponds to the original images. The second (middle) and third (right) columns, 208 and 210, are edited with smaller and larger interpolation distances. The difference is that in the second row, when the distance is greater (i.e., more of a smile is added compared to column 208), no glasses have been added. That is, the second row applies the disentanglement. Thus, row 202 shows the respective facial images of the same subject in each image. From left to right, the subject has: i) no smile and no glasses, ii) a smile and no glasses, and iii) a smile and glasses. On line 204, the images show the same subject as on line 202. The images of line 204 show the application of an untangling that is not applied to line 202.From left to right, the subject has: i) no smile and no glasses, ii) a smile and no glasses, and iii) a smile and no glasses.

[0053] During the one-step optimization operations, as shown with reference to Figs. IB and IC, the important dimensions of the gradient from f\ tt are masked to 0 in n^mnre. The nomenclature |nU indicates the absolute value of the gradient vector in dimension i. Unlike the original multiclass classification setting for Grad-CAM, where ReLU is chosen as the activation, the absolute value is used here, as in the binary classification setting. The dimensions that negatively influence the semantic score also contain relevant information.

[0054] In more detail, according to one embodiment, a method is proposed for determining (e.g., learning) meaningful instructions in the latent space of a GAN to manipulate the semantics in the GAN outputs. A well-trained GAN model learns a mapping that maps a d-dimensional latent code into the latent space Z of the GAN (e.g., ZQR). <! sur une image x = dans l’espace d’image X. La sémantique interprétable par l’homme existe dans l’espace d’image X et inclut, par exemple, l’âge, le sexe, les lunettes, le sourire, la posture de la tête d’une personne ou autre observation sémantique. Compte Given a series of notation functions Ai... for K semantics, K maps are obtained from the latent space Z to the semantic spaces C^.. Ck Ç R respectively, where:

[0055] । )

[0056] and where 4 denotes the / f-th semantic note in the image generated on the basis of the latent code S. According to one embodiment, each of the semantics is distinct. For example, instead of a single attribute "Smile with teeth," these attributes are separated into "smile" and "teeth." Control of several semantics simultaneously (smile with teeth) can be achieved by combining the control for smile and teeth (adding upward direction vectors).

[0057] Given the accuracy and independence of 51... sk, it is expected that when the k-th semantic attribute of the output image changes, its score ci will change accordingly, but other semantic scores will remain approximately the same. It is assumed that useful information is embedded in the mappings from GAN latent spaces to semantic spaces, which can be exploited to find semantically significant directions. In particular, to control d, it is proposed to interpolate the latent code along the gradient direction of such a mapping function. To simplify the computation, each mapping function 5^(7)) is parameterized, with a neural network fk trained on matched samples of GAN latent codes and corresponding semantic labels, generated by the original sk(g(z)) (e.g., the ground truth / human perception notation function). The direction is computed as follows:

[0058] ni = yf (A) = 2 '

[0059] where the vector ni indicates the direction and the fastest rate of increase in ci to z. The latent optimization technique (according to the embodiments), used during training to optimize the latent code, follows an assumption that the gradient of a GAN discriminator D points in the direction that indicates a better sample. A GAN discriminator can be considered as a rating function for "reality". This view is extended to multiple (e.g., K) semantics. Intuitively, for each semantic target, interpolating the latent code along the gradient from the rating function of the respective semantic attribute modifies the semantic rating, and the semantic attribute is manipulated accordingly. To manipulate the semantic attribute k once, z is updated with:

[0060] z = z + an* ( 3 )

[0061] where has a hyperparameter that controls the interpolation direction and step size. Note that unlike previous work [SYTZ20, HHLP20], which discovered semantically significant linear directions applicable to all the data points in the latent space of GAN, the present method and / or technique find unique directions for each data point: when z changes, changes simultaneously due to the non-linear nature of equation 1, and a new direction is calculated for the updated e at each optimization step.

[0062] UNTANGLING OF ATTRIBUTES DURING HANDLING

[0063] There exists a method and / or technique, according to one embodiment, for minimizing semantic attribute entanglement. Attribute entanglement can (e.g., sometimes) occur during latent code interpolation along the found directions described above. As used here, and as a person skilled in the art will understand, two or more semantic attributes are entangled when, during interpolation, the manipulation of one semantic attribute affects the other or other of the two or more attributes. It is observed that interpolation along the original directions can alter non-target semantics, but such effects can occasionally be eliminated by randomly excluding dimensions in the direction vector used for interpolation.Therefore, it is assumed that in the discovered ^dimensional directional vector, only certain dimensions are responsible for modifying the target semantic attribute, while the other dimensions indicate a bias learned from the training data. For example, in the direction that increases a person's age, glasses appear during the manipulation, because glasses and age are correlated in the data.

[0064] Grad-CAM [SCD+17] is a class discrimination localization technique that provides visual explanations for CNN-based models using gradient information. According to a method and / or technique described herein, dimensions are filtered based on the gradient amplitudes of the semantic notation function. This results in more disentangled controls. Grad-CAM notably measures the importance of neurons by:

[0065] nC_ y àl,(4)

[0066] where the gradient of the note yc of class L with respect to the activation map A in all width and height dimensions is globalized on average. The final density map is obtained by calculating: [°°671 = ReLU ( 5 )

[0068] According to one embodiment, is considered as the only activation map, and the importance of the i-th dimension is calculated by

[0069] = |nd (6)

[0070] By definition of gradients, the value of n% in its i-th dimension indicates the rate of change of ci induced by a small change in '■ in the same dimension. Intuitively, dimensions in nk with a larger value have more influence on 4, and those with smaller values ​​are less relevant. However, such irrelevant dimensions can have high gradient amplitudes when calculating the control for another semantic attribute n™, m*k. Therefore, even a small amount of change in these dimensions during optimization for the A-th semantic attribute could potentially affect CP, leading to an attribute entanglement problem. Consequently, according to one embodiment, the gradient dimensions for any A-th semantic attribute with a value not less than a certain threshold are considered important, while any dimension important in the prediction of the semantic attribute that appears to be coupled to the target from the target direction is excluded.According to one embodiment, even dimensions important for k can be hidden if k is entangled with m and they share important dimensions.

[0071] Fig. 3 describes the 300 operations, in pseudocode form, for computing the new direction with disentangled attributes according to one embodiment. In line 1, it is noted that the target semantics is entangled with another in the direction il1!. In line 2, it is noted that the notation function / , for the entangled semantics m is and a threshold of ' is established to identify significant gradients (i.e., large gradient amplitudes). In line 3, the gradient ( / 4) of the entangled semantics m is determined from its respective function. In line 4, each of the ? dimensional elements [J" is examined with respect to \ where the elements L”' = I ff'11 . . In line 5, where the gradients are identified by the threshold, the corresponding gradient amplitudes in the dimensions for k are set to 0: [i] = 0 for i EE, , and a final form of is returned (e.g. provided) on line 6.

[0072] At each step, to increase the semantic score 4 once with the same semantic unraveled, is updated with:

[0073] 4 = < + « / 4(7)

[0074] with ^ calculated following operations 300 of the pseudocode algorithm of Fig. 3. It will be understood that the operations can loop and repeat themselves in such a way that z is calculated several times to continue the interpolation along the direction of the target semantic attribute, for example, until an image (e.g., gÇz) 104B) exhibiting the desired characteristics is produced.

[0075] DETAILS OF THE IMPROVEMENT METHOD

[0076] According to one embodiment, the GAN part of the network architecture structure 102 (e.g., GAN S 108) adapts the StyleGAN2 structure [KLA+20], for example, as pre-formed on the FFHQ dataset [KLA18]. Optimization is performed in the VF space of StyleGAN2 [KLA+20]. 400 images were generated with StyleGAN2, from which 30 images were manually selected as positive / negative samples for each target semantic, and 10% of the selected pairs were used for evaluation. Several binary classifiers (e.g., 110 respective cases) were trained for different target semantics simultaneously so as to learn multiple labels, minimizing the sum of all binary cross-entropy losses. According to one embodiment, each classifier (e.g., comprises two fully connected layers, with a hidden layer size of 16, a ReLU activation, and a Sigmoid activation at its output neuron). For the filtering threshold, it was determined, e.g., experimentally, that the 100th largest gradient amplitude of nV channels works well in practice.For the value a, 0.4 was used in all experiments.

[0077] According to one embodiment, to remove the dimensions of the gradient vector that are important for the semantic attribute m, and therefore m is entangled with k, the gradient vector Hz for m is determined. Each dimension of this vector is evaluated (e.g., a sorting process can be performed) to determine the 100th largest value, using the absolute value for the dimension. This 100th largest value becomes the threshold with respect to m during filtering. For each i-th dimension in the vector of m which is greater than or equal to the 100th highest value, this corresponding f-th dimension is set to 0 in the vector of k. For each semantic attribute m entangled with k, the gradient vector of m is determined and evaluated forf in the same way, then the respective dimensions are masked in the gradient vector of k.

[0078] RESULTS AND DISCUSSIONS

[0079] The evaluation results are presented in terms of methods, systems, and techniques (e.g., on StyleGAN2 [KLA+20] pre-trained on the FFHQ dataset [KLA18] of network architecture 102, in accordance with the embodiments as described). Figure 4 illustrates an array of images showing the results of 400 manipulations for 5 attributes: age, pose, sex, glasses, and smile. These qualitative examples are edited with learned instructions trained on 60 samples per attribute. Figure 5 illustrates an array of 500 images showing the attribution untangling. Figures 6 and 7 illustrate respective arrays of 600 and 700 images showing comparative results with other work [SYTZ20, WLS21, HHLP20].

[0080] Table 400 presents the manipulation results on 5 different attributes, i.e., single-attribute manipulation results. For each group of three samples from a row (e.g., 402), the middle image is the original image synthesized by StyleGAN2, and the left / right images correspond to the result of the synthesis based on the original latent code edited along the negative / positive direction found in accordance with the method or technique disclosed in this document (i.e. as output of network architecture 102 (via GAN 8 108)).

[0081] The approach performs well in both the negative and positive directions for all attributes. In particular, for the age attribute, it has been observed that not only is the approach capable of creating / eliminating subtle aging effects such as wrinkles or acne, but it is also capable of modifying the shape of the face to a high degree, while still preserving the person's identity. The approach is also capable of adding glasses while keeping irrelevant semantics virtually unchanged, meaning that many realistic image samples can be created with glasses, despite the absence (i.e., lack) of such a semantic attribute in the original FFHQ training data.

[0082] Table 500 shows the attribute untangling results in Fig. 5. For each group of three images in each row (e.g., 502, 504, 506, and 508), the first image on the left (e.g., 502A, 502D, 504A, and 504D) is the original image synthesized by StyleGAN2. The adjacent left and right images (502B / 502C, 502E / 502F, 504B / 504C, and 504E / 504F) correspond to the synthesis result (via StyleGAN2) based on the original latent code edited along the negative / positive directions found by the method or technique described herein (e.g., using smaller and larger interpolation distances). The first two rows, 502 and 504, illustrate that the aging direction is entangled with the glasses direction. By excluding the dimensions important for the glasses classification from the aging direction during optimization, the two attributes can be disentangled (e.g.row 504) while obtaining similar aging results. Similarly, for the last two rows 506 and 508, the approach excluding important dimensions as disclosed here successfully finds new directions where the direction of sex is less entangled with the smile.

[0083] It has been observed that, although the controls found by the method or technique disclosed herein are generally independent, there are always cases of failure where manipulating one semantic attribute affects another, and the most common entanglements are age with glasses and gender with a smile. The results indicate that by excluding the important dimensions responsible for modifying the logits of the entangled attribute during optimization, a slightly different direction could be found where the irrelevant semantic attribute is less affected but where the target semantic attribute is still successfully manipulated.

[0084] COMPARISONS

[0085] The results of the methods and / or techniques taught in this application can be compared to InterFaceGAN [SYTZ20] and GANSpace [HHLP20], by For example, three semantic attributes were used: smile, sex, and age. Attributes not supported by some of the previously known methods and implementations are not shown. For [SYTZ20], the SVM boundaries used the same training data as that used to train the gan g and the auxiliary networks in [Fig. 1]. For [HHLP20], the officially disclosed instructions were used. Table 600 shows in [Fig. 6] the directions found by each of the three implementations, where the first two are previously known, and the bottom row illustrates the results according to a present embodiment (e.g., using network architecture 102 with the GAN g 108 and the auxiliary networks 110).For each group of three samples in a row, the middle image is the original image synthesized by StyleGAN2, the left and right images correspond to the synthesis result based on the original latent code edited with the directions found by each method to decrease / increase the target semantic scores.

[0086] Overall, the method disclosed in this document outperforms the unsupervised GANSpace method, which generates less realistic images with a non-obvious change in target semantics, and finds directions similar to In-terFaceGAN. Nevertheless, it is observed that the method disclosed here is better at disentangling attributes with channel filtering inspired by Grad-CAM, compared to the conditional manipulation technique proposed by [SYTZ20], which adjusts the target direction by subtracting its projected vector from the direction to edit the tangled semantics, and some examples are shown in [Fig.7].

[0087] Table 700 ([Fig. 7]) illustrates a comparison of attribute disentanglement results for the method disclosed herein and a single prior known method – InterFaceGAN [SYTZ20]. The method disclosed here disentangles attributes better during manipulation. In the gender-smile entanglement example of the first two rows, the smile attribute is better preserved when editing gender in the second of these rows (the present disclosed method). In the age-glasses entanglement example of the bottom two rows, the present disclosed method finds an disentangled control that better preserves irrelevant semantics, e.g., the person's facial expression and hairstyle.

[0088] An example of GAN use is illustrated and described in the Applicant's US patent application number 16 / 683,398, filed on November 14, 2019 and entitled "System and Method for Augmented Reality by translate an image using Conditional cycle-consistent Generative Adversarial Networks (ccGans)", which is incorporated herein by reference.

[0089] According to one embodiment, the disclosed technologies and methodologies include developer-related methods and systems for defining (by conditioning, for example) a CNN architecture comprising a GAN and classes Auxiliary f-factors for classifying semantic attributes. The GAN can be controlled to interpolate the semantic attributes as they have been trained. User-related methods and systems are also illustrated, such as where the trained generator model (e.g., gCO generator) is used at runtime to process a source image A for image-to-image translation to obtain a synthesized image x = g(x) having (or not) a particular semantic attribute(s).

[0090] Figure 8 is a functional diagram of a GUI 800, according to one embodiment. The GUI 800 can be presented via a display device (not shown). According to one embodiment, a user can specify the target semantic attribute(s) and the desired quantity of the attribute. For example, +smile, -age can identify the attributes, and a quantity such as a percentage can specify an amount to be displayed in the synthesized image when the generator is called. Since the interpolation process is continuous, pre-trained image classifiers can be used to quantify the quantity of the semantic attribute (or the amount of change) and find a desired manipulation strength.

[0091] The GUI 800 illustrates a plurality of 802 input controls provided to receive selective input for each target semantic attribute for which the CNN 102 architecture has been configured and trained. According to this embodiment, each input control (e.g., 802A) illustrates a cursor-type control for a respective individual attribute and to identify granular input. Other types of input controls (radial controls, text boxes, buttons, etc.) can be used for granular or other input. The controls can receive a percentage value, a range selection, a scalar value, or other values. For example, an age attribute can be associated with a range of integers and accept granular inputs approximating age in years or decades, etc. Others can select relative ranges for the quantity of the attribute present—small, medium, large—for example.These relative ranges can be associated with respective granular values ​​such as 15%, 50% and 85%, or with other values.

[0092] By selecting an "Apply" control (804), the semantic attribute input(s) are applied when the generator is called. The generator output is controlled by semantic attribute controls derived from the respective auxiliary network semantic attribute classifiers. The semantic attribute input(s) is / are applied via these attribute controls to a source image x (806) to generate a synthesized image x' (808) (e.g., an output image). The synthesized image can be represented by x' = g(x), as controlled by the attribute controls. And the attribute controls react to the semantic attribute input from 802 interface controls. Although all available semantic attributes have a control, a user can choose not to modify an attribute and, therefore, the generator does not need to interpolate along a direction identified in an associated semantic control.

[0093] A source image 806 can be identified for use (e.g. downloaded, copied from a storage device, obtained via a camera input to obtain a selfie, etc.) via a "source image" control 810. The synthesized x image obtained 808 can be saved via a "save image" control 812.

[0094] In one embodiment, before applying the controls when the generator is called, the attributes are disentangled so that the synthesized image is generated with minimized entanglement. In one embodiment, detangling can be enabled or disabled via one or more controls (not shown). For example, in one embodiment, age and glasses can be selectively detangled, or smile and gender can be selectively detangled, or both of these entanglements can be detangled.

[0095] Although individual semantic attribute controls are provided for each attribute formed in the illustrated embodiment, according to one (not illustrated) embodiment, fewer controls (e.g., for a single attribute or two attributes, etc.) are provided. Although separate controls are provided for individual attributes, according to one (not illustrated) embodiment, a single control may be provided for combined attributes (e.g., age and gender). A multi-attribute control is computed by vector computation. To create a face with less of a smile, more glasses, and an older appearance (higher age), the GUI is configured to receive inputs for the respective directions, namely -smile, +glasses, and +age. According to one embodiment of computer device operations, the inputs are associated with respective gradient vectors (of associated semantic attribute controls), and the vectors are summed and normalized.The generator interpolates (linearly) along the calculated direction (i.e., combined directions) to produce the output image.

[0096] In one embodiment, the GUI is provided by a first computing device and the generator is provided by another computing device located remotely from the first computing device. The other computing device may provide the controllable GAN generator as a service. The first computing device providing the GUI is configured to communicate the input image and semantic attribute direction inputs (e.g., as percentage values ​​or other inputs) to the remotely located computing device providing (e.g., running) the generator. Such a remotely located computing device may provide an application programming interface (API) or other interface for receiving the source image (or a selection) and the semantic attribute direction inputs. Mantic. The remote computing device can calculate the direction vector and call the generator by applying semantic attribute checks. In another embodiment, the computing device providing the GUI can be the same as the computing device providing the generator.

[0097] In one embodiment, in addition to the developer aspects (e.g., used during training) and the target computing device aspects (used during inference), a person skilled in the art will understand that aspects of computer program products are disclosed, whereby instructions are stored in a non-transient storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.), which, when executed, cause a computing device to perform any one of the method aspects disclosed herein. In one embodiment, a computing device comprises a processor (e.g., a microprocessor (e.g., a CPU, a GPU, a plurality thereof), a microcontroller, etc.) that executes computer-readable instructions such as those stored in the storage device.According to one embodiment, the computer device includes circuits (for example, "designed for this purpose") which execute the functions of the instructions, for example, without needing to read such instructions.

[0098] Aspects related to the e-commerce system are also presented and described. In one embodiment, a user's computer device is configured as a client computer device with respect to the e-commerce system. The e-commerce system, for example, stores a computer program for such a client computer device. Thus, the e-commerce system includes, as a component thereof, a computer program product that stores the instructions which, when executed by the client computer device (e.g., a processing unit thereof), configure such a client computer device. These and other aspects will be self-evident.

[0099] Fig. 9 is a functional diagram of a computer system 900. In one embodiment, the computer system 900 comprises a plurality of computing devices which, in another embodiment, include servers, development computers (PCs, laptops, etc.) and mobile devices such as smartphones and tablets, etc. A network model training environment 902 is shown, comprising hardware and software for defining and configuring, for example via conditioning, a CNN architecture based on GANs 102. The architecture 102 includes a GAN-based generator 108 (e.g., generator S) and an auxiliary network of classifiers 110 (e.g., a plurality of mapping functions / ). In one embodiment, the GAN and mapping functions are co-conditioned, and the latent space '■ for the GAN is updated on the basis of results of the semantic functions as indicated.

[0100] According to one embodiment, the generator 108 and the auxiliary network 110, collectively CNN 102, once trained, are provided for use on a target device such as one of the mobile devices 910, 912 of the system 900 or other devices such as 913. The mobile devices 910 are representative user computing devices, for example, consumer users. It is understood that these users may use other forms of computing devices such as a desktop computer, a workstation, etc. Device 913 represents a computing device for pre-training the generation of training data (hence, a training data generation device).According to this embodiment, such a computing device uses the generator 108 to generate additional image data that can be easily labeled with semantic attributes and used to train a network model (e.g., under surveillance). The form factor of the device 913 can be a server, a laptop, a desktop computer, etc., and does not need to be a consumer-type mobile device such as a tablet, smartphone, etc.

[0101] In one embodiment, the network model training environment 902 employs, at least in part, a GAN model (generator 108) that is pre-trained for the image task (e.g., face image generation). The generator 108 is pre-trained, for example, using an image dataset 914 stored on a data server 916. In one embodiment, the generator 108 is a custom-developed model. In another embodiment, the generator 108 is publicly available, for example, via an open-source license. The dataset can be developed and made available in the same way. Depending on the type of image task and the network architecture (e.g., supervised), the training is supervised and the dataset is annotated according to this training. In other scenarios, the training is not supervised, and the data is defined accordingly.According to one embodiment, the GAN model (generator 108) is further conditioned in its form as illustrated in [Fig IA], using, for example, labeled facial images for semantic attributes.

[0102] According to one embodiment, the generator 108 and the auxiliary network 110 are incorporated into an augmented reality (AR) application 920. Although not shown, according to one embodiment, the application is developed using an application developer computing device for specific target devices having specific hardware and software, particularly an operating system configuration. According to one embodiment, the AR application 920 is an origin application configured to run in a specific origin environment such as an environment defined for a particular operating system (and / or hardware). According to one embodiment, the RA 920 application takes the form of a browser-based application, for example, configured to run in a browser environment of the target device.

[0103] According to one embodiment, the RA 920 application is distributed (e.g. downloaded) by user devices such as mobile devices 910 and 912. Native applications are often distributed via an application distribution server 922 (e.g. a "Bank" operated by a third-party service), although this is not required.

[0104] According to one embodiment (not shown), the AR 920 application does not include the CNN 102 architecture itself (does not include the generator and auxiliary network). Instead, the application is configured with an interface to communicate with a remote device providing these components as a service (not shown), for example, as a cloud-based service. Storing and running the generator and auxiliary network is resource-intensive and may be too large / demanding for some computing devices. Other reasons may also be considered in the AR application paradigm.

[0105] In one embodiment, the AR 920 application is configured to provide an augmented reality experience (e.g., via an interface) to a user. For example, an effect is applied to an image via processing by the generator 108. The mobile device includes a camera (not shown) for capturing an image (e.g., a still image or video, whether or not it is a selfie). In one embodiment, the effect is applied to the image, for example, to the video image, in real time (and displayed on a display device of the mobile device) to simulate an effect on a user during video capture. When the camera position is changed, the effect is applied in response to the video frame(s) as captured to simulate augmented reality. As will be seen, real-time operation is limited by processing resources.In one embodiment, an effect is not simulated in real time but is delayed, which can impact the augmented reality experience.

[0106] According to one embodiment, the computer devices are coupled for communication via one or more networks (e.g. 922), which include wireless or other networks, public or other networks, etc.

[0107] By way of example, but not limited to, a 924 e-commerce system is web-based and provides a browser-based RA 920A application as a component of an e-commerce service provided by the 924 e-commerce system. The 924 e-commerce system comprises a configured computer device and a datastore 8926 (e.g., a database or other configuration). The datastore 926 stores data on products, services, and information Related training (e.g., techniques for applying a product). The 926 database or other data storage devices (not shown) stores recommendation rules or other forms of product and / or service recommendations, etc., to help a user choose from available products and services. The products and services are presented via a user interface displayed on the user's computer (mobile) device. The simplification of the 924 e-commerce system will be evident.

[0108] According to one embodiment, the browser-based AR 920A application (or AR 920 application) provides an augmented reality customer experience such as the simulation of a product, technique, or service provided or facilitated by the 924 e-commerce system. It will be understood that the AR 920 application, according to the embodiment, is also configured to provide e-commerce services such as via a connection to the 924 e-commerce service.

[0109] By way of example, but not limited to, a product includes a cosmetic product (e.g., makeup), an anti-aging or rejuvenating product, and a service includes a cosmetic, anti-aging, or rejuvenating service. A service includes a treatment or other procedure. The product or service relates to a part of the human body such as the face, hair, or nails. In one embodiment, a computing device (such as the 912 mobile device) thus configured provides a 912A facial effect unit comprising processing circuits configured to apply at least one facial effect to the source image and to generate one or more virtual instances of a source image with the effect applied (e.g., displayed) on an e-commerce interface of the computing device, which is facilitated by the e-commerce system.In one embodiment, the facial effect unit 912A uses the generative adversarial network (GAN)-based generator (&) and the auxiliary network 110 as described herein to produce the source image with applied effect. In one embodiment, the computing device provides a user experience unit 912B comprising processing circuits for determining at least one product or service from the database 926 and for generating one or more virtual cases of a recommendation on an e-commerce interface for purchasing products or services. In one embodiment, the at least one product is associated with a respective facial effect, and the facial effect unit applies the respective facial effect to provide a virtual trial experience.

[0110] In one embodiment, the user experience unit 912B is configured to present a graphical user interface (e.g., browser-based or otherwise) for interaction with the computing device 912 and the e-commerce system 924. In one embodiment, the e-commerce system 924 is thus configured to provide the AR application for execution by client computing devices. such as a mobile device (e.g. 912) and is cooperatively configured to provide e-commerce services to the client computing device in order to facilitate recommendations (product / service) for AR simulation via the client computing device (e.g. 8912) and facilitate purchases.

[0111] Thus, any computing device, but especially mobile devices, provides a computing device for translating an image from a first domain space to a second domain space. The computing device includes a storage unit storing a generator based on a generative adversarial network (GAN) (g), configured to generate images controlled for semantic attributes. In one embodiment, the computing device includes a processing unit configured to (e.g., via the RA 920 application): receive the source image; receive an input to identify at least one semantic attribute to be controlled, for example, including input to refine the semantic attribute (e.g., a percentage); provide the image to the generator 8 to obtain a synthesized (e.g., new) image in response to the semantic attribute input; and provide the new image for presentation.

[0112] According to one embodiment, the generator is configured to synthesize a particular semantic attribute that includes one of: a facial feature including age, sex, smile or other facial feature; a pose effect; a makeup effect; a hairstyle effect; a nail effect; a cosmetic or dental surgery effect including one of rhinoplasty, facelift, blepharoplasty, implant, otoplasty, teeth whitening, teeth straightening or other cosmetic or dental effect; and a device effect including one of ocular device, oral device, hearing device or other device effect.

[0113] In one embodiment, the synthesized output of the generator 8 (e.g., the synthesized image or the new image) is provided to a second network model, for example, for a face editing task. The second network model may consist of a second generator or feature detector and a simulator for applying an effect (e.g., producing another new image). The effect may be presented in an AR interface. In one embodiment, the processing unit is configured to provide the synthesized image (or another new image defined from it) in an augmented reality interface to simulate an effect applied to the image. In one embodiment, the effect includes any one of a makeup effect, a hairstyle effect, a nail effect, a cosmetic or dental surgery effect, a device effect, or another applied simulation effect.According to one embodiment, the source image includes an applicable part (e.g. face, hair, nails or body part) of a subject such as a user of the . device.

[0114] The 902 network model training environment provides a computing device configured to execute a method, such as a configuration method, by conditioning a GAN-based generator and an auxiliary network of classifiers that share the same latent space. It will be understood that embodiments of the computing device aspect of the 902 network model training environment and all associated embodiments, for example, the generator or model, apply to the training method aspect, with appropriate adaptation.

[0115] Fig. 10 is a flowchart of operations 1000 according to a present embodiment. The operations provide a method such as training (e.g., by conditioning) in one embodiment. In step 1002, the operations configure by conditioning a generator 8 based on a generative adversarial network (GAN) 108 with an auxiliary training network 110. The training updates the latent space Z. According to one embodiment, the generator 8 108 with the auxiliary training network 110 are trained with labeled images having labels of respective semantic attributes. The co-training forms the auxiliary network 110 using the latent space Z, and the respective gradients of the auxiliary network 110 (e.g., where each semantic attribute has a corresponding binary classifier) ​​are useful for controlling the operation of the GAN-based generator g during image generation.

[0116] In step 1004, the operations provide the generator 8 108 and the auxiliary network 110 for use by a computing device to generate an image. A product aspect of the computer program and related computing device will be evident, as well as other aspects.

[0117] Figure 11 is a flowchart of operations 1100 according to this embodiment. The operations 1100 are carried out by a computer device such as 910, 912, 913 or 924.

[0118] In 1102, the operations provide a generator and an auxiliary network sharing a latent space. The generator is configured to generate synthesized images exhibiting human-interpretable semantic attributes. The auxiliary network includes a plurality of semantic attribute classifiers, notably one semantic attribute classifier for each semantic attribute to be controlled (by the generator) when generating a synthesized image from a source image. Each semantic attribute classifier is configured to classify the presence of one (associated) of the semantic attributes in the images and provide a meaningful direction for controlling one of the semantic attributes in the generator's synthesized images.

[0119] When requesting the generator to generate the synthesized image from the image In the source, operations in 1104 apply a respective semantic attribute check to control a respective semantic attribute in a synthesized image generated from a source image. The respective semantic attribute check responds to the meaningful direction provided by the classifier associated with the semantic attribute.

[0120] The practical application of Generator 8 includes generating and providing (large) quantities of more accessible labeled synthetic data to train (or further train) one or more network models. For example, such data can be useful for improving the results of a face-editing model. For instance, in a hair segmentation task, a segmentation model might initially be trained on unbalanced data with unevenly distributed hair colors, leading to unsatisfactory segmentation results for a minority hair color group. To address this issue and create a more balanced dataset, Generator 8 and / or the techniques presented herein can be used to edit hair colors from existing labeled data and generate abundant data for the minority group.Thus, the use of a 108 generator (and a 110 network) includes the generation of labeled synthetic images for training and modeling, and may also include training the model with the labeled synthetic images.

[0121] The practical implementation may include all or part of the features described in this document. These aspects, features, and various combinations thereof, as well as others, may be expressed in the form of methods, devices, systems, means of performing functions, program products, and other ways, combining the features described in this document. A number of embodiments have been described. However, it shall be understood that various modifications may be made without deviating from the spirit and scope of the processes and techniques described in this document. Furthermore, additional steps may be provided, or steps may be omitted, from the described process, and other components may be added to or removed from the described systems. Therefore, other embodiments fall within the scope of the following claims.

[0122] Throughout the description and claims of this specification, the terms "include" and "contain" and their variations mean "including but not limited to" and are not intended to exclude (and do not exclude) other components, integers, or steps. Throughout this specification, the singular encompasses the plural unless the context requires otherwise. In particular, where the indefinite article is used, the specification is to be understood as considering plurality as well as singularity, unless the context requires otherwise.

[0123] The features, integers, attributes, or groups described in conjunction with a particular aspect, embodiment, or example of the invention shall be understood as applicable to any other aspect, embodiment, or example, unless they are inconsistent with them. All features disclosed herein (including the claims, abstract, and accompanying drawings), and / or all steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of these features and / or steps are mutually exclusive. The invention is not limited to the details of the preceding examples or embodiments.The invention extends to any new feature, or any new combination, of the features disclosed in this specification (including any claim, abstract, and attached drawing) or to any new feature, or any new combination, of the steps of any disclosed method or process. List of cited documents.

[0124] [ACB17] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein ge- nerative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.

[0125] [BDS18] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.

[0126] [CCK+18] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, et Jaegul Choo. Stargan : Unified generative adversarial networks for multi-domain image-to-image translation. Dans Proceedings of the IEEE conférence on computer vision and pattern récognition, pages 8789-8797, 2018.

[0127] [CVB21] Anton Cherepkov, Andrey Voynov, et Artem Babenko. Navigating the gan parameter space for semantic image editing. Dans Proceedings ofthe 1EEE / CVF Conférence on Computer Vision and Pattern Récognition, pages 3671-3680, 2021.

[0128] [GPAM+ 14] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, et Yoshua Bengio. Generative adversarial nets. Advances in neural information processing Systems, 27, 2014.

[0129] [HHLP20] Erik Hârkônen, Aaron Hertzmann, Jaakko Lehtinen, et Sylvain Paris. Ganspace : Discovering interprétable gan Controls. arXivpreprint arXiv:2004.02546, 2020.

[0130] [HRU+17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, et Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing Systems, 30, 2017.

[0131] [KLA18] Tero Karras, Samuli Laine, et Timo Aila. A style-based generator ar chitecture for generative adversarial networks. arXiv preprint arXiv:1812.04948, 2018.

[0132] [KLA+20] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, et Timo Aila. Analyzing and improving the image quality of stylegan. Dans Proceedings ofthe IEEE / CVF Conférence on Computer Vision and Pattern Récognition, pages 8110-8119, 2020.

[0133] [LLWT15] Ziwei Lin, Ping Luo, Xiaogang Wang et Xiaoou Tang. Deep learning face attributes in the wild. Dans Proceedings ofthe IEEE international conférence on computer vision, pages 3730-3738, 2015.

[0134] [PBH20] Antoine Plumerault, Hervé Le Borgne, et Céline Hudelot. Controlling ge- nerative models with continuons factors of variations. arXiv preprint arXiv:2001.10238, 2020.

[0135] [SCD+17] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, et Dhruv Batra. Grad-cam : Visual explanations from deep networks via gradient-based localization. Dans Proceedings ofthe IEEE international conférence on computer vision, pages 618-626, 2017.

[0136] [SYTZ20] Yujun Shen, Ceyuan Yang, Xiaoou Tang, et Bolei Zhou. Interfacegan : Interpreting the disentangled face représentation leamed by gans. IEEE transactions on pattern analysis and machine intelligence, 2020.

[0137] [SZ21] Yujun Shen et Bolei Zhou. Closed-form factorization of latent semantics in gans. Dans Proceedings ofthe IEEE / CVF Conférence on Computer Vision and Pattern Récognition, pages 1532-1540, 2021.

[0138] [VB20] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In International Conference on Machine Learning, pages 9786-9796. PMLR, 2020.

[0139] [WDB+19] Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimization for generative adversarial networks. arXiv preprint arXiv:1912.00953, 2019.

[0140] [WLS21] Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Di Sentangled Controls for stylegan image generation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 12863-12872, 2021.

[0141] [ZKL+ 16] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.

Claims

Claims

1. A computing device comprising: a processor and a storage device, the storage device containing instructions that, when executed by the processor, cause the computing device to: provide a generator and an auxiliary network sharing a latent space, the generator configured to generate synthesized images having human-interpretable semantic attributes and the auxiliary network comprising a plurality of semantic attribute classifiers, including a semantic attribute classifier for each semantic attribute to be controlled when generating a synthesized image from a source image, each semantic attribute classifier configured to classify a presence of one of the semantic attributes in images and provide meaningful direction for controlling the one of the semantic attributes in the synthesized images of the generator;and upon requesting the generator to generate the synthesized image from the source image, applying a respective semantic attribute control to control a respective semantic attribute in the synthesized image, the respective semantic attribute control responding to the meaningful direction provided by the classifier associated with the semantic attribute.;

2. The computing device of claim 1, wherein the significant direction comprises a gradient direction, and the instructions cause the computing device to calculate a respective semantic attribute control from a respective gradient direction of the classifier associated with the semantic attribute.

3. The computing device of claim 2, wherein to control two or more semantic attributes, the instructions cause the computing device to combine each respective gradient direction of each classifier associated with the two or more semantic attributes to be controlled.

4. A computing device according to claim 1 or 2, wherein the instructions cause the computing device to unravel the respective semantic attribute control before application to the generator.

5. The computing device of claim 4, wherein: each respective semantic attribute control comprises a respective gradient direction vector calculated from parameters of the classi- respective semantic attribute control associated with the respective semantic attribute control; and disentangling a respective semantic attribute control from another respective semantic attribute control comprises removing significant data dimensions of the gradient direction vector associated with the respective semantic attribute control from the gradient direction vector associated with the respective semantic attribute control.

6. The computing device of claim 5, wherein the deleted significant data dimensions meet a threshold identifying those data dimensions having an absolute value greater than or equal to the threshold.

7. The computing device of claim 6, wherein, if the z-th data dimension for the gradient direction vector associated with the respective semantic attribute control is identified by the threshold, the corresponding i-th data dimension for the gradient direction vector associated with the respective other semantic attribute control is reset to zero.

8. A computing device according to any one of claims 1 to 7, wherein the instructions cause the computing device to receive an input identifying at least one of the respective semantic attributes to be monitored with respect to the source image.

9. The computing device of claim 8, wherein the input identifying at least one of the respective semantic attributes comprises a granular input for determining an amount of semantic attribute to apply when generating the synthesized image.

10. The computing device of any one of claims 1 to 9, wherein the instructions cause the computing device to perform any one or more of: providing the generator and the auxiliary network for generating the synthesized images as a service; providing an e-commerce interface for purchasing a product or service; providing a recommendation interface for recommending a product or service; and providing an augmented reality interface using the synthesized image to provide an augmented reality experience.

Images