Systems and methods for augmented reality using conditional cycle consistency to generate image-to-image translation models

Abstract

Systems and methods relate to a network model that applies effects to images, such as augmented reality effects (e.g., makeup, hair, nails, etc.). The network model uses conditional cycle consistency to generate an image-to-image transformation model, transforming the image from a first domain space where the effect is not applied to a second continuous domain space where the effect is applied. To render arbitrary effects not seen at training time (e.g., lipstick), the spatial representation of the effect is a continuous domain (e.g., a conditional variable vector) learned by encoding simple sample images of the effect, such as those available as product samples, and no effect. The model is trained end-to-end in an unsupervised manner. To condition the generator of the model, convolutional conditional batch normalization (CCBN) is used to apply a vector that encodes a reference sample image representing makeup characteristics.

Description

[0001] Cross-references

[0002] This application claims domestic benefit and / or priority to U.S. Provisional Application No. 62 / 767,769, filed on November 15, 2018, the entire contents of which are incorporated herein by reference where permitted. Technical Field

[0003] The present disclosure relates to image processing, and more particularly, to creating augmented reality through image processing, using conditional cycle consistency to generate an image-to-image conversion model (e.g., a neural network model) to convert images. Background Art

[0004] Real-time virtual makeup applications provide users with an interactive experience in augmented reality with high identity retention and detailed realism. Ideally, users like to preview makeup products (e.g., lipstick or eye shadow) without actually trying them on. In addition, virtual try-ons give users the freedom to adjust the relevant attributes of the product (e.g., color and gloss) according to their personal preferences.

[0005] Most of these applications rely on classical machine learning or computer vision techniques, which involve two steps: detecting facial landmarks using a cascade of regression forests or other methods; and rendering the output image using traditional graphics techniques and the detected landmarks. This two-step solution is fairly fast and produces credible results in most cases, but has several limitations. Notably: (i) the shape of the rendered makeup is limited by the accuracy of the detected landmarks; (ii) the color and sheen of the virtual makeup need to be manually designed based on estimates of the characteristics of the actual product, lighting direction and intensity, etc.; and (iii) the combination between the virtual makeup and the original image needs to be carefully adjusted according to the individual's skin tone.

[0006] Due to these and other limitations, this solution cannot easily scale to a large number of products while maintaining realism. To support new products, these parameters must be adjusted until they produce renderings that look sufficiently like the actual product. This process is time-consuming and inaccurate, and therefore leads to poor scaling as the size of the product catalog increases.

[0007] There has been work done to explore and learn the facial feature space in a purely unsupervised manner. However, the task of makeup application is more stringent in the sense that only the relevant regions should be modified while other parts or attributes of the face should remain as identical as possible (identity preservation problem).

[0008] Image-to-image conversion

[0009] Manipulating specific facial attributes can be viewed as an image-to-image translation problem. [1] (references are listed below and incorporated into this paper by citation where permitted) and CycleGAN [2] GANs are used to solve this dual-domain problem. The former balances paired image datasets, while the latter can learn a transformation model from two unpaired datasets due to cycle consistency loss, which reduces the number of possible mapping functions in the learning space. A significant disadvantage of CycleGAN is that it cannot transform between more than two domains. Based on the architecture of CycleGAN, PairedCycleGAN is more efficient when transforming no-makeup photos into the makeup style of reference photos and transforming reference photos into no-makeup photos. [3] The makeup generation problem is specifically addressed by optimizing the cycle consistency loss. The generator for makeup application takes as input 1) a source no-makeup image and 2) a reference image with target makeup applied, and outputs the source image with makeup applied while maintaining its identity. The main difference between the approach in PairedCycleGAN and CycleGAN is that the two generators corresponding to makeup application and makeup removal are asymmetric functions, where one conveys style while the other removes it. While this work clearly demonstrates that GANs with cycle consistency loss are capable of learning complex lipstick features and accurately rendering them, the need for reference images is problematic for practical use cases.

[0010] If the model will be used as a virtual try-on feature in an online store, reference images will be required for each product. Furthermore, aspects such as lighting and identity in the reference images will affect the lipstick reproduction in subtle ways, limiting the model's ability to encode the true lipstick properties rather than those displayed in a single instance.

[0011] Conditional Image Generation

[0012] Recent developments in generative models address the problem of modeling real-world high-dimensional data distributions. The two main streams of generative models are generative adversarial networks and [4] and log-likelihood-based methods (e.g., autoregressive models [5] , Variational Autoencoder [6] and reversible generation flow [7] ). These models can depend on a sub-distribution of the real data fed into the labels during training. The details of how to implement this conditional probability model vary in recent literature.

[0013] Conditional GAN ​​(cGAN) [8] Make the generator distributed and give the conditioning variable vector The difference between subsets of the target data distribution is minimized. Mirza et al.[9] and Reed et al.

[10] Attempts have been made to concatenate the conditional vector to the input of the discriminator or to the feature maps between intermediate convolutional layers.

[11] There is also exploration of using label information to supervise additional classifiers built on top of the feature vectors in the discriminator. Miyato et al. [8] Try using the output projection method, which solves the inner product between the conditional vector in the discriminator and the feature vector.

[0014] In general, for cGAN, the discriminator and generator need to be tuned to achieve the best performance according to recent literature and experiments. Miyato et al. [8] (incorporated herein in its entirety by reference) draws on Perez et al.

[12] Based on the idea of ​​​​the CNN, the conditional batch normalization (CBN) layer is used to regulate the generator of cGAN. Summary of the invention

[0015] Therefore, it would be ideal to have an end-to-end model that can learn the conditional makeup space itself. That is, by training on images and / or videos of real-world individuals wearing makeup, the model can learn how to preserve the identity of the individual while at the same time modifying the image to produce a realistic rendering of the makeup.

[0016] A conditional cycle-consistent generative image-to-image translation model is disclosed, e.g., a modified CycleGAN architecture with added features such as cGAN features. The added features are employed to minimize the difference between a generator distribution and a subset of a target data distribution that gives a vector of conditioning variables encoding reference samples representing augmented reality image properties (e.g., makeup properties). More specifically, to condition the generator, a vector encoding a reference sample image representing properties of an effect (e.g., makeup) is applied using convolutional conditional batch normalization (CCBN).

[0017] A computing device is provided for converting an image from a first domain space to a second continuous domain space according to an effect to be applied to the image. The computing device includes: a storage unit storing a conditional cycle consistency image-to-image conversion model, the image-to-image conversion model being configured to convert the image to the second continuous domain space, wherein the model is adjusted by training using a plurality of encodings representing physical characteristics of the effect to be applied in a continuous manner, and wherein the plurality of encodings are encoded from a plurality of corresponding discrete instances of the effect to be applied, the corresponding discrete instances including instances representing no effect to be applied; and a processing unit configured to: receive an image and a desired instance of the effect to be applied; provide the image and the desired instance to the model to obtain a converted image; and provide the converted image for presentation.

[0018] The effect to be applied may be one of a makeup effect (eg, virtual lipstick), a nail effect (eg, a fingernail or toenail effect, such as a manicure and a nail treatment effect), and a hair dyeing effect.

[0019] The image may be free of the effect to be applied, and the first domain space is defined by a plurality of images each free of the effect to be applied.

[0020] The model may include a conditional cycle-consistent generative adversarial network (ccGAN) model.

[0021] The model may include a generator (G) and a discriminator (D) that are separately trained using a plurality of encodings encoded by an encoder (E).

[0022] The generator (G) may be conditioned using multiple encodings that operate normally with a convolutional batch conditional on each corresponding encoding in the multiple encodings.

[0023] The discriminator (D) may be conditioned using a plurality of encodings for an output projection operation of each respective encoding of the plurality of encodings.

[0024] The encoder may be configured to encode, for each of the respective instances including a sample image representing no effect to be applied, a physical characteristic of each of the respective instances of the effect to be applied from the respective sample image.

[0025] The model may include a trained generator (G).

[0026] The computing device may be coupled to a data storage (e.g., a data storage device that may be configured as a database), the data storage being configured to store a plurality of codes; and the processing unit may be configured to provide the model with a corresponding code of the plurality of codes corresponding to the desired instance of the effect to be applied. The data storage may store one or more untrained instance codes for corresponding instances of the effect to be applied that are not trained to the model; and the processing unit may be configured to provide the model with a corresponding untrained instance code of the one or more untrained instance codes corresponding to the desired instance of the effect to be applied.

[0027] The parameters θ of the generator G can be optimized by minimizing the adversarial loss and the cycle consistency loss in two complementary directions according to the following equation:

[0028]

[0029] The parameters θ of the discriminator D can be optimized by maximizing the adversarial loss in two complementary directions according to the following formula:

[0030]

[0031] The image may include a selfie image or a selfie video, and in the case of a selfie video, the processing unit is configured to convert the image of the selfie video to apply an effect thereto to create a 3D effect, in particular a 3D makeup effect.

[0032] The image may be pre-processed by a processing unit before being provided to the model.

[0033] The converted image may include a region of an image, and wherein the processing unit is configured to blend the converted image into the image for presentation.

[0034] The computing device may include or be coupled to a camera to capture images for receipt by the processing unit. The computing device may include a server, and the processing unit may be configured to receive the images and the desired instance from a client device in communication with the server.

[0035] The storage device may store instructions for an interface to provide a user with a desired instance selection function to define the desired instance, and the processor may be configured by the instructions to provide an interface, receive input, and define the desired instance in response to the input. The input may include a custom effect that has not been previously trained to a model. The storage device may store instructions for a search function to search for previously calculated encodings for defining the desired instance. The processor may be configured by the instructions to use the input and the search function to find the best match for the custom effect from the calculated encoding to define the desired effect. The storage device may store instructions for providing an encoding function to define a new encoding for the desired effect, and the processor may be configured by the instructions to use the input and the encoding function to define the desired effect for the custom effect.

[0036] A computing device configured to generate a conditional cycle-consistent image-to-image conversion model is provided, the model being configured to convert an image from a first domain space to a second continuous domain space according to an effect to be applied to the image. The computing device includes: a storage device storing a model including a generator, a discriminator, and an encoder, the model being configured to adjust the generator and the discriminator by training using multiple encodings from the encoder that represent physical properties of the effect to be applied in a continuous manner, and wherein the multiple encodings are encoded from multiple corresponding discrete instances of the effect to be applied, the corresponding discrete instances including instances representing no effect to be applied; and a processing unit configured to: receive training data and provide the training data to the model in batches to define a trained model. The model may include a conditional cycle-consistent generative adversarial network (ccGAN) model.

[0037] To receive the training data, the processing unit may be configured to: receive a plurality of real images x from a first domain space X, the first domain space including a domain space without applying an effect and each x has no effect applied; receive a plurality of real images y from a second continuous domain space Y j, the second continuous domain space includes the domain space to which the effect is applied and each y j having instances j to which an effect is applied, wherein j=1, 2, 3, ... denotes each of the corresponding discrete instances of the effect; receiving a plurality of reference images z j to represent each of the corresponding discrete instances of the effect; and receiving a white image z0 corresponding to no effect to be applied. The processing unit may also be configured to create parameters θ for the generator (G) and the discriminator (D) G and θ D To provide training data in batches, the processing unit may be configured to operate by repeatedly encoding the generator (G) and the discriminator (D) using the encoder in batches and epochs determined from the training data. j ) is encoded; the encoder is used to encode the code E(z0) for the generator; the image x and the code E(z0) are provided to the generator (G) j ) to generate a pseudo image in the second continuous domain space Feed the fake image to the discriminator (D) and the real image y corresponding to the pseudo image j And the code E(z j ) to generate the output d j ; Provide fake images to the generator (G) and encode E(z0) to output x'; and optimize the parameters θ of the generator (G) by minimizing the adversarial loss and cycle consistency loss G , and optimize the parameters θ of the discriminator (D) by maximizing the adversarial loss according to D :

[0038]

[0039]

[0040] Those of ordinary skill will also recognize equivalent method and computer program product aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a representation of the data flow of a model trained using a conditional GAN ​​with cycle consistency loss.

[0042] Figure 2 is a graphical representation in tabular form of the effects of downsampling, dilation, and number of residual block layers according to the instance.

[0043] Figure 3 is a graphical representation in tabular form showing the encoding space learned by interpolation over the color dimension (hue from 0 to 360°).

[0044] Figure 4 is a graphical representation in tabular form showing the encoding space learned by interpolation over the humidity dimension (from 0 to 1).

[0045] Figure 5 is a graphical representation in tabular form showing the interpolated learned encoding space given synthetic sample images on the test set. It should be noted that no real ground truth images are used during training.

[0046] The inventive concepts are best described by certain embodiments thereof, which are described herein with reference to the accompanying drawings, wherein like reference numerals represent like features throughout. It should be understood that the term "invention" as used herein is intended to encompass the inventive concepts underlying the embodiments described below, and not just the embodiments themselves. It should be further understood that the overall inventive concepts are not limited to the illustrative embodiments described below, and the following description should be read in this light. Although primarily described with reference to virtual lipstick as a makeup effect, as will be appreciated by one of ordinary skill in the art, the teachings herein may be applied to other augmented reality effects, including but not limited to makeup, nails, and hair coloring. DETAILED DESCRIPTION

[0047] Recently, conditional generative image-to-image translation networks such as conditional GANs with cycle consistency loss have achieved remarkable results on the task of image translation between domains. The main advantage of these methods is that they can be trained on unpaired data, which helps many problems, including augmented reality. It is almost impossible or prohibitive in terms of time and resources to obtain two real-world images, one with makeup effects such as lipstick and the other without makeup effects, while keeping all other variables such as pose and lighting the same. In other words, it is difficult to obtain a sufficient amount of paired real-world data such as real-world data for supervised learning.

[0048] Additional questions related to the effects to be applied can be considered. For example, in the lipstick domain (but similarly in other domains), a tuning parameter can be adopted so that it can represent the various different lipsticks we wish to apply, where this tuning parameter can be thought of as specifying a subdomain. This condition cannot be expressed simply as a discrete one-hot encoding of the lipstick product, because this approach is not scalable and does not affect the fact that lipsticks exist in a shared continuous space of physical properties. A method is disclosed that creates a representation of the physical properties (of the effect) of the lipstick by encoding a simple sample image and uses the encoding to condition an image-to-image translation model (e.g., a GAN). The method formulates the problem as a translation between 1) a unique no-lipstick domain and 2) a continuous lipstick domain.

[0049] Having described the task and the motivation behind it, it will be appreciated that this document discloses: (i) the training of a model that transforms images into a continuous domain space defined by physical (effect) lipstick properties, wherein (ii) the model automatically learns representations of these physical properties from simple inputs that contain all the necessary information about the lipstick (effect) such as sample images.

[0050] Model Architecture

[0051] Currently, we choose the CycleGAN architecture with some additional modifications [2] , to provide conditional cycle consistency to generate an image-to-image translation model. We replace the original two-domain architecture with a single pair of discriminator / generator and conditioning. In this way, it can be extended to more than two translation directions in a multi-dimensional continuous encoding space. In order to render arbitrary lipsticks that are not seen at training time, we represent the space of lipsticks as a continuous domain learned by encoding simple sample images of the products. As mentioned above, lipstick is an instance of a physical effect and other effects are expected (e.g., other facial or other skin makeup, hair dye, nail effects, etc.).

[0052] We also use the output projection method introduced in cGAN, which shows better training stability and less mode collapse.

[0053] From CycleGAN [2] Modify both the discriminator model architecture and the generator model architecture. The generator named ResNetGenerator is an encoder-bottleneck-decoder type architecture. The discriminator named NLayerDiscriminator is a regular encoder type architecture. The network details are listed below.

[0054] For better conditioning, we condition the discriminator with output projection and conditional batch normalization for the generator. We apply spectral normalization to all weights in both the discriminator and the generator.

[0055] Figure 1 A data flow 100 representing a model graph is shown. A no-lipstick image x 102 (e.g., a no-makeup image from a no-makeup domain) and a lipstick image y are provided. j 104 (e.g., makeup images from the makeup domain), where j represents the lipstick identity (e.g., in the target subdomain of makeup, j=1, 2, 3, ...). For each lipstick product j, a sample reference image z is provided j 106.

[0056] First, the reference image z is fed through the encoder 108 j 106 generates a representation of lipstick j to generate a representation including 110A and 110B e j =E(z j ) 110. Using the encoder 110A and the generator G 112, a pseudo lipstick image is generated 114 = G(x; e jG ).image 114 and y j 104 (i.e., the fake lipstick image and the real lipstick image) are passed to the discriminator D 116 to generate Figure 1 The reference numerals 118A and 118B are

[0057] Finally, (although not shown) cycle consistency is applied to Remove the lipstick. To this end, the generator G 112 is conditioned on the no-lipstick case, which is performed by encoding a blank white sample denoted z0, resulting in

[0058] Makeup style The encoding of 110A uses the convolutional conditional batch normal operation to condition the generator G 104. That is, the output of the encoder E is adapted to condition the CBN at different residual blocks of the generator network. This is achieved by inserting a small convolutional network between the encoder and each CBN of a given block (not all blocks must be conditioned by CBN, but ablation studies have shown that conditioning only one block is suboptimal). When training the generator G 112, all CCBNs are trained in order to retrieve the best weights for adaptive code conditioning.

[0059] coding 110B uses the output projection operation to condition the discriminator D 116. The encoder E 108 is a shallow network consisting of convolutional layers and adaptive average pooling layers, whose output is then reshaped into a one-dimensional encoding

[0060] The model architecture for each of the generator G 112, discriminator D 116, and encoder E 108 can also be found in the tables below relating to network details. The reference encoders are described in the tables for generator G and discriminator D, respectively.

[0061] During training, the parameters of the generator G 112 are optimized by minimizing the adversarial loss and the cycle consistency loss in two complementary directions:

[0062]

[0063]

[0064]

[0065]

[0066] Among them, p data (x) and p data (y) is the corresponding distribution of parameter data in the no-makeup domain and the makeup domain.

[0067] The parameters of the discriminator D 116 are optimized by maximizing the adversarial loss in two complementary directions.

[0068]

[0069]

[0070]

[0071] When inferring

[0072] During inference, only the generator G 112 and the encoder E 108 (or its encoding) associated with the generator G 112 are used, but in a slightly different way than during training. The process can be summarized as follows:

[0073] 1. For each lipstick j and its sample image z j , in the product catalog, obtain its code E G (z j ), and store it in a data storage (e.g., a database) for later use. If multiple sample images of a given lipstick are available, a more reliable code can be inferred by averaging the codes from each image.

[0074] 2. To render a given lipstick j on selfie image x, get the encoding of j from the database And run the generator G(x;E G (z j )).

[0075] 3. If the model was trained on a selfie cropped to the lip area, blend the output of the generator back into the original selfie image at the correct location.

[0076] The main difference between inference and training is that the encoding can be pre-computed for each product in the lipstick catalog. If necessary, these encodings can also be manually slightly adjusted on test selfies to produce renderings that better reflect real products. These encodings can be modified or combined in many ways to render lipsticks that do not actually exist. Users can be enabled to design their own lipsticks (e.g., custom effects that are desired instances of the effect to be applied) by providing input to the interface, such as by moving a digital slider (e.g., to select a value for the encoding). The search function can then find real products (e.g., custom effects) that closely match the user's creation (e.g., using the minimum Euclidean distance to the encoding in the product catalog). The encoding of the effect can then be provided to the generator along with an input image (e.g., still or video) to produce a modified image to virtually try out the user's creation. The custom effect can be encoded and provided to the generator at runtime. Thus, in a computing device having a processor and a storage unit storing instructions, these instructions configure the operation of the computing device when executed by the processor, and the storage device can store instructions for the interface to provide the user with a desired instance selection function that defines the desired instance. The processor may then provide an interface, receive input, and define the desired instance in response to the input.

[0077] The input may define a custom effect that was not previously trained to the model. The storage device may store instructions for a search function to search for previously calculated encodings for defining a desired instance. The processor may use the input and the search function to find the best match for the custom effect from the calculated encodings to define the desired effect. The storage device may store instructions for providing an encoding function to define a new encoding for the desired effect. The processor uses the input and the encoding function to define the desired effect from the custom effect.

[0078] When retraining

[0079] At any time, a new lipstick can be added to the product catalog simply by storing its sample encoding in the database (data storage). As described in the previous section, this will not require any additional training. However, for long-term maintenance of the application, it may be advantageous to periodically retrain the model when new lipsticks are added with physical properties that deviate significantly from those originally used to train the model.

[0080] For example, the model may have been originally trained on ordinary flat colors, and therefore will not generalize well to glittery lipsticks with heterogeneous properties. In this case, the model would need to be trained using samples of glittery lipsticks and their synthetic appearance when applied to different people. However, it should be noted that this is not a common case, as in practice the vast majority of lipsticks share the same type of physical properties.

[0081] The operations for training and for inference are summarized and shown in the following operation list:

[0082]

[0083]

[0084] Broadly, for a training operation, the corresponding generator and discriminator parameters are initialized. As is well known, training is performed in epochs, which include batches. In a training instance, training is performed using training data, which for each batch includes real images from a first domain without applied effects, real images from a second domain with applied effects, and encodings with applied effects. Encodings with no effects to be applied (white effect images) are also used. The model components (generator and discriminator) process the training data and determine a loss function. The corresponding generator and discriminator parameters are adjusted according to the loss function for the next training instance.

[0085] An additional operation step includes collecting a makeup product dataset (effect dataset). A dataset with large variations in products, identities, and environments preferably defines a continuous space. Ideally, the training images for conditional CycleGAN are instance images of people using different products in different contexts. For example, for each product, an additional operation may collect 10 sample reference images. Here, a variety of products will be the most important factor in order to cover the makeup encoding space and generalize well new unseen products belonging to the same space. In contrast, the variation and number of images of people who do not use any makeup can be relatively small. Following the above list of operations, the model is trained by adjusting on the physical representation of the makeup products being jointly learned by the encoder. The vectors of the encoded effects can be stored in a data storage including a storage device (e.g., a database) to provide to the model at inference time, rather than encoding samples at inference time. The data storage can define a catalog of effects.

[0086] Additional operations can be used to add new lipsticks to the catalog (data store) with or without retraining. Since we want to avoid retraining the model every time we add a new product to the catalog we support, the ability of the model to generalize to new products is a major differentiator from the prior art. In any case, if the new product to be added (e.g., a specific effect) belongs to the same feature space as the products on which the model has been trained, then adding a new product to the catalog is very straight forward. In other cases (e.g., where the new product is an outlier and is not in the feature space on which the model has been trained or if otherwise desired), the model can be retrained using the specific effect. For example, if the product to be added is significantly different from all existing products in the catalog (e.g., a new glitter lipstick when the catalog contains only a single shade), or if a large number of new products are to be added to the catalog, the model will be fine-tuned on those new products and retrained on the entire combined new dataset.

[0087] The innovation of the proposed image conversion method is that the generator and discriminator perform their tasks by learning and using the physical properties of the lipstick, which are extracted from simple sample images. Therefore, if the model is presented with samples of new products that it has never seen at training time, its task will remain the same and it should be able to generalize appropriately. It does not need to see any examples of what the new product looks like when applied to the face, because it can extract the physical properties of the product and know how those properties affect the final appearance.

[0088] To generalize, the model is trained to see enough variation in lipstick types during training to be able to learn the properties of the lipstick and its effect when applied to a person. For example, if the model is never shown an example of metallic lipstick, it will not learn to represent the unique properties of the material that makes up the metal, nor know how to use those properties in the rendering. On the other hand, if it has seen several types of metallic lipstick and is presented with one of a new color after training, it will be able to encode its properties and render it correctly, as long as it has also seen similar colors during training. It should be noted that it is not required to have seen similarly colored metallic lipsticks; seeing any type of lipstick with a similar color should be sufficient.

[0089] Abstractly, we can see that the number of different lipsticks required for training is a function of the size of the feature space of lipsticks we wish to render. While this value is difficult to quantify, it suggests an optimal data collection strategy that prioritizes varying lipstick types over varying environments with the same lipstick (i.e., many lipsticks with a few instances each, rather than a few lipsticks with many instances each).

[0090] Experiments and Results

[0091] For all our experiments, we use a dataset of images with and without lipstick. To simplify the experiments and their analysis, we do not consider other types of makeup, although the described methods can also be applied to them.

[0092] Evaluation Metrics

[0093] Evaluation of learned image-to-image translation models is often difficult due to the lack of consistent real-world images and varies depending on the nature of the task. In the case of makeup application, the lip region should ideally look as realistic as if the corresponding real lipstick were applied, while changes elsewhere on the face should be imperceptible.

[0094] First, we visually inspect the quality of the delta image, which is the normalized difference between the generated image and the input image. This can be done without the real ground truth image.

[0095] Although we can obtain consistent ground truth when evaluating training on synthetic datasets, it is usually difficult to obtain consistent ground truth on real makeup datasets. Instead of instance-wise or point-wise comparison, a common way to evaluate the performance of generative models is to compare the distribution difference between all ground truth images and generated images, regardless of whether they are consistent or not.

[0096] To reduce noise during evaluation, we only compare the lip regions of the generated images and the true ground truth images. Common ways to evaluate generative models include Frechet Inception Distance (FID)

[15] and Starting Score (IS)

[16] .

[0097] Downsampling, dilation, and number of layers

[0098] In the literature, multiple levels of downsampling make CNNs computationally practical on object classification tasks. However, for tasks that require detailed spatial information (e.g., image translation and reconstruction), this loss of spatial resolution can have a significant impact on the performance of the model. In contrast, increasing the dilation factor of the convolution or using a large number of residual blocks in the generator helps maintain spatial resolution and can be used to accurately model detailed lipstick distributions. Figure 2 Ablation studies on the impact of varying these three hyperparameters are shown. The corresponding evaluation results based on the metrics described in Section 4.1 are shown in Table 1.

[0099]

[0100] Table 1: Evaluation results for the 'all' test case with snapshots at epoch 400.

[0101] We found that the 9layer-1downsample-32dilation model gave the best performance. However, for real-time applications, such as makeup preview on mobile devices, its computation and storage usage exceeded practical limits. In this case, reducing the number of residual blocks to 2layers is a practical way to compromise between performance and computation.

[0102] Continuous encoding of the synthetic lipstick dataset

[0103] For proof of concept, we first try to train a hand-designed encoding space with five dimensions (R, G, B, humidity, glossDetail). Figure 1 In contrast, no reference encoder is needed in this case. Encoding for a specific lipstick Equal to the (R, G, B, humidity, glossDetail) attribute vector of the lipstick, and the reverse encoding towards the lipstick-free domain is a constant vector (-1, -1, -1, -1, -1).

[0104] Figure 3 and Figure 4 Some images of unshared identities in the test set are shown, generated by inserting this multidimensional encoding space in the RGB dimension (hue = 0-360°) and the wetness dimension (wetness = 0-1), respectively, at some equally spaced sample points that do not necessarily appear during training. Reference image encoding on the synthetic lipstick dataset

[0105] Figure 2 The images generated from the model trained on the synthetic RGB sample dataset are shown. The training set in this dataset was generated according to the following operations:

[0106] 1) Sample (R, G, B) point j from uniform distribution U(R, G, B).

[0107] 2) Pass this color and no-makeup photo x to a traditional virtual makeup try-on program to generate a synthetic image y of this color with lipstick j.

[0108] 3) Generate five synthetic sample images with similar colors to j and different shapes.

[0109] 4) Repeat steps 1 to 3 for all your no-makeup photos.

[0110] The test set contains all the identities in the training set, but with different face positions or orientations. It also contains additional identities that did not appear in the training set ( Figure 2 , 3, 4, and 5). A reference sample image is generated based on a constant array of twelve equally spaced hue interpolated color points that do not necessarily appear in the training set. During testing, for each image in the test set, the model places each of those twelve lipsticks on the test image.

[0111] Exemplary Computing Environment

[0112] The above method may be implemented in a computing device or system thereof (such as a client-server model). A neural network model may be provided via an application (e.g., on a computing device) or via a web browser (e.g., on a server computing device that communicates with a client device (e.g., a user device such as a tablet, smartphone, laptop, desktop, kiosk, or other form factor) to enable a user to take a selfie (an image or a video with the image) and virtually try on makeup in an augmented reality interface. The neural network model may be provided as a service from the cloud (e.g., receiving a no-makeup image and a makeup selection; and returning a "modified" or transformed image showing makeup applied to the no-makeup image).

[0113] The user may be provided with an interface (e.g., a graphical user interface (GUI)) to select or configure makeup to be applied, take a selfie image or selfie video, and view the selfie image or selfie video with the makeup effects applied. The selfie video may be converted to apply the effects in real time or near real time (e.g., to provide a 3D makeup effect that appears in the converted video).

[0114] Two or more generated images, each having different makeup applied, may be stored and presented (eg, side by side) for comparison in a GUI.

[0115] The interface may be activated / invoked in response to a gesture (eg, on a touch screen of the computing device), pointer input, voice input, text input, etc., and the computing device is configured with input and output devices and capabilities accordingly.

[0116] Instructions (software) may be provided (eg, for storage to a memory unit of a computing device) to configure the operation of the computing device, such as when the instructions are executed by a processing unit of the computing device.

[0117] In one aspect, a computing device converts an image from a first domain space to a second continuous domain space according to an effect to be applied to the image. The computing device includes: a storage unit storing a conditional cycle consistency generative adversarial network (ccGAN) model configured to convert the image to the second continuous domain space, wherein the ccGAN model is adjusted by training with a plurality of encodings representing physical properties of the effect to be applied in a continuous manner, and wherein the plurality of encodings are encoded from a plurality of corresponding discrete instances of the effect to be applied, the corresponding discrete instances including instances representing no effect to be applied; and a processing unit configured to: receive an image and a desired instance of the effect to be applied; provide the image and the desired instance to the model to obtain a converted image; and provide the converted image for presentation.

[0118] The effect to be applied may be a makeup effect, such as virtual lipstick as shown and described.

[0119] The effect to be applied may be a hair color effect. That is, the proposed method for makeup style transfer can be extended to hair color transfer / hair dyeing. Hair dyeing may be similarly applied for the purpose of modifying hair image pixels using a similar method - adjusting the generator G and the discriminator D using training hair dye images via the encoder E. The training images represent discrete instances of the effect (i.e., hair color), but are encoded to provide enough data to enable image transformation over a continuous (sub)domain for the effect to be applied at inference, as in the virtual lipstick example. In a similar manner, training is performed with a collection of images X of people with a variety of hair colors, covering the hair color space j of interest for dyeing applications. The training images X of the object (e.g., person) should be taken under a realistic range of lighting conditions. The training images will also include a sufficient number of different identities (i.e., different people) for identity preservation training.

[0120] To define the hair samples (e.g., image z) used for training by the encoder E, j), another set of images of the training subject's hair is collected from the first set of images. This second set of images is collected, for example, under controlled lighting conditions so that the hair samples for color instance j are not contaminated by objectionable factors like varying random lighting conditions. Sample hair image patches are extracted from these controlled images and used to encode specific hair color samples. The population of hair colors in training will define the color space. Similar to cosmetic product encoding, the hair color space need only be sampled sparsely, but preferably over a sufficient range to give the user a wider selection and more accurate results for interpolation within a continuous domain, such as when a color effect is desired but does not match the training instances. In inference, hair samples not seen during training will be encoded and used to change the hair color of the test subject (image), similar to a makeup effect. The no-hair effect z0 can be similarly defined as described herein for the makeup effect.

[0121] In addition to the computing device and method aspects, one of ordinary skill will understand that a computer program product aspect is disclosed in which instructions stored in a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.) configure a computing device to perform any method aspects stored therein.

[0122] The actual implementation may include any or all of the features described herein. These and other aspects, features and different combinations may be expressed as methods, devices, systems, means for performing functions, program products, and the features described herein are otherwise combined. A number of embodiments have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided, or steps may be eliminated from the described processes, and other components may be added to the described systems or removed. Therefore, other embodiments are within the scope of the appended claims.

[0123] Throughout the description and claims of this specification, the words "comprise" and "contain" and their variations mean "including but not limited to", and they are not intended to (and do not) exclude any other components, integers or steps. Throughout this specification, unless the context clearly indicates otherwise, the singular includes the plural. In particular, when the indefinite article is used, this specification should be understood to contemplate plural as well as singular, unless the context requires otherwise.

[0124] Unless there is an incompatible description, features, integers, characteristics, compounds, chemical moieties or groups described in conjunction with a particular aspect, embodiment or example of the invention should be understood to be applicable to any other aspect, embodiment or example. All features disclosed in this disclosure (including any attached claims, abstract and drawings), and / or all steps of any method or process so disclosed, may be combined in any combination, except that at least some of such features and / or steps are mutually exclusive combinations. The present invention is not limited to the details of any foregoing examples or embodiments. The present invention extends to any novelty or any novel combination of all features disclosed in this specification (including any attached claims, abstract and drawings), or any novelty or any novel combination of steps of any method or process disclosed.

[0125] Importantly, the neural network model transforms the computing device from a general computing device to a special device that has not been seen before. The apparatus and methods provided according to the neural network model and other aspects enable the transformation or reduction of a specific item to a different state or thing - converting an image from one domain space to a second continuous domain space where effects are added. The apparatus and methods provided according to the neural network model and other aspects involve specific features and / or functions and / or operations, either (or both) of which 1) are different from routine conventional activities that are well understood in the art, or 2) add unconventional steps that limit the aspect to specific useful applications.

[0126] Network Details

[0127]

[0128]

[0129] Table 2: Generator network architecture.

[0130]

[0131] Table 3: Discriminator network architecture.

[0132] References

[0133] [1] P.Isola, J.-Y.Zhu, T.Zhou, and AAEfros, "Image-to-image translation with conditional adversarial networks," in Conference on Computer Vision andPattern Recognition, 2017.

[0134] [2]J.-Y.Zhu,T.Park,P.Isola,and A.A.Efros,“Unpaired image-to-imagetranslation using cycle-consistent adversarial networks,”in Computer Vision(ICCV),2017IEEE International Conference on,2017.

[0135] [3]H.Chang,J.Lu,F.Yu,and A.Finkelstein,“PairedCycleGAN:Asymmetricstyle transfer for applying and removing makeup,”in CVPR 2018,June 2018.

[0136] [4]I.Goodfellow,J.Pouget-Abadie,M.Mirza,B.Xu,D.Warde-Farley,S.Ozair,A.Courville,and Y.Bengio,“Generative adversarial nets,”in Advances in NeuralInformation Processing Systems 27(Z.Ghahramani,M.Welling,C.Cortes,N.D.Lawrence,and K.Q.Weinberger,eds.),pp.2672–2680,Curran Associates,Inc.,2014.

[0137] [5]A.van den Oord,N.Kalchbrenner,and K.Kavukcuoglu,“Pixel recurrentneural networks,”CoRR,vol.abs / 1601.06759,2016.

[0138] [6]D.P.Kingma and M.Welling,“Auto-Encoding Variational Bayes,”ArXive-prints,Dec.2013.

[0139] [7]D.P.Kingma and P.Dhariwal,“Glow:Generative Flow with Invertible1x1Convolutions,”ArXiv e-prints,July 2018.

[0140] [8]T.Miyato and M.Koyama,“cGANs with projection discriminator,”inInternational Conference on Learning Representations,2018.

[0141] [9]M.Mirza and S.Osindero,“Conditional generative adversarial nets,”arXiv preprint arXiv:1411.1784,2014.

[0142]

[10] S.Reed,Z.Akata,X.Yan,L.Logeswaran,B.Schiele,and H.Lee,“Generativeadversarial text-to-image synthesis,”in Proceedings of The 33rd InternationalConference on Machine Learning,2016.

[0143]

[11] Y.Choi,M.Choi,M.Kim,J.-W.Ha,S.Kim,and J.Choo,“Stargan:Unifiedgenerative adversarial networks for multi-domain image-to-image translation,”arXiv preprint arXiv:1711.09020,2017.

[0144]

[12] E.Perez,H.de Vries,F.Strub,V.Dumoulin,and A.C.Courville,“Learningvisual reasoning without strong priors,”CoRR,vol.abs / 1707.03017,2017.

[0145]

[13] E.Perez,H.De Vries,F.Strub,V.Dumoulin,and A.Courville,“LearningVisual Reasoning Without Strong Priors,”in ICML 2017’s Machine Learning inSpeech and Language Processing Workshop,(Sidney,France),Aug.2017.

[0146]

[14] T.Miyato,T.Kataoka,M.Koyama,and Y.Yoshida,“Spectral normalizationfor generative adversarial networks,”in International Conference on LearningRepresentations,2018.

[0147]

[15] M.Heusel,H.Ramsauer,T.Unterthiner,B.Nessler,G.Klambauer,andS.Hochreiter,“Gans trained by a two time-scale update rule converge to a nashequilibrium,”CoRR,vol.abs / 1706.08500,2017.

[0148]

[16] T.Salimans,I.J.Goodfellow,W.Zaremba,V.Cheung,A.Radford,andX.Chen,“Improved techniques for training gans,”CoRR,vol.abs / 1606.03498,2016.

Claims

1. A computing device for converting an image from a first domain space to a second continuous domain space according to an effect to be applied to the image, the computing device include: a storage unit storing an image-to-image conversion model generated by conditional cycle consistency, the image-to-image conversion model being configured to convert the image to the second continuous domain space, wherein the model is adjusted by training using a plurality of encodings representing physical properties of the effect to be applied in a continuous manner, and wherein the plurality of encodings are encoded from a plurality of corresponding discrete instances of the effect to be applied, the corresponding discrete instances including instances representing no effect to be applied; and a processing unit configured to, at inference time: receive the image and a desired instance of the effect to be applied; provide the image and the desired instance to the model to obtain a transformed image; and provide the transformed image for presentation; wherein the model comprises a generator (G) at inference time, wherein, at training time, the generator (G) is trained together with a discriminator (D) using a plurality of the encodings encoded by an encoder (E), the encodings comprising encodings with and without effects to be applied; and, According to the following formula, the parameter θ of the generator (G) G It includes the parameters optimized by minimizing the adversarial loss and the cycle consistency loss; and the parameters θ of the discriminator (D) D Contains the parameters optimized by maximizing the adversarial loss: Among them, z j a jth reference image comprising a plurality of reference images representing each respective discrete instance of an effect to be applied; and z 0 Includes a white image corresponding to no effect to be applied.

2. The computing device according to claim 1, in, The effects to be applied include augmented reality effects.

3. The computing device according to claim 1, in, The effect to be applied includes one of a makeup effect, a nail effect, and a hair dyeing effect.

4. The computing device according to claim 1, in, The model includes a conditional cycle consistency generative adversarial network ccGAN model.

5. The computing device according to claim 1, in, The image does not have the effect to be applied, and the first domain space is defined by a plurality of images each without the effect to be applied.

6. The computing device according to claim 1, in, The generator (G) is conditioned using a plurality of said encodings by using a convolutional conditional batch normal operation for each corresponding encoding of the plurality of said encodings.

7. The computing device according to claim 1, in, The discriminator (D) is conditioned using a plurality of said codes by using an output projection operation for each respective one of said plurality of codes.

8. The computing device according to claim 1, in, The encoder is configured to encode, for each of the respective instances including a sample image representing no effect to be applied, a physical characteristic of each of the respective instances of the effect to be applied from the respective sample image.

9. The computing device according to claim 1, in, The model includes the trained generator (G).

10. The computing device according to claim 9, in, The computing device is coupled to a data storage device, the data storage device comprising a storage device configured to store a plurality of the codes; and The processing unit is configured to provide to the model a corresponding encoding of the plurality of encodings corresponding to the desired instance of the effect to be applied.

11. The computing device according to claim 10, in, The data storage stores one or more untrained instance encodings for corresponding instances of the effect to be applied that are not trained to the model; and The processing unit is configured to provide the model with a corresponding one of the one or more untrained instance encodings that corresponds to the desired instance of the effect to be applied.

12. The computing device according to claim 1, in, About formula (1): Among them, x is the image in the first domain space, y is the image in the second continuous domain space, and p data (x) and p data (y) is the distribution of parameter data in the first and second domain spaces, respectively.

13. The computing device according to claim 1, in, Regarding formula (2): Among them, x is the image in the first domain space, y is the image in the second continuous domain space, and p data (x) and p data (y) is the distribution of parameter data in the first and second domain spaces, respectively.

14. The computing device according to claim 1, in, The image includes a selfie image or a selfie video, and in the case of the selfie video, the processing unit is configured to convert the image of the selfie video to apply the effect to the image of the selfie video to create a 3D effect.

15. The computing device according to claim 1, in, The converted image comprises a region of the image, and wherein the processing unit is configured to blend the converted image into the image for presentation.

16. The computing device according to claim 1, in, One of the following holds true: a. The computing device includes a camera; and b. The computing device is coupled to a camera; The image is thereby captured for receipt by the processing unit.

17. The computing device according to claim 1, in, The computing device includes a server, and the processing unit receives the image and the desired instance from a client device in communication with the server.

18. The computing device according to claim 1, in, The storage device stores instructions for an interface to provide a desired instance selection function to a user to define the desired instance, and wherein the processor provides the interface, receives input, and defines the desired instance in response to the input.

19. The computing device according to claim 18, in, The input includes custom effects that were not previously trained into the model.

20. The computing device according to claim 19, in, The storage device stores instructions for a search function to search for previously calculated codes for defining the desired effect, and wherein the processor uses the input and the search function to find a best match for the custom effect from the calculated codes to define the desired effect.

21. The computing device of claim 19, in, The storage device stores instructions for providing an encoding function to define a new encoding for the desired effect, and wherein the processor uses the input and the encoding function to define the desired effect for the custom effect.

22. A computing device configured to generate a conditionally cycle-consistent generated image-to-image translation model, the model configured to translate an image from a first domain space to a second continuous domain space according to an effect to be applied to the image, the computing device include: a storage device storing the model, the model comprising a generator, a discriminator, and an encoder, the model being configured to condition the generator and the discriminator by training using a plurality of encodings from the encoder representing in a continuous manner the physical characteristics of the effect to be applied, and wherein the plurality of encodings are encoded from a plurality of corresponding discrete instances of the effect to be applied, the corresponding discrete instances including instances representing no effect to be applied; and A processing unit, wherein the processing unit is configured to: receiving training data and providing the training data in batches to define the model to be trained; in: In order to receive the training data, the processing unit is configured to: receiving a plurality of real images x from the first domain space X, the first domain space X comprising a domain space without effect applied and each x having no effect applied; Receive multiple real images y from the second continuous domain space Y j , the second continuous domain space Y includes the domain space to which the effect is applied and each y j having an instance j to which the effect is applied, wherein j=1, 2, 3, ... represents each of the corresponding discrete instances of the effect; Receive multiple reference images z j to represent each of said corresponding discrete instances of said effect; and Receives a white image z corresponding to no effect to be applied 0 ; The processing unit is also configured to create parameters θ for the generator (G) and the discriminator (D) G and θ D ;and In order to provide the training data in batches, the processing unit is configured to operate as follows: Repeatedly, with batches and training epochs determined from the training data, The encoder is used to encode E(z) for the generator (G) and the discriminator (D). j ) for encoding; Use the encoder to encode E(z 0 ) for encoding; Provide the image x and the encoding E(z j ) to generate a pseudo image in the second continuous domain space The fake image is provided to the discriminator (D) and the real image y corresponding to the pseudo image j And the code E(z j ) to generate the output d j ; Providing the pseudo image to the generator (G) and code E(z 0 ) to output x'; and The parameters θ of the generator (G) are optimized by minimizing the adversarial loss and the cycle consistency loss according to the following formula: G , and optimize the parameters θ of the discriminator (D) by maximizing the adversarial loss D : Among them, z j a jth reference image comprising a plurality of reference images representing each respective discrete instance of an effect to be applied; and z 0 Includes a white image corresponding to no effect to be applied.

23. The computing device according to claim 22, in, The effect to be applied includes one of a makeup effect, a nail effect, and a hair dyeing effect.

24. The computing device of claim 22, in, A generator (G) is conditioned using a plurality of said encodings by using a convolutional conditional batch normal operation for each corresponding encoding of the plurality of said encodings.

25. The computing device of claim 22, in, The discriminator (D) is conditioned using a plurality of said codes by using an output projection operation for each respective one of said plurality of codes.

26. A computer-implemented method for converting an image from a first domain space to a second continuous domain space according to an effect to be applied to the image, the method include: storing in a storage unit an image-to-image conversion model generated by conditional cycle consistency, the image-to-image conversion model being configured to convert the image to the second continuous domain space, wherein the model is adjusted by training using a plurality of encodings representing physical properties of the effect to be applied in a continuous manner, and wherein the plurality of encodings are encoded from a plurality of corresponding discrete instances of the effect to be applied, the corresponding discrete instances including instances representing no effect to be applied; and By processing unit: receiving the image and a desired instance of the effect to be applied; providing the image and the desired instance to the model to obtain a transformed image; and providing the converted image for presentation; wherein the model comprises a generator (G) and a discriminator (D) respectively trained using a plurality of the encodings encoded by an encoder (E); and, According to the following formula, the parameter θ of the generator (G) G Includes parameters optimized by minimizing the adversarial loss and cycle consistency loss; and the parameters θ of the discriminator (D) D Contains the parameters optimized by maximizing the adversarial loss: Among them, z j a jth reference image comprising a plurality of reference images representing each respective discrete instance of an effect to be applied; and z 0 Includes a white image corresponding to no effect to be applied.

27. The method according to claim 26, in, The model is defined from a conditional cycle consistency generative adversarial network ccGAN model.

28. The method according to claim 26, in, The effect to be applied includes one of a makeup effect, a nail effect, and a hair dyeing effect.

29. The method according to claim 26, in, The image includes an image without the effect to be applied, and the first domain space is defined by a plurality of images each without the effect to be applied.

30. The method according to claim 26, in, The generator (G) is conditioned using a plurality of said encodings by using a convolutional conditional batch normal operation for each corresponding encoding of the plurality of said encodings.

31. The method according to claim 26, in, The discriminator (D) is conditioned using a plurality of said codes by using an output projection operation for each respective one of said plurality of codes.

32. The method according to claim 26, in, The encoder is configured to encode, for each of the respective instances including a sample image representing no effect to be applied, a physical characteristic of each of the respective instances of the effect to be applied from the respective sample image.

33. The method according to claim 26, in, The model includes the trained generator (G).

34. A method according to claim 33, comprising retrieving a corresponding one of a plurality of said codes corresponding to the desired instance of the effect to be applied from a data storage device including a storage device storing a plurality of said codes, for providing to the model.

35. The method according to claim 34, in, The data store stores one or more untrained instance encodings for corresponding instances of the effect to be applied that are not trained to the model; and The method includes retrieving from the database to provide to the model a corresponding one of the one or more untrained instance encodings that corresponds to the desired instance of the effect to be applied.

36. The method according to claim 26, in, About formula (1) Where x is the image in the first domain space, y is the image in the second continuous domain space, p data (x) and p data (y) is the distribution of parameter data in the first and second domain spaces, respectively.

37. The method according to claim 26, in, About formula (2) Among them, x is the image in the first domain space, y is the image in the second continuous domain space, and p data (x) and p data (y) is the distribution of parameter data in the first and second domain spaces, respectively.

38. The method of claim 26, comprising capturing the image using a camera coupled to the processing unit via the processing unit to receive the image.

39. The method according to claim 26, in, The image includes one of a selfie image and a selfie video, and in the case of the selfie video, the method converts the image of the selfie video to apply the effect to the image of the selfie video to create a 3D effect.

40. The method of claim 26, comprising pre-processing the image by the processing unit before providing it to the model.

41. The method according to claim 26, in, The converted image comprises a region of the image, and wherein the method comprises fusing the converted image to the image for presentation by the processing unit.

42. The method according to claim 26, include: providing an interface including a desired instance selection function to define the desired instance; And receiving input via the desired instance selection function, and defining the desired instance in response to the input.

43. The method according to claim 42, in, The input includes custom effects that were not previously trained into the model.

44. The method of claim 43, comprising searching previously calculated encodings for defining the desired effect using the input to find a best match for the custom effect from among the calculated encodings to define the desired effect.

45. The method of claim 43, comprising using an encoding function to define a new encoding for the desired effect in response to the input to define the desired effect for the custom effect.

46. ​​A computer-implemented method for generating a model of an image-to-image translation model generated by conditional cycle consistency, the model being configured to transform an image from a first domain space to a second continuous domain space according to an effect to be applied to the image, the method include: storing the model in a storage device, the model comprising a generator, a discriminator, and an encoder, the model being configured to adjust the generator and the discriminator by training using a plurality of encodings from the encoder representing in a continuous manner the physical characteristics of the effect to be applied, and wherein the plurality of encodings are encoded from a plurality of corresponding discrete instances of the effect to be applied, the corresponding discrete instances including instances representing no effect to be applied; and By processing unit: receiving training data and providing the training data to the model in batches to define the trained model; and in: In order to receive the training data, the method comprises: receiving a plurality of real images x from the first domain space X, the first domain space X comprising a domain space without effect applied and each x having no effect applied; Receive multiple real images y from the second continuous domain space Y j , the second continuous domain space Y includes the domain space to which the effect is applied and each y j having an instance j to which the effect is applied, wherein j=1, 2, 3, ... represents each of the corresponding discrete instances of the effect; Receive multiple reference images z j to represent each of said corresponding discrete instances of said effect; and Receive a white image z corresponding to no effect to be applied 0 ; The method further comprises creating, by the processing unit, parameters θ for the generator (G) and the discriminator (D) G and θ D ;and In order to provide the training data in batches, the method includes: Repeatedly, with batches and training epochs determined from the training data, The encoder is used to encode E(z) for the generator (G) and the discriminator (D). j ) for encoding; Use the encoder to encode E(z 0 ) for encoding; Provide the image x and the encoding E(z j ) to generate a pseudo image in the second continuous domain space The fake image is provided to the discriminator (D) and the real image y corresponding to the pseudo image j And the code E(z j ) to generate the output d j ; Providing the pseudo image to the generator (G) and code E(z 0 ) to output x'; and The parameters θ of the generator (G) are optimized by minimizing the adversarial loss and the cycle consistency loss according to the following formula: G , and optimize the parameters θ of the discriminator (D) by maximizing the adversarial loss D : Among them, z j a jth reference image comprising a plurality of reference images representing each respective discrete instance of an effect to be applied; and z 0 Includes a white image corresponding to no effect to be applied.

47. The method according to claim 46, in, The effects to be applied include augmented reality effects.

48. The method according to claim 47, in, The effect to be applied includes one of a makeup effect, a nail effect, and a hair dyeing effect.

49. The method according to claim 46, in, A generator (G) is conditioned using a plurality of said encodings using a convolutional conditional batch normal operation for each corresponding encoding of said plurality of said encodings.

50. The method according to claim 46, in, The discriminator (D) is conditioned using a plurality of said codes using an output projection operation for each respective one of said plurality of said codes.

51. A computer program product comprising a non-transitory storage unit storing instructions which, when executed by a processing unit of a computing device, configure the computing device to perform a method of transforming an image from a first domain space to a second continuous domain space according to an effect to be applied to the image, the method include: receiving the image and a desired instance of the effect to be applied; providing the image and the desired instance to a model to obtain a transformed image; and providing the converted image for presentation; wherein the model comprises an image-to-image translation model generated by conditional cycle consistency, the image-to-image translation model being configured to translate the image into the second continuous domain space, wherein the model is conditioned by training using a plurality of encodings representing physical properties of the effect to be applied in a continuous manner, and wherein the plurality of encodings are encoded from a plurality of corresponding discrete instances of the effect to be applied, the corresponding discrete instances comprising instances representing no effect to be applied; wherein the model comprises a generator (G) and a discriminator (D) respectively trained using a plurality of the encodings encoded by an encoder (E); and, According to the following formula, the parameter θ of the generator (G) G Includes parameters optimized by minimizing the adversarial loss and cycle consistency loss; and the parameters θ of the discriminator (D) D Contains the parameters optimized by maximizing the adversarial loss: Among them, z j a jth reference image comprising a plurality of reference images representing each respective discrete instance of an effect to be applied; and z 0 Includes a white image corresponding to no effect to be applied.

Images