Applying continuous effects with model estimation class embedding

Abstract

A method is provided for processing images using a deep learning model that achieves continuous effect simulation by a unified network in which a simple (effect class) estimator is embedded in a conventional encoder-decoder architecture. The estimator allows learning of model-estimated class embeddings for all effect classes, representing continuous effect information without manual efforts in selecting appropriate anchor effect groups. Given a target age class, a personalized age embedding is derived that considers two aspects of facial aging: 1) a personalized residual age embedding at the subject's model-estimated age, which preserves the subject's aging information, and 2) a typical facial aging basis at the target age that encodes the aging patterns shared among the entire population. Training is shown including an AR app that generates recommendations and provides e-commerce services.

Description

[Technical field]

[0001] 《Cross reference》 This application claims priority to U.S. Provisional Application No. 63 / 129,794, filed December 23, 2020, and French Application No. FR 2105404, filed May 25, 2021, the entire contents of each of which are incorporated herein by reference.

[0002] This application relates to image processing and image processing using neural network techniques to generate updated images and continuously apply effects such as facial image aging. [Background technology]

[0003] Facial aging, also known as age progression, aims to aesthetically render input facial images with natural aging or rejuvenation effects while preserving personal identity information. With recent advances in deep learning, face synthesis has also shown substantial improvements in image fidelity and age accuracy in simulated facial images [10, 41, 24]. The main challenge to solve various remaining problems (e.g., continuous aging) is the lack of data. For example, many research works on facial aging [20, 41, 43, 10] need to group images into 4-5 age groups (<30, 30-40, 40-50, >50, etc.), and due to the limited amount of data at each age, they can only generate images within the target age group. Another important issue is how to maintain individual characteristics in age progression, since aging patterns may vary from individual to individual.

[0004] Traditional facial aging mainly includes two approaches: physical model-based [3, 42] and prototype-based [37, 16]. Physical model-based approaches often consist of complex physical modeling, taking into account skin wrinkles, facial shape, muscle variations, and hair color. This type of approach usually requires a huge amount of data and is computationally very expensive. Prototype-based approaches first explore group-based designs by calculating the average face in a predefined age group, so they cannot retain personalized aging information. Furthermore, all these approaches are not applicable to continuous facial aging.

[0005] Following the success of recent generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs) [9] for image translation tasks, researchers have devoted efforts to adapting those methods to face synthesis. IPCGAN

[41] shows significant progress in generating face images with obvious age effects by implementing an age estimation loss. A later variant

[43] generates a pyramidal structure for the classifier to improve the understanding of facial aging at multiple scales. Among these methods, continuous aging was not explored. He et al.

[10] introduced a multi-branch generator for group-based training and proposed the idea of ​​approximating continuous aging via linear interpolation of latent representations between two adjacent age groups. The authors in

[24] tackle the problem with a similar linear interpolation approach, instead performed on the learned age latent code between two adjacent groups. These types of methods assume that age progression is linear between two adjacent groups, and the learned group embeddings can be used directly as median age embeddings. As a result, this can result in a shift in the target age in the generated images. Intuitively, this nonlinearity can be interpreted as people not aging at the same rate in different stages. Furthermore, such interpolation-based methods may alter personal features when separation is imperfect. Summary of the Invention

[0006] To address the above issues, a novel approach is proposed to achieve the application of continuous facial effects such as aging by an integrated network, in which a simple class estimator (e.g., age estimator for aging effects, smile progression (class) for continuous smile effects, etc.) is embedded into a regular encoder-decoder architecture. This allows the network to learn model-estimated class (e.g., age, smile, etc.) embeddings for all progression stages or classes (e.g., age, smile degree, etc.), thus representing the continuous effect information without manual efforts in selecting appropriate anchor progression stage (e.g., age, smile degree, etc.) groups. Given a target age in the age example (target age is one of the classes in the continuous effect), a personalized age embedding is derived that considers two aspects of facial aging: 1) a personalized residual age embedding at the subject's current age in the image, which preserves the individual aging information, and 2) a typical facial aging basis at the target age, which encodes the aging pattern shared among the entire population. Detailed computation and training mechanisms are described. The computed target age embeddings are then used to generate the final image. Experiments on the FFHQ

[15] and CACD2000 [5] datasets are detailed. Results show significant improvements over the state of the art in various aspects, both qualitatively and quantitatively.

[0007] In the age context, embodiments include a novel method for deriving personalized age embeddings for face aging tasks by self-estimating (e.g., "self" refers to estimation by a model (e.g., model estimation)) continuous age embeddings and jointly training an age estimator with a generator. Experiments and analysis quantitatively and qualitatively demonstrate that the generated images better retain personalized information, achieve more accurate aging control, and present finer aging details. The continuous aging approach according to embodiments herein generates images with better aligned target ages and better retains more detailed personal features without manual efforts to define appropriate age groups.

[0008] The proposed techniques and methods for model estimating personalized age embeddings from associated discriminative models can be easily applied to other conditional image-to-image translation tasks without introducing extra complexity. In particular, tasks with continuous states and modeling (e.g., non-smile, smile, etc.) can benefit from this setup.

[0009] In one embodiment, a method is provided that includes providing an integrated age simulation model for generating a new image at a target age of a subject from an input image of the subject, and generating the new image using the model, wherein the integrated age simulation model provides a plurality of respective model-estimated age embeddings at each of a plurality of continuous ages representing continuous aging information, the model-estimated age embeddings being learned through joint training of a generator, an age estimator being embedded in an encoder-decoder architecture of the model, the age estimator being configured to determine a model-estimated age of the subject from respective encoder-generated features in response to respective input images, and the generator generating the new image using the encoder-generated features from the input image transformed by each of the model-estimated age embeddings determined according to the target age and the model-estimated age of the subject.

[0010] In one embodiment, the encoder-decoder architecture comprises an age estimator for estimating a model-estimated age of the subject in the input image.

[0011] In one embodiment, the model's encoder processes the input image and the age estimator processes features generated by a good encoder to determine the model's estimated age.

[0012] In one embodiment, the features generated by the encoder are transformed by a personalized age embedding comprising a respective one of the model-estimated age embeddings determined according to the model-estimated age and a respective one of the model-estimated age embeddings determined according to the target age. In one embodiment, the personalized age embedding comprises a personalized residual age embedding determined from a plurality of respective model-estimated age embeddings according to a model-estimated age to preserve identity information of the subject, and a typical age embedding comprising a respective one of the model-estimated age embeddings determined according to the target age to represent a shared aging pattern among an entire population.

[0013] In one embodiment, the personalized age embedding is applied according to an affine transformation.

[0014] In one embodiment, the generator generates the new image at the target age by processing the encoded features transformed by the personalized age embedding.

[0015] In one embodiment, the model comprises one or both of a deep learning neural network model and a generative adversarial network-based model.

[0016] In one embodiment, the method provides a recommendation interface for obtaining recommendations for products and / or services.

[0017] In one embodiment, the method includes providing an e-commerce purchasing interface for purchasing one or both of the product and the service.

[0018] In one embodiment, the method receives the input image and provides the new image for display.

[0019] In one embodiment, the input image and the new image each include a face of the subject.

[0020] According to one embodiment, a method is provided that includes providing an integrated model for generating new images in a target class of continuous effects for a subject from an input image of the subject, and generating the new images using the model, the model providing a plurality of respective model estimated class embeddings in each of a plurality of continuous target ranges representing continuous effect information, the model estimated class embeddings being learned through joint training of a generator, an effect estimator being embedded in an encoder-decoder architecture of the model, the effect estimator being configured to determine a model estimated class for each subject from respective encoder generated features in response to each input image, and the generator generating the new images using the model estimated class embeddings determined according to the target class and the encoder generated features from the input images transformed by each of the model estimated classes of the subject.

[0021] According to one embodiment, a method is provided for providing a domain transfer model for transferring an input image to a new image, successively applying a continuous effect to transform the input image to a target class of a plurality of continuous classes of the continuous effect using a plurality of respective model-estimated class embeddings learned by a model for each of the continuous classes of the continuous effect, and transferring the input image to the new image using the domain transfer model. According to one embodiment, the continuous effect is an aging effect, and the target class is a target age. According to one embodiment, when transferring the input image, the domain transfer model operates as follows: a) generating coded features of the input image, b) transforming the coded features and transforming the coded features using a typical age embedding including a personalized residual age embedding determined from a plurality of respective model-estimated class embeddings according to a model-estimated age of the subject in the input image to preserve subject identity information, and each of the model-estimated residual age embeddings determined according to the target age to represent a shared aging pattern among the entire population, and c) generating the new image using the transformed coded features. According to one embodiment, the model includes an age estimator for determining the model-estimated age. According to one embodiment, the age estimator comprises a classifier trained together with an encoder (of said model), said encoder being configured to generate said encoded features, and said age estimator is trained to determine a model-estimated age for each of the subjects in a novel image using each encoded feature encoded by said encoder. According to one embodiment, said model-estimated class embeddings are defined during training of said age estimator together with said encoder, relating each of said model-estimated class embeddings to said respective model-estimated ages.

[0022] According to one embodiment, the method includes providing a recommendation for at least one of a product and a service associated with the sequence effect. According to one embodiment, the recommendation is generated in response to one or both of a skin analysis of the input image and a user input of preferences. According to one embodiment, the target age is determined in response to the recommendation. According to one embodiment, the computing device is configured to communicate with an e-commerce service for the recommendation. According to one embodiment, the computing device is configured to provide an annotated image generated from the input image to present the recommendation. According to one embodiment, the method provides an e-commerce interface for purchasing a product, a service, or both. According to one embodiment, the method includes receiving the input image from a camera. According to one embodiment, the sequence effect is an aging effect, the product includes one of a rejuvenating product, an anti-aging product, and a cosmetic makeup product, and the service includes one of a rejuvenating service, an anti-aging service, and a cosmetic service.

[0023] According to one embodiment, there is provided a computing apparatus comprising a processing unit and a storage device coupled thereto, the storage unit storing instructions which, when executed by the processing unit, configure the computing apparatus to perform a method according to any one of the method embodiments.

[0024] According to one embodiment, there is provided a computer program product comprising a non-transitory storage device storing instructions which, when executed by a processing unit of a computing device, configure said computing device to perform a method according to any one of the method embodiments.

[0025] According to one embodiment, there is provided a computing device comprising a processing unit and a storage device coupled thereto, the storage unit storing instructions which, when executed by the processing unit, cause the computing device to recommend at least one of a product and a service, and provide an age simulation image comprising a new image generated from an input image and a target age, the new image being generated according to any one of the embodiments of the method. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 is a block diagram of a model architecture showing a training configuration, according to an embodiment of the present disclosure. [Figure 2A] 1 is an array of images showing an input image with results from two models according to embodiments of the present specification, and results from a prior art model for an example showing more aging detail and identity retention in the results from the two models according to respective embodiments of the present specification. [Figure 2B] 1 is an array of images showing an input image with results from two models according to embodiments of the present specification, and results from a prior art model for an example showing more aging detail and identity retention in the results from the two models according to respective embodiments of the present specification. [Figure 2C] 1 is an array of images showing an input image with results from two models according to embodiments of the present specification, and results from a prior art model for an example showing more aging detail and identity retention in the results from the two models according to respective embodiments of the present specification. [Figure 2D] 1 is an array of images showing an input image with results from two models according to embodiments of the present specification, and results from a prior art model for an example showing more aging detail and identity retention in the results from the two models according to respective embodiments of the present specification. [Figure 2E]1 is an array of images showing an input image with results from two models according to embodiments of the present specification, and results from a prior art model for an example showing more aging detail and identity retention in the results from the two models according to respective embodiments of the present specification. [Figure 2F] 1 is an array of images showing an input image with results from two models according to embodiments of the present specification, and results from a prior art model for showing more aging detail and identity retention in the results from the two models according to respective embodiments of the present specification. [Figure 3A] 1 is an array of images showing an input image with results from a model according to an embodiment of the present disclosure, and results from three prior art models for an example. [Figure 3B] 1 is an array of images showing an input image with results from a model according to an embodiment of the present disclosure, and results from three prior art models for an example. [Figure 3C] 1 is an array of images showing an input image with results from a model according to an embodiment of the present disclosure, and results from three prior art models for an example. [Figure 3D] 1 is an array of images showing an input image with results from a model according to an embodiment of the present disclosure, and results from three prior art models for an example. [Figure 4] 1 is an array of images showing an input image with results from a model according to embodiments herein and an enlarged face crop of those results showing more details of aging. [Diagram 5] A and B are arrays of images showing input images with results from a model according to an embodiment of the present specification, and results from a prior art model for two examples showing continuous aging over 4 years from age 21. [Figure 6] 13A and 13B show confusion matrices for continuous aging comparing results from a model according to an embodiment of the present disclosure with results from a prior art model. [Figure 7]11 is an array of images illustrating linear interpolation between transformed discriminant codings from a model according to one embodiment of the present disclosure. [Figure 8] 1 is an array of images showing input images and results from a first model according to an embodiment herein using residual embedding, and a second (comparator) model according to an embodiment herein using no residual embedding. [Figure 9] FIG. 1 is a block diagram of a computer system including multiple computing devices according to one embodiment.

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Face synthesis, especially including facial aging, has been one of the major topics that has substantially improved image fidelity by using generative adversarial networks (GANs). Most existing face aging approaches split the dataset into several age groups and utilize a group-based training strategy, which inherently lacks the ability to provide fine-tuned continuous aging synthesis. In one embodiment, a unified network structure is provided that embeds a linear age estimator into a GAN-based model, where the embedded age estimator is trained together with the encoder and decoder to estimate the age of the face image and provide a personalized target age embedding for age progression / regression. The personalized target age embedding is synthesized by incorporating both a personalized residual age embedding of the current age and an exemplar-face aging basis of the target age, where all the prior aging bases are derived from the learned weights of the linear age estimator. This formulation brings an integrated perspective of estimating age and generating personalized aged faces, where a model-estimated age embedding can be learned for each age. Qualitative and quantitative evaluations on different datasets further demonstrate significant improvements in continuous face aging aspects over the state-of-the-art.

[0036] Related Research -Facial aging model Existing approaches can be classified as physical model-based approaches [3, 42, 34] and prototype-based approaches [31, 37, 16, 17]. Physical model-based approaches focus on creating models to address specific sub-effects of aging, such as skin wrinkles [42, 2, 3], craniofacial growth [38, 27], muscle structure [34, 28] and facial components [35, 36]. These approaches are often highly complex and typically require sequences of facial images of the same person at different ages and expert knowledge of the mechanisms of aging. Prototype-based approaches [31, 37, 4] explore the problem of facial progression using group-based learning, where an average face is estimated within each age group. However, personalized aging patterns and identity information are not well preserved in such strategies. In [40, 44, 33], a sparse representation of the input image is used to represent personalized facial transformation patterns. Although such approaches can preserve personalized aging patterns to some extent, the synthesized images have quality issues.

[0037] Recently, deep learning approaches have been adopted to model personalized aging transformations. Wang et al.

[39] proposed a recurrent neural network model that leverages a series of recurrent forward passes for a smoother transition from young to old. Later GAN-based works [18,41,43] have shown impressive breakthroughs in terms of image fidelity. Li et al.

[18] designed three subnets for local patches and fused local and global features to obtain smooth synthetic images. IPCGAN

[41] enforces age estimation loss and identity loss on the generated images to achieve good face aging effects. Further efforts have been made to address the age accuracy and identity persistence. Yang et al.

[43] and Liu et al.

[20] introduce the modification of discriminator losses to derive a more accurate age in the output image. The authors of

[21] improved the quality of the synthetic images by using wavelet packet transformation and multiple facial attribute encoding. However, these methods [41, 43, 20] condition the output image by concatenating one-hot vectors that represent target age groups. To obtain continuous aging conditions, the vectors are expanded to much larger dimensions, which makes training unstable and more complex. Moreover, it requires a huge amount of training images.

[0038] Although several studies [46,1,32] that aim to interpolate features in latent space have provided direction to support continuous aging, they are limited in their ability to generate high-quality images while preserving identity. In

[10] , the authors proposed to linearly interpolate feature vectors from adjacent age groups based on group-based training to achieve continuous age progression. Similarly,

[24] linearly interpolates between two adjacent anchor age embeddings. These methods follow the assumption that embeddings are linearly aligned between anchors, making the determination of anchor ages important. In this work, we present a continuous model-estimated age embedding that does not involve manual effort while achieving better continuous aging modeling.

[0039] -Generative Adversarial Networks Generative adversarial networks [9] have been a popular choice for image-to-image translation tasks. CycleGAN

[47] and Pix2Pix

[14] explored image translation between two domains using unpaired and paired training samples, respectively. More recent works [6, 19] proposed training techniques to enable multi-domain translation. In [22, 23], the authors first explored conditional image generation as an extension of basic GANs. Later works [7, 26] further demonstrated superiority for many conditional image translation tasks by transforming and inserting conditions into the model in a more effective way.

[0040] -Face age estimation The task of predicting apparent age refers to a regression problem of estimating a continuous numerical value for each given face image. Deep Expectation of Apparent Age (DEX)

[30] proposed a method to achieve a mean absolute error (MAE) of 3.25 on MORPH II

[29] by combining classification and regression losses. Pan et al.

[25] proposed to use a mean-variance loss on the probability distribution to further improve the MAE of MORPH II to 2.16.

[0041] <<Embodiment of Model Architecture>> FIG. 1 is a diagram of a model architecture 100 according to one embodiment. The model 100 is shown in a training configuration. In this embodiment, an input image 102 and a real image 104 are received for processing, and a fake image 106 is generated. The input image includes an image of a subject at a real or current age, which may be any age. The age relates to the age of the subject in the image, not the age of the image itself. The real image 104 includes an image of another subject at a target age (e.g., the image is a real image that represents the features of the target domain (specific age)). The target age may be any age relative to the real / current age, including the same age.

[0042] In this embodiment, the age estimator is trained together with the image generator, and E (e.g., 108) is the feature (e i ) 110, and C (e.g., 112) is branched for the age estimation task. C has output β θ 114. The personalized age embedding transform (PAT, Equation 2 116) is based on two components: 1) a residual aging basis at the current age (e.g., a model-estimated age determined by C); and 2) an example of a typical facial aging basis at the target age 118 (e.g., the target age is taken as input). The PAT generates an output γ φ120. In one embodiment, the target age is a particular absolute age (e.g., "29" for 29 years old). In one embodiment, the target age is a delta age or age difference (e.g., a positive or negative integer (e.g., "-5")) relative to the current age in the image or other baseline age.

[0043] In this embodiment, the transformed discriminative coding (e i ,t i ) (eg, at 126) is generated for decoding by G (eg, at 128) to generate the fake image 106 (output).

[0044] The overall model (100) is trained using age losses (e.g., 130, 132, and 134), an identity loss 136, and an adversarial loss 138 via a discriminator D (140). As explained further below, a second encoder E^ 142 and a second age estimator C^ 146 are used (e.g., in the training configuration of the model 100). (Note that "E^" represents the letter "E" with a "^" directly above it, and similarly.)

[0045] 1 illustrates a first group 148 and a second group 150, each depicted in a dashed box. The first grouping 148 highlights encoder components according to one embodiment, while the second grouping 150 highlights decoder or generator side components useful in a training environment, which do not participate in backpropagation during training.

[0046] 1, according to one embodiment, the model comprises four components: 1) a discriminative encoding module E 108, 2) an age estimation module C 112, 3) a personalized age embedding transformation module PAT 116, and 4) an age face generation module G 128. The encoding network E is i(102) is applied to extract identity information from e i =E(x i ) is then used to obtain a probability distribution 114 of ages for the discriminant coding 110. Based on the model estimated age distribution and the target age t 118, a personalized age embedding transformation PAT is applied to the discriminant coding e i Finally, the synthesized face (fake image 106) is transformed into the discriminant coding PAT(e i ;t) (eg, 126 by generator G 128).

[0047] All modules are jointly optimized end-to-end under three objectives according to an embodiment, including the mean-variance age loss

[25] (130, 132 and 134) for accurate aging, the L1 reconstruction loss 136 for identity preservation and the adversarial loss 138 for image realism.

[0048] Unlike many previous face aging tasks [41, 10] that require a pre-trained age classifier to guide the facial aging training, the model 100 directly obtains an embedding of the model-estimated age by utilizing an integrated framework to simultaneously achieve facial aging and age estimation. More preferably, the embedded age estimator 112 not only enables personalized continuous age transformation in a more accurate manner compared to the interpolation-based approach, but also provides guidance for facial image generation (e.g., in G 128).

[0049] -Formulation Discriminative Age Estimation Module (C): In previous studies [41, 10], facial aging and facial age estimation are treated as two independent tasks, where the age estimation models, usually classifiers, are pre-trained separately and then used to guide the generator to achieve a natural aging effect. According to one embodiment of the present specification, since the two above tasks are intrinsically related, one unified structure can be used to achieve both goals by sharing the encoder E.

[0050] In one embodiment of model 100, an age estimator C 112, which includes a global average pooling layer and a fully-connected layer, is branched off from E 108. Finally, θ 114) Age probability distribution p i ∈R K can be obtained by implementing a softmax function, where K denotes the number of age classes. In one embodiment, K = 100. The parameter m i is the distribution p i is determined from.

[0051] The integrated design may provide three advantages. First, it eliminates the need to obtain a well-trained age estimator model in advance. Second, the age estimation on the identity encoding helps the model to establish a more age-specific identity representation. Third, the weights W in the fully connected layer can be used to C is also used as an age embedding base (with the bias term set to zero) that encodes typical face information from a metric learning perspective. Notationally,

number

[0052] Although described as aging simulation or aging effect in the embodiment of Fig. 1, other multi-class domain transfers that apply continuous effects are contemplated. In a general sense, it is a class embedding where the embedding criteria for age is a latent representation of a particular class, such as age=51, or smile degree=40.

[0053] Personalized Age Embedding Transformation (PAT): Facial aging is an inherently challenging and ambiguous task as various facial signs / symptoms correspond to different ages for different people at different stages. Hence, personalization is desired when performing facial aging. In model 100, this personalization is achieved by applying a probability distribution of ages p i,j ∈R K and the basics of typical facial aging i,j ∈R D where i denotes sample i and j∈1,2,···,K denotes age. For any target age t i To obtain a basis for personalized aging for , the process is formulated as the following operations.

number

[0054] this

number

number

number

[13] , E(x i )=e i An affine projection transformation is applied to derive scale and shift coefficients with respect to the original identity that encodes

number

number

[0055] Continuous Aging: Because the aging criterion from the fully connected layer encodes any single age (e.g., in one-year increments, according to one embodiment), continuous aging is naturally supported by choosing an arbitrary target age (as input 118). Some previous group-based approaches can support continuous aging via linear interpolation in the latent space, but the anchor age groups need to be carefully selected.

[0056] However, the techniques and methods of embodiments explicitly model fine-tuned age progression by learning the fundamentals of aging separately for each age (e.g., in classes 1, 2, 3, ..., K, where each class spans a year).

[0057] -Objective According to one embodiment, the design objectives ensure that the synthesized facial image 106 reflects accurate age progression / regression, preserves identity, and appears realistic.

[0058] Mean-variance age loss: The age loss plays two roles in the network: 1) it helps the estimator (C) learn a good aging basis for all ages; and 2) it guides the generator (G) by estimating the ages of the generated fake images. To achieve both goals, according to one embodiment, the mean-variance age loss proposed by

[25] is adopted. Given an input image x i and age label y i Given, the mean variance loss is defined as:

number

number

number

[0059] In addition to being more effective than other losses in the age estimation task, the mean-variance loss also satisfies the desire to learn a relatively focused age distribution while capturing the continuity of age for adjacent age bases, according to one embodiment. According to one embodiment, the supervised age loss is:

number

[0060] To guide facial aging, according to one embodiment, an embedded age estimator 146 is applied both at the transformed discrimination coding level and at the generated image level (shown in FIG. 1 ), according to one embodiment.

number

[0061] When the age estimator C^ 146 and the coder E^ 142 are used on the transformed discriminant coding 126 and the fake image 106, their respective weights are not updated during backpropagation.

[0062] L1 Reconstruction Loss: Another important aspect, according to one embodiment, is to preserve the identity of the individual. An L1 pixel-wise reconstruction loss (e.g., identity loss 136) is applied to the synthesized face by setting the target age to its model estimated age. Specifically, it is formulated as follows:

number

[0063] To enforce the identity criteria, we experimented with cycle-consistency loss, as proposed in StarGAN [6]. We found that a pixel-wise L1 reconstruction loss was sufficient to achieve our goal without extensive efforts to tune hyper-parameters.

[0064] Adversarial Loss: To generate high fidelity images, according to one embodiment, a GAN loss (e.g., 138) is applied in an unconditional adversarial training manner. More specifically, according to an embodiment, a PatchGAN

[14] discriminator (140) is employed, optimized for hinge loss and formulated as follows:

number

number

[0065] In the experiment, we train the classifier using a target age t i It is observed that sampling examples of age equal to or close to helps to stabilize the learning process.

[0066] According to an embodiment, all objectives are jointly optimized with different balancing coefficients as follows:

number

[0067] -experiment Datasets: We evaluated our model on FFHQ

[15] and CACD2000 [5]. FFHQ contains 70000 images with resolution 1024x1024. Following data preprocessing steps as

[24] , images with id 0-68999 were used as training set and images with id 69000-69999 were used for testing. Images were filtered for low reliability in distinguishing gender, low reliability in estimating age, wearing dark glasses, extreme poses and angles based on facial attributes annotated by Face++ (Face Attribute Annotation API: URL www.faceplus.com).

[0068] The annotation from

[24] only contains age group labels, so the age label information was taken from

[45] . To align both age group labels and age labels, images whose age labels do not match the age group labels are further filtered. This results in 12488 male images and 13563 female images for training, and 279 male images and 379 female images for testing. CACD2000 consists of 163446 images with ages ranging from 14 to 62 years, and 10% were randomly taken for evaluation. Face++ was used to separate the images into males and females using Dlib (Dlib toolkit: URL dlib.net), and facial landmarks were extracted.

[0069] Implementation: Because aging patterns differ between males and females, two separate models were trained on the FFHQ dataset for both 256x256 and 512x512 resolutions. The model architecture is modified based on Cycle-GAN

[47] . mv1 and λ mv2 is set to 0.05 and 0.005 in Equation 4. fake1 and λ fake2 are set to 0.4 and 1 in Equation 6. In Equation 10, λ age , λ idt , λ advare set to 0.05, 1 and 1, respectively.

[0070] -Qualitative evaluation Face Age: FFHQ test results are shown in comparison with results from

[24] . Images from

[24] are generated using the provided code (Lifespan official code: URL github.com / royorel / Lifespan_Age_ Transformation_Synthesis). To illustrate model performance across different ages, six input examples A-F from four representative age groups (<30, 30-40, 40-50, >50) are shown, and results are generated for each group and shown in arrays 200, 202, 204, 206, 208, and 210 in Figures 2A, 2B, 2C, 2D, 2E, and 2F. The input image is in the left-most column, and the four target ages proceed to the right. The top row shows Lifespan, and the middle and bottom rows show the output of the model according to an embodiment of this specification, where the bottom row is image size 512x512.

[0071] The target ages of the model (shown in the right column of the input images) are selected as 25, 35, 45 and 55, respectively. As shown in Figures 2A, 2B, 2C, 2D, 2E and 2F, the images generated by the model according to the embodiments herein show fewer artifacts and more obvious aging details such as changes in beard color (examples 200 and 204 in Figures 2A and 2C) and wrinkles in different parts of the face (see examples 202, 204, 206 and 208 in Figures 2B, 2C, 2D and 2E). Convincing details in example 210 in Figure 2F show that personal features (mole 210A) are well preserved using the model.

[0072] Also, in examples 300, 302, 304 and 306 of Figures 3A, 3B, 3C and 3D, images were generated directly on CACD2000 using models trained on FFHQ at a resolution of 256x256 (bottom (fourth) row) for comparison with CAAE

[46] (top (first) row), IPCGAN

[41] (second row) and S2 GAN

[10] (third row). Input images 308, 310, 312 and 314 are boxed. The images demonstrated are examples presented in

[11] , the latest study on CACD2000. For all age groups (11-30, 31-40, 41-50 and >50, left to right columns), the models according to the embodiments herein show more obvious and fine-grained aging effects compared to all previous studies.

[0073] Aging Detail: Images generated using models according to embodiments herein exhibit a significant level of aging detail on different parts of the face (e.g., face crops). The three examples in array 400 in FIG. 4 show three enlarged face crops 402, 404, and 406 (bottom row) from a generated image (middle row), which give a clear and detailed view of wrinkle enhancement, skin smoothness, and changes in beard and eyebrow color. The leftmost example 402 shows a real / target age pair of 36 / 55 (i.e., the input image shows an individual aged 36, and the generated image shows an image at a target age of 55). The center example 404 shows a real / target age pair at 33 / 55, and the rightmost example 406 shows a real / target age pair at 66 / 30.

[0074] Continuous Aging: In Figs. 5A and 5B, image arrays 500 and 502 show some examples of continuous aging results, comparing images generated by the model in the second and fourth rows with images generated by the model of Lifespan

[24] in the first and third rows. In each array, the input image is shown in the leftmost column. Four age stages were selected for presentation (e.g., 21, 25, 29,..., 65 in the left-to-right columns). A gradual and smooth natural aging process (e.g., changes in wrinkle depth, beard and facial pigmentation) can be observed from the images generated by the model according to one embodiment, while preserving personal characteristics. However, the interpolation-based approach in Lifespan lacks the ability to generate images of well-aligned target ages and does not preserve certain personalized information. Furthermore, the model according to one embodiment produces more realistic aging effects with minimal artifacts.

[0075] -Quantitative evaluation Identity Preservation: A face verification rate metric was used to evaluate identity preservation. Specifically, the evaluation protocol of

[10] was followed on an age group basis for a fair comparison with previous studies. Face verification rates were calculated for all combinations of image pairs, i.e., (test, 10-29), (test, 30-39), ..., (30-39, 40-49), (40-49, 50-59). A face verification score was obtained from Face++ and the threshold was set as 76.5 (@FAR=1e-5). The complete results are shown in Tables 1 and 2 for CACD2000 and FFHQ, respectively. As the results suggest, the model according to one embodiment achieves the highest face verification rate for both datasets among all candidates, indicating that it best meets the identity preservation requirement of the task. [Table 1]

[0076] Aging accuracy: To evaluate aging accuracy, an unbiased age estimator was used to infer the aging of the generated images. To compare with previous group-based methods based on CACD2000, images were generated to match the age group settings of CACD2000. The size of the age group was adaptively increased or decreased by a factor of 10 from the actual age of the input image as the target age for generation, i.e., a target age of 33 was used to generate images with age group 30-40 if the current age was 23. To perform a direct comparison, without access to the evaluation age estimator of

[10] or its pre-trained model to evaluate the model according to the embodiment of this specification, the aging estimation results of Face++ were used on the model according to the embodiment of this specification and one of the accessible prior studies IPCGAN

[41] , which was also evaluated in

[10] to show a relative comparison. The evaluation of FFHQ follows the same procedure as CACD2000. The evaluation results are shown in Tables 3 and 4 for CACD2000 and FFHQ, respectively. The results suggest that the models according to the embodiments evaluated herein using Face++ have more plausible mean ages in each age group than IPCGAN

[41] and Lifespan on CACD2000

[24] , and perform similarly to Lifespan on FFHQ. [Table 2]

[0077] Image Fidelity: For image fidelity, the models according to the embodiments herein were evaluated on both datasets using the Frechet Inception Distance (FID)

[12] metric. As in the previous image generation setting, FID scores were calculated for generated images corresponding to the same age groups as those on CACD2000. To compare with FFHQ's

[24] , FID scores were calculated for generated images sharing the same age group range. The results are shown in Table 5 (FID rating: lower is better). On both datasets, the models according to the embodiments herein achieved the lowest FID scores, which quantitatively demonstrates the superiority in the image quality aspect. [Table 3]

[0078] -Model interpretability and ablation testing Continuous Aging: To evaluate how well models according to embodiments herein generate synthetic images in a continuous setting, an age estimator was used to predict the age of each of the generated pseudo-images of 1) models aged 25 to 65 years, and 2) a linear interpolation approach performed between anchor aging criteria. The anchor basis was generated by taking the average of all aging criteria within an age group. The age step was selected as 3 based on the MAE of the estimator.

[0079] Using age estimators jointly trained on the FFHQ dataset, we computed confusion matrices for the age accuracy for each approach. Confusion matrices 600 and 602 in Figures 6A and 6B, respectively, show that the fake images generated according to our model exhibit a more obvious continuous aging trend with much higher aging accuracy than the fake images generated by the linear interpolation approach.

[0080] Interpolation between two identities in latent space: In the image array 700 of FIG. 7, it is further shown that the model according to the embodiment herein also learns disentangled representations of age and identity in latent space. FIG. 7 shows linear interpolation between transformed identity codings where the real image is in the box (leftmost and rightmost columns) in three examples of pairing. From left to right, we performed linear interpolation between the transformed identity codings of the two images with the same target age of 65. Images for interpolation coding were generated. As shown, the identities change gradually while maintaining their respective ages. Personal features such as eye color and tooth shape change smoothly from person to person.

[0081] Use of residual embedding: A feature of the model architecture according to one embodiment is the formulation of a personalized aging embedding that incorporates both the personalized aging features of individuals and the aging effects shared among the entire population. To better illustrate and understand the effectiveness of the design, we run the comparator model without adding a residual embedding (i.e., using the aging basis of typical faces at the target age a i,j =t i ) and compared to a model according to an embodiment where residual embedding was added. Image array 800 in FIG. 8 displays two examples comparing results without residual embedding (first and third rows) and with residual embedding (second and fourth rows). The input image is in the leftmost column, and the age ranges 11-30, 31-40, 41-50, and >50 correspond to columns from left to right. In both examples, a tendency towards more unnatural artifacts and typical facial modifications is observed in the images generated without residual embedding.

[0082] "application" In one embodiment, the disclosed techniques and methods include a developer-related method and system for defining (e.g., through conditioning / training) a model having a generator and age estimator for image-to-image transformations that provide age simulation. The generator exhibits continuous control using multiple model-estimated age embeddings across multiple successive ages learned by conditioning to generate smooth transformations between the original image (input image) and the transformed image (new image). In one embodiment, the image is a face (e.g., a face). In one embodiment, a personalized age embedding (determined from multiple model-estimated age embeddings using a target age and the model-estimated ages of the original image) is used to transform encoder-generated features from the encoder components of the model.

[0083] In one embodiment, a model according to embodiments herein for image-to-image transformation is incorporated into a computer-implemented method (e.g., an application) or computing device or system to provide a virtual reality, augmented reality, and / or modified reality experience. The application is configured to facilitate a user taking a selfie image (or video) using, for example, a camera-equipped smartphone or tablet device, and a generator G applies a desired effect, such as for playback or other presentation by the smartphone or tablet device.

[0084] In one embodiment, the generator G taught herein is configured to load and run on a publicly available consumer smartphone or tablet (e.g., target device). An example configuration includes a device with the following hardware specifications: Intel® Xeon® CPU E5-2686 v4 @ 2.30GHz, configured with 1 core and 1 thread profile only. In one embodiment, the generator G is configured to load and run on a computing device with more resources, including a server, desktop, gaming computer, or other device with multiple cores and running on multiple threads. In one embodiment, the generator G is provided as a (cloud-based) service.

[0085] Those skilled in the art will appreciate that in one embodiment, in addition to aspects of developer (e.g., used at training time) and target (used at inference time) computing devices, aspects of a computer program product are disclosed in which instructions are stored on a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.) to configure the computing devices to perform any of the aspects of the methods disclosed herein.

[0086] 9 is a block diagram of a computer system 900 according to one embodiment. The computer system 900 comprises a number of computing devices (902, 904, 906, 908, 910, and 950) including a server, developer computers (PCs, laptops, etc.), and typical user computers (PCs, laptops, and smaller form factor (personal) mobile devices such as smartphones and tablets, etc.). In an embodiment, the computing device 902 provides a network model training environment 912 comprising hardware and software to define a model for image-to-image transformation providing continuous aging in accordance with the teachings herein. Components of the network model training environment 912 include a model trainer component 914 for defining and configuring, such as by conditioning, a model including E 108, C 112, PAT 116, G 128, E^ 142, C^ 146, and D 140. Components 140, 142, and 146 are configurations for training, but are not used as run-time components, such as for generating new images in run-time (inference time) applications.

[0087] In an embodiment, the conditioning is performed according to the model network architecture 100 of Figure 1 or the like. In this embodiment, a data server (e.g., 904) or other form of computing device stores an image data set 926 of images, such as for training and other purposes, and is coupled through one or more networks, representatively shown as network 928, which couples any of the computing devices 902, 904, 906, 908, and 910. The network 928 is, by way of example, wireless or other, public or other. It will also be understood that the system 900 is simplified. At least any of the services may be implemented by more than one computing device.

[0088] Once trained, the trained model 100 may be further defined as desired to include a runtime component and provided as a trained model 930. In accordance with the techniques and methods herein, in an embodiment, the trained model 930 is made available for use in a variety of ways. In one method, in one embodiment as shown in FIG. 9, the trained model 930 is provided as a cloud service 932 or other software as a service (SaaS) via a cloud server 908. A user application, such as an augmented reality (AR) application 934, is defined for use with the cloud service 932 that provides an interface to the trained model 930. In one embodiment, the AR application 934 is provided for distribution (e.g., via download) from an application distribution service 936 provided by the server 906.

[0089] Although not shown, in one embodiment, the AR application 934 is developed using an application developer computing device for a particular target device having particular hardware and software, particularly an operating system configuration, etc. In one embodiment, the AR application 934 is a native application configured for execution in a particular native environment, such as one defined for a particular operating system (and / or hardware). Native applications are often distributed via an application distribution service 936, which is configured as an e-commerce "store" operated by a third-party service, although this is not required. In one embodiment, the AR application 920 is, for example, a browser-based application configured to run in a browser environment of the target user device.

[0090] The AR application 934 is provided for delivery (e.g., download) by a user device, such as the mobile device 910. In one embodiment, the AR application 934 is configured to provide an augmented reality experience (e.g., via an interface) to a user. For example, an effect is applied to an image by processing by the estimated time generator 930. The mobile device has a camera (not shown) for capturing an image (e.g., a photographed image 938), which in one embodiment is a still image including a selfie image. Effects are applied to the photographed image 938 using image processing techniques that provide image-to-image transformation. An age simulation (aged) image (new image) 940 is defined and displayed on a display device (not shown) of the mobile device 910 to simulate the effect on the photographed image 938. The position of the camera may be further altered in response to the photographed image(s) and effects applied to simulate an augmented reality. It will be appreciated that the photographed image defines a source, input or original image, and the aged image defines a new, transformed or transformed image or image to which the effect is applied.

[0091] In the cloud service paradigm of the embodiment of Figure 9, captured images 938 are provided to a cloud service 932 where they are processed by a trained model 930 to perform an image-to-image transformation with successive aging to define an aged image 940. The aged image 940 is communicated to a mobile device 910 for display, storage, sharing, etc.

[0092] In one embodiment, the AR application 934 provides an interface (not shown) for operating the AR application 934, e.g., a graphical user interface (GUI), which may be voice-enabled. The interface is configured to enable image capture, communication with a cloud service, and display, save, and / or share of the converted image (e.g., aged image 940). In one embodiment, the interface is configured for a user to provide input for the cloud service, such as defining a target age. In one embodiment, the input includes an age delta. As previously discussed, in one embodiment, the input includes a product / service selection. For example, the product / service selection is associated with an age delta for rejuvenating the input image. In one example, the input may be lifestyle factors, such as smoking rates, sun exposure rates, or other factors that contribute to the appearance of premature aging. The lifestyle factors may be associated with an age delta for applying to the input image.

[0093] In the embodiment of FIG. 9, the AR application 934 or another application (not shown) provides access (e.g., via a communication interface) to a computing device 950 that provides an e-commerce service 952. The e-commerce service 952 comprises a recommendation component 954 for providing (personalized) recommendations for products, services, or both. In an embodiment, such products and / or services are rejuvenating or anti-aging products and / or services, etc. In an embodiment, such products and / or services are related to, for example, specific skin signs. In an embodiment, a captured image from the device 910 is provided to the e-commerce service 952. Analysis of the skin signs is performed by a skin sign analyzer model 956, etc., using deep learning according to an embodiment. Image processing using the trained model analyzes the skin (e.g., facial zones related to specific skin signs) to generate a skin analysis including a score for at least some of the skin signs. Individual score values ​​can be generated on the images, for example, using a (dedicated) aging signature estimation model (e.g., a type of classifier) ​​based on the ResNet

[27] architecture as previously described to analyze the training set data.

[0094] In an embodiment, the skin indications (e.g., their scores) are used to generate personalized recommendations. For example, each product (or service) is associated with one or more skin indications and a particular score (or range of scores) of such indications. In this embodiment, the information is stored in a database (e.g., 960) for use by the e-commerce service 952, such as through appropriate look-ups that match user data to product and / or service data. In one embodiment, further user data used by the recommendation component 954 includes any of gender, ethnicity, and location data, etc.

[0095] In this embodiment, a score of the skin indication of the user's captured image is provided by the e-commerce service for display via an AR application 934, such as an AR application interface. For example, in one embodiment, the score is generated or modified using other means, such as by application of a rule or other code.

[0096] In one embodiment, an annotated image is provided from a user's captured image (i.e., an input image), where, for example, the annotated image includes the input image annotated with either a skin indication score, a description / information regarding the skin indication associated with such score, product information associated with such score, or service information associated with such score.

[0097] In one non-limiting embodiment, a user receives personalized product recommendations as recommended by e-commerce service 952. The user selects a particular product or service. The selected product or service is associated with its age delta (e.g., a rule that may be determined based on the subject's chronological age, length of product use, other demographic or geographic data, etc.) and invokes modification of the input image. The modification, for example, simulates the subject's age in the input image to generate a new image with a new target age. The input image and the target age determined from the product or service selection may be provided to cloud service 932 to receive an aged image (e.g., an instance of 940).

[0098] 9 embodiment, the e-commerce service 952 is configured with a purchase component 958 to facilitate the purchase of a product or service. The product or service may include cosmetics or services, etc. Although not shown, the e-commerce service 952 and / or the AR application 934 may provide image processing of the captured image to simulate the cosmetics or services, such as applying makeup to the captured image to generate an image to which the effect is applied.

[0099] While the photographed image is used as the source image for processing in the above embodiment, in one embodiment, other source images (e.g., from a source other than the camera of the device 910) are used. One embodiment may use photographed images or other source images. In one embodiment, such images, whether photographed or another image, are high resolution images to improve the user experience for which the trained model 930 is trained. Although not shown, in this embodiment, the images used by the skin signature analyzer model 956 are downscaled when analyzed. For such analysis, other image pre-processing is performed.

[0100] In one embodiment, the AR application 934 can prompt the user regarding quality features (i.e., lighting, centering, background, hair occlusion, etc.) to improve performance. In one embodiment, the AR application 934 rejects an image if it does not meet certain minimum requirements and is unsuitable.

[0101] 9 as a mobile device, in one embodiment computing device 910 may have different form factors, as noted above. Rather than (or in addition to) providing the trained model 930 as a cloud service, it may be hosted and executed locally on a particular computing device that has sufficient storage and processing resources.

[0102] Accordingly, in one embodiment, the AR application 934 is configured to provide an interface for receiving an input image, communicate the input image to a recommendation service to receive a skin analysis, provide recommendations including at least one recommended product or service responsive to the analysis, display the skin analysis, provide an interface for displaying the at least one recommendation, provide an interface for selecting a product or service from the recommendations, and responsive to the selection, generate an age simulation image using a target age and the input image associated with the selected product or service, present it via the interface, and provide an interface for purchasing the product or service via the e-commerce service.

[0103] In one embodiment, the AR application communicates the recommendations, which are provided without performing a skin analysis, e.g., based on a set of user preferences, e.g., selecting a recommendation for a facial area or for one or more specific skin indications, is provided.

[0104] In one embodiment, for example, if the second target age does not refer to the use of the recommended product, the AR application 934 generates a second age-simulation image at the second target age. In one embodiment, two age-simulation images are presented simultaneously for comparison. Effects such as makeup and hair effects can be applied to any of the age-simulation images.

[0105] In one embodiment, the computing device includes a camera and the processing unit receives the original image from the camera.

[0106] In one embodiment, the product comprises one of a rejuvenating product, an anti-aging product, and a cosmetic makeup product. In one embodiment, the service comprises one of a rejuvenating service, an anti-aging service, and a cosmetic service.

[0107] In one embodiment, a computing device, such as the mobile device 910, is configured to perform methods according to the computing device aspects described above. Other aspects, such as computer program product aspects, will be apparent.

[0108] In one embodiment, a computing device configured to perform a method, such as a method in which a network model training environment is configured by tuning an age simulation generator (based on a GAN), is provided.

[0109] In one embodiment, a computing device is provided that includes a facial effects unit including processing circuitry configured to apply at least one continuous effect to a source image and generate one or more virtual instances of the image with the effect applied on an e-commerce interface, the facial effects unit utilizing an encoder and estimator with a generator to simulate the applied continuous effect (e.g., aging), the applied continuous effect having continuous control for each class of effect (e.g., age across an age range, degree of smile across a smile range, etc.).

[0110] In one embodiment, a computing device presents product and / or service recommendations and comprises a recommendation unit including a processing circuit configured to receive a selection of a product and / or service, the product and / or service being associated with a target age (e.g., a modifier such as a delta to a current age or an absolute age number). The facial effect unit is configured to generate an image with a target age effect applied in response to the selection, thereby simulating the effect of the product and / or service on the source image. In one embodiment, the recommendation unit is configured to obtain the recommendation by invoking a skin indication analyzer to determine a current skin indication score using the source image and using the current skin indication score to determine the product and / or service. In one embodiment, the skin indication analyzer is configured to analyze the source image using a deep learning model. In one embodiment, the target age is defined from an aging target modifier associated with the product / service.

[0111] In addition to the age-related embodiment, the network structure, method and technique proposed herein can also be applied to other multi-class domain transfer tasks to avoid group-based training and achieve more accurate continuous modeling. It will be understood that a domain transfer task involves transforming a source image from one domain to another, such as when an effect is applied. "Multi-class" refers here to various degrees or granularity of progression for continuous effects. In the age example of the continuous effect, a class K was discussed. For the smile-related continuous effect, the class can represent the degree of smile, e.g., at 1% granularity as in one embodiment. In the smile example, the age estimator C and its training component C^ are adapted (e.g., via training) as an estimator of smile estimation (e.g., to predict the degree of smile rather than age). The estimator is useful to determine model-estimated class embeddings in each of the continuous granularity ranges (classes) of the continuous effect representing the continuous effect information.

[0112] Other multi-class effects (e.g., domain shifts) may be contemplated, including non-facial effects (e.g., hair loss, weight gain, etc.). Thus, the generator can generate a continuous effect image in a target class from the composite coder and estimator, where the target is one of the classes (i.e., granularity range) of effects.

[0113] Thus, in one embodiment, a method is provided that includes providing a unified model for generating new images in a target class of continuous effects for a subject from an input image of the subject. The model provides a plurality of respective model-estimated class embeddings in each of a plurality of continuous classes representing continuous effect information. The model-estimated class embeddings are learned by joint training of a generator and an estimator embedded in an encoder-decoder architecture of the model. The estimator is configured to determine a model-estimated class for each subject using respective features generated by the encoder in response to the respective input images. The generator generates the new images using features generated from the input images transformed by each of the model-estimated class embeddings determined in accordance with the target classes.

[0114] In one embodiment, the continuous effect is an aging effect and the target range is a particular one of the age classes (eg, one of the degrees of the continuous effect), for example, an integer year.

[0115] 《Conclusion》 In this study, we present a new approach to the facial aging challenge, with a particular focus on the continuous aging aspect. By introducing an age estimation module into the GAN-based generator, a unified framework is proposed for learning continuous aging criteria. The designed PAT module further enhances the personalization of typical facial aging criteria, which overall results in more natural and realistic generated face images. Experiments qualitatively and quantitatively show superior performance in terms of aging accuracy, identity preservation and image fidelity on two datasets compared to previous studies. Moreover, the proposed network structure can also be applied to other multi-class domain transfer tasks to avoid group-based training and achieve more accurate continuous modeling. As mentioned before, one example is the smile effect applied to faces. Instead of an age estimator, the estimator of the continuous effect (e.g., C112) includes a degree of smile effect estimator.

[0116] A practical implementation may include any or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, and other ways of combining the features described herein. Some embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided or steps may be eliminated from the described processes, and other components may be added or removed from the described systems. Accordingly, other aspects are within the scope of the claims.

[0117] Throughout the description and claims of this specification, the words "comprise" and "contain" and variations thereof mean "including, but not limited to," and are not intended to exclude other elements, integers, or steps. Throughout this specification, the singular encompasses the plural unless the context requires otherwise. In particular, when the indefinite article is used, it is to be understood that the specification contemplates the plural as well as the singular, unless the context requires otherwise.

[0118] It should be understood that features, integers, properties or groups described in connection with a particular aspect, embodiment or example of the invention are applicable to any other aspect, embodiment or example, except where they are incompatible therewith. All of the features disclosed herein (including any accompanying claims, abstract and drawings), and / or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of any of the foregoing examples or embodiments. The invention extends to any novel one or any novel combination of features disclosed herein (including any accompanying claims, abstract and drawings), or any novel one or any novel combination of steps of any method or process disclosed. 《References》 The following publications are incorporated herein by reference where permitted: [1] Grigory Antipov, Moez Baccouche, and Jean-Luc Dugelay. Face aging with conditional generative adversarial networks. In 2017 IEEE international conference on image processing (ICIP), pages 2089-2093. IEEE, 2017. [2] Yosuke Bando, Takaaki Kuratate, and Tomoyuki Nishita. A simple method for modeling wrinkles on human skin. In Pacific Conference on Computer Graphics and Applications, pages 166-175. Citeseer, 2002. [3] Laurence Boissieux, Gergo Kiss, Nadia Magnenat Thalmann, and Prem Kalra. Simulation of skin aging and wrinkles with cosmetics insight. In Computer Animation and Simulation 2000, pages 15-27. Springer, 2000. [4] D Michael Burt and David I Perrett. Perception of age in adult caucasian male faces: Computer graphic manipulation of shape and colour information. Proceedings of the Royal Society of London. Series B: Biological Sciences, 259(1355):137-143, 1995. [5] Bor-Chun Chen, Chu-Song Chen, and Winston H Hsu. Cross-age reference coding for age-invariant face recognition and retrieval. In European conference on computer vision, pages 768-783. Springer, 2014. [6] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8789-8797, 2018. [7] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 8188-8197, 2020. [8] Harm de Vries, Florian Strub, Jeremie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6594-6604. Curran Associates, Inc., 2017. [9] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Advances in neural information processing systems, 3(06), 2014.

[10] Zhenliang He, Meina Kan, Shiguang Shan, and Xilin Chen. S2gan: Share aging factors across ages and share aging trends among individuals. In Proceedings of the IEEE International Conference on Computer Vision, pages 9440-9449, 2019.

[11] Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen. Attgan: Facial attribute editing by only changing what you want. IEEE Transactions on Image Processing, 28(11):5464-5478, 2019.

[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, pages 6626-6637, 2017.

[13] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017.

[14] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. CVPR, 2017.

[15] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401-4410, 2019.

[16] Ira Kemelmacher-Shlizerman, Supasorn Suwajanakorn, and Steven M Seitz. Illumination-aware age progression. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3334-3341, 2014.

[17] Andreas Lanitis, Christopher J. Taylor, and Timothy F. Cootes. Toward automatic simulation of aging effects on face images. IEEE Transactions on pattern Analysis and machine Intelligence, 24(4):442-455, 2002.

[18] Peipei Li, Yibo Hu, Qi Li, Ran He, and Zhenan Sun. Global and local consistent age generative adversarial networks. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 1073-1078. IEEE, 2018.

[19] Ming Liu, Yukang Ding, Min Xia, Xiao Liu, Errui Ding, Wangmeng Zuo, and Shilei Wen. Stgan: A unified selective transfer network for arbitrary image attribute editing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3673-3682, 2019.

[20] Si Liu, Yao Sun, Defa Zhu, Renda Bao, Wei Wang, Xiangbo Shu, and Shuicheng Yan. Face aging with contextual generative adversarial nets. In Proceedings of the 25th ACM international conference on Multimedia, pages 82-90, 2017.

[21] Yunfan Liu, Qi Li, and Zhenan Sun. Attribute-aware face aging with wavelet-based generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11877-11886, 2019.

[22] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.

[23] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning, pages 2642-2651, 2017.

[24] Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, and Ira Kemelmacher-Shlizerman. Lifespan age transformation synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.

[25] Hongyu Pan, Hu Han, Shiguang Shan, and Xilin Chen. Mean-variance loss for deep age estimation from a face. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5285-5294, 2018.

[26] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Gaugan: semantic image synthesis with spatially adaptive normalization. In ACM SIGGRAPH 2019 Real-Time Live! 2019.

[27] Narayanan Ramanathan and Rama Chellappa. Modeling age progression in young faces. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), volume 1, pages 387-394. IEEE, 2006.

[28] Narayanan Ramanathan and Rama Chellappa. Modeling shape and textural variations in aging faces. In 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition, pages 1-8. IEEE, 2008.

[29] Karl Ricanek and Tamirat Tesafaye. Morph: A longitudinal image database of normal adult age-progression. In 7th International Conference on Automatic Face and Gesture Recognition (FGR06), pages 341-345. IEEE, 2006.

[30] Rasmus Rothe, Radu Timofte, and Luc Van Gool. Dex: Deep expectation of apparent age from a single image. In Proceedings of the IEEE international conference on computer vision workshops, pages 10-15, 2015.

[31] Duncan A Rowland and David I Perrett. Manipulating facial appearance through shape and color. IEEE computer graphics and applications, 15(5):70-76, 1995.

[32] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 9243-9252, 2020.

[33] Xiangbo Shu, Jinhui Tang, Hanjiang Lai, Luoqi Liu, and Shuicheng Yan. Personalized age progression with aging dictionary. In Proceedings of the IEEE international conference on computer vision, pages 3970-3978, 2015.

[34] Jinli Suo, Xilin Chen, Shiguang Shan, Wen Gao, and Qionghai Dai. A concatenational graph evolution aging model. IEEE transactions on pattern analysis and machine intelligence, 34(11):2083-2096, 2012.

[35] Jinli Suo, Feng Min, Songchun Zhu, Shiguang Shan, and Xilin Chen. A multi-resolution dynamic model for face aging simulation. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8. IEEE, 2007.

[36] Jinli Suo, Song-Chun Zhu, Shiguang Shan, and Xilin Chen. A compositional and dynamic model for face aging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(3):385-401, 2009.

[37] Bernard Tiddeman, Michael Burt, and David Perrett. Prototyping and transforming facial textures for perception research. IEEE computer graphics and applications, 21(5):42-50, 2001.

[38] James T Todd, Leonard S Mark, Robert E Shaw, and John B Pittenger. The perception of human growth. Scientific American, 242(2):132-145, 1980.

[39] Wei Wang, Zhen Cui, Yan Yan, Jiashi Feng, Shuicheng Yan, Xiangbo Shu, and Nicu Sebe. Recurrent face aging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2378-2386, 2016.

[40] Wei Wang, Yan Yan, Stefan Winkler, and Nicu Sebe. Category specific dictionary learning for attribute specific feature selection. IEEE Transactions on Image Processing, 25(3):1465-1478, 2016.

[41] Zongwei Wang, Xu Tang, Weixin Luo, and Shenghua Gao. Face aging with identity-preserved conditional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7939-7947, 2018.

[42] Yin Wu, Nadia Magnenat Thalmann, and Daniel Thalmann. A plastic-visco-elastic model for wrinkles in facial animation and skin aging. In Fundamentals of Computer Graphics, pages 201-213. World Scientific, 1994.

[43] Hongyu Yang, Di Huang, Yunhong Wang, and Anil K Jain. Learning face age progression: A pyramid architecture of gans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 31-39, 2018.

[44] Hongyu Yang, Di Huang, Yunhong Wang, Heng Wang, and Yuanyan Tang. Face aging effect simulation using hidden factor analysis joint sparse representation. IEEE Transactions on Image Processing, 25(6):2493-2507, 2016.

[45] Xu Yao, Gilles Puy, Alasdair Newson, Yann Gousseau, and Pierre Hellier. High resolution face age editing. arXiv preprint arXiv:2005.04410, 2020.

[46] Zhifei Zhang, Yang Song, and Hairong Qi. Age progression / regression by conditional adversarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5810-5818, 2017.

[47] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232,2017. <Other> <Means> The method of technical idea 1 includes providing an integrated age simulation model for generating a new image of a subject at a target age from an input image of the subject, the integrated age simulation model providing a plurality of respective model-estimated age embeddings at each of a plurality of continuous ages representing continuous aging information, the model-estimated age embeddings being learned through joint training of a generator, an age estimator being embedded in an encoder-decoder architecture of the model, the age estimator being configured to determine a model-estimated age of the subject from features generated by the respective encoders in response to each input image, and the generator generating the new image using the features generated by the generator from the input image transformed by each of the model-estimated age embeddings determined according to the target age and the model-estimated age of the subject. The method of technical idea 2 is the method according to technical idea 1, wherein the encoder-decoder architecture comprises an age estimator for estimating a model-estimated age of the subject in the input image. The method of technical idea 3 is the method described in technical idea 2, in which an encoder of the model processes the input image to determine features generated by the encoder, and the age estimator processes the features generated by the encoder to determine the age estimated by the model. The method of technical idea 4 is the method described in technical idea 2, in which the features generated by the encoder are transformed by personalized age embeddings including: a. each of the model-estimated age embeddings determined according to the model-estimated age; and b. each of the model-estimated age embeddings determined according to the target age. The method of technical idea 5 is the method described in technical idea 4, wherein the personalized age embedding includes: a. a personalized remaining age embedding determined from a plurality of respective model-estimated age embeddings according to the model-estimated age to retain identity information of the subject; and b. a typical age embedding including each one of the model-estimated age embeddings determined according to the target age to represent a shared aging pattern among the entire population. The method of technical idea 6 is the method of technical idea 4 or 5, wherein the embedding of the personalized age is applied according to an affine transformation. A method of technical idea 7 is a method according to any one of technical ideas 4 to 6, in which the generator generates the new image at the target age by processing the encoded features transformed by the personalized age embedding. A method of technical idea 8 is a method according to any one of technical ideas 1 to 7, in which the model is composed of one or both of a deep learning neural network model and a generative adversarial network-based model. The method of technical idea 9 provides a recommendation interface for obtaining recommendations for one or both of products and services in the method of any one of technical ideas 1 to 8. The method of technical idea 10 provides an electronic commerce purchasing interface for purchasing one or both of a product and a service in the method of any one of technical ideas 1 to 9. The method of technical idea 11 is a method according to any one of technical ideas 1 to 10, further comprising receiving the input image and providing the new image for display. A method according to technical idea 12 is a method according to any one of technical ideas 1 to 11, wherein the input image and the new image each include a face of the subject. The method of technical idea 13 provides an integrated model for generating new images in a target class of continuous effects for a subject from an input image of the subject, the model providing a plurality of respective model estimated class embeddings for each of a plurality of continuous classes representing continuous effect information, the model estimated class embeddings being learned through joint training of a generator, a class estimator being embedded in an encoder-decoder architecture of the model, the class estimator being configured to determine a model estimated class for each subject from features generated by the respective encoders in response to each input image, and the generator generating the new images using the model estimated class embeddings determined in response to the target class and features generated by the encoders from the input images transformed by each of the model estimated classes of the subject. The method of technical idea 14 provides a domain transfer model for transferring an input image to a new image, applies a continuous effect to transform the input image to a target class of a plurality of continuous classes of the continuous effect using a plurality of respective model-estimated class embeddings learned by the model for each of the classes, and transfers the input image to the new image using the domain transfer model. The method of technical idea 15 is the method of technical idea 14, wherein the progression effect includes an aging effect, each of the plurality of model-estimated class embeddings includes an embedding of a respective model-estimated age, and the target class includes a target age. The method of Technical Idea 16 is the method of Technical Idea 15, in which, when transferring the input image, the domain transfer model generates encoded features of the input image, transforms the encoded features using typical age embeddings including a personalized remaining age embedding determined from a plurality of respective model-estimated age embeddings according to model-estimated ages of subjects in the input image, and each one of the model-estimated remaining age embeddings determined according to the target age to represent an aging pattern shared among an entire population, and generates the new image using the transformed encoded features. The method of technical idea 17 is the method of technical idea 16, wherein the model includes an age estimator that determines the model-estimated age. The method of technical idea 18 is the method of technical idea 17, wherein the age estimator comprises a classifier trained together with an encoder, the encoder being configured to generate the coded features, and the age estimator is trained to determine a model estimated age for each of the subjects in a novel image using each coded feature coded by the encoder. The method of technical idea 19 is the method of technical idea 18, wherein the model-estimated age embeddings are defined during the training of the age estimator together with the encoder, and each of the model-estimated age embeddings is associated with a respective model-estimated age. The method of technical idea 20 is a method according to any one of technical ideas 14 to 19, further comprising providing a recommendation for at least one of a product and a service associated with the continuous effect. The method of technical idea 21 is the method of technical idea 20, wherein the recommendation is generated in response to one or both of a skin analysis of the input image and a user input of preferences. The method of technical concept 22 is the method of technical concept 20 or 21, in which the target age is determined according to the recommendation. The method of technical idea 23 is the method of any one of technical ideas 20 to 22, further comprising providing an annotated image generated from the input image for presenting the recommendation. The method of technical concept 24 provides an electronic commerce interface for purchasing products, services, or both in the method of any of technical concepts 20 to 23. The method of technical idea 25 is a method according to any one of technical ideas 20 to 24, wherein the continuous effect is an anti-aging effect, the product includes one of a rejuvenating product, an anti-aging product, and a cosmetic makeup product, and the service includes one of a rejuvenating service, an anti-aging service, and a cosmetic service. A computing device of technical idea 26 comprises a processing unit and a storage device coupled thereto, the storage unit storing instructions that, when executed by the processing unit, configure the computing device to perform any of the methods described in technical ideas 1 to 25. The computer program product of technical idea 27 includes a non-transitory storage device storing instructions that, when executed on a processing unit of a computing device, configure the computing device to perform any of the methods described in technical ideas 1 to 25. The computing device of technical idea 28 comprises a processing unit and a storage device coupled thereto, the storage unit being configured to store instructions that, when executed by the processing unit, cause the computing device to: a. recommend at least one of a product and a service; and b. provide an age simulation image including a new image generated from an input image and a target age, the new image being generated according to any of the methods described in technical ideas 1 to 25.

Claims

1. 1. A method comprising: providing an integrated age simulation model for generating, from an input image of a subject, a new image of the subject at a target age, the method comprising: the integrated age simulation model provides embeddings of a plurality of respective model-estimated ages at each of a plurality of continuums of ages representing continuous aging information; The model-estimated age embedding is learned through joint training of a generator and an age estimator is embedded into the encoder-decoder architecture of the joint age simulation model; the age estimator is configured to determine, in response to each input image, a model estimated age of the subject from features produced by each encoder; The method, characterized in that the generator generates the new image from the input image transformed by each of the embeddings of model-estimated ages determined according to the target age and a model-estimated age of the subject, using features generated by the generator.

2. 2. The method of claim 1, wherein the encoder-decoder architecture comprises an age estimator for estimating a model-estimated age of the subject in the input image.

3. 3. The method of claim 2, wherein an encoder of the integrated age simulation model processes the input image to determine features generated by the encoder, and wherein the age estimator processes the features generated by the encoder to determine an age estimated by the integrated age simulation model.

4. The features generated by the encoder are a. each of said model-estimated age embeddings determined according to said model-estimated age; and b. each of the model-estimated age embeddings determined as a function of the target age.

5. The personalized age embedding may include: a. a personalized residual age embedding determined from a plurality of respective model-estimated age embeddings as a function of the model-estimated age to preserve identity information of the subject; and b. a typical age embedding comprising each one of said model-estimated age embeddings determined according to said target age to represent a shared aging pattern among an entire population.

6. The method of claim 4 or 5, characterized in that the personalized age embedding is applied according to an affine transformation.

7. A method according to any one of claims 4 to 6, characterized in that the generator generates the new image at the target age by processing features generated by the encoder transformed by the personalized age embedding.

8. A method according to any one of claims 1 to 7, characterized in that the integrated age simulation model is composed of one or both of a deep learning neural network model and a generative adversarial network-based model.

9. providing a recommendation interface for obtaining product and / or service recommendations; providing an e-commerce purchasing interface for purchasing products and / or services; 9. A method according to claim 1, comprising one or more of: receiving the input image; providing the new image for display; and each of the input image and the new image including a face of the subject.

10. 1. A method for providing an integrated model for generating new images in a target class of continuous effects for a subject from an input image of the subject, the method comprising: the model provides a plurality of respective model estimated class embeddings in each of a plurality of continuous classes representing continuous effect information; The model estimated class embeddings are learned through joint training of generators; The class estimator is embedded in the encoder-decoder architecture of the model; the class estimator is configured to determine a model estimated class of each subject from features generated by each encoder in response to each input image; The method, characterized in that the generator generates the new image using features generated by the encoder from the input image transformed by each of the model estimated class embeddings determined according to the target class and the model estimated class of the subject.

11. A computing device comprising a processing unit and a storage device coupled thereto, A computing device, characterized in that the storage device stores instructions which, when executed by the processing unit, configure the computing device to perform any of the methods of claims 1 to 10.

12. 11. A computer program product comprising: a non-transitory storage device storing instructions that, when executed on a processing unit of a computing device, configure the computing device to perform any of the methods of claims 1 to 10.

Images