Continuous effects apply via model-estimated class embeddings

Abstract

Methods, devices, and techniques are provided for processing images using deep learning models to enable continuous effect simulation through a unified network in which simple (effect class) estimators are embedded into a regular encoder-decoder architecture. The estimators allow learning of model-estimated class embeddings for all effect classes (e.g., degree of progression of effects) to represent continuous effect information without the need for manually selecting an appropriate set of anchor effects. In an embodiment, given a target age class, a personalized age embedding is derived that considers two aspects of facial aging: 1) a personalized residual age at the model-estimated age of the subject that preserves the subject’s aging information; and 2) an exemplary facial aging base at the target age that encodes aging patterns shared across the entire population. Training and runtime (inference time) embodiments are described that include generating recommendations and providing AR applications for e-commerce services.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Provisional Application No. 63 / 129794, filed December 23, 2020, and French Application No. FR 2105404, filed May 25, 2021. The entire contents of each application are incorporated herein by reference. Technical Field

[0003] This application relates to image processing and to image processing using neural network techniques to generate updated images to continuously apply effects such as aging facial images. Background Art

[0004] Facial aging, also known as age progression, aims to render input facial images with natural aging or rejuvenation effects in an aesthetic manner while preserving the individual's identity. With recent advances in deep learning, face synthesis has also shown substantial improvements in image fidelity and age accuracy of simulated facial images [10, 41, 24]. The main challenge in solving various legacy problems (e.g., continuous aging) is the lack of data. For example, many studies on facial aging [20, 41, 43, 10] require classifying images into 4-5 age groups (e.g., <30, 30-40, 40-50, 50+), and due to the limited amount of data for each age group, only images within the target age group can be generated. Another important issue is how to maintain individual characteristics during aging, as each person's aging pattern may be different.

[0005] Traditional facial aging methods mainly include two approaches: physics-based [3,42] and prototype-based [37,16]. Physics-based methods usually include complex physical modeling, taking into account skin wrinkles, facial shape, muscle changes, hair color, etc. This type of method usually requires a large amount of data and is very computationally expensive. Prototype-based methods first explored group-based designs by calculating the average face within a predefined age group, which cannot preserve personalized age information. In addition, all of these methods are not applicable to continuous facial aging.

[0006] With the recent success of generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs) [9] on image translation tasks, researchers have focused on applying these methods to face synthesis. IPCGAN

[41] achieved significant progress in generating facial images with noticeable aging effects by implementing an age estimation loss. A later variant

[43] created a pyramid structure for the discriminator to improve the understanding of facial aging at multiple scales. Continuous aging was not explored in these methods. He et al.

[10] introduced a multi-branch generator for group-based training and proposed the idea of ​​approximating continuous aging by linearly interpolating the latent representation between two adjacent age groups. The authors of

[24] also used a similar linear interpolation method to solve this problem, which was performed on the learned age latent code between two adjacent groups. These types of methods assume that the age progression between two adjacent groups is linear and that the learned group embedding can be directly used as the median age embedding. Therefore, this may lead to a shift in the target age in the generated images. Intuitively, this nonlinearity can be interpreted as: people at different stages age at different rates. Furthermore, this interpolation-based approach may alter individual characteristics when disentanglement is imperfect. Summary of the Invention

[0007] To address the above problems, a novel approach is proposed to implement the application of continuous facial effects (e.g., aging) through a unified network, where simple class estimators (e.g., age estimator for aging effect, smile progression (class) for continuous smile effect, etc.) are embedded into a conventional encoder-decoder architecture. This allows the network to learn class (e.g., age, smile, etc.) embeddings estimated by models for all progressive stages or classes (e.g., age, smile level, etc.), thereby representing the continuous effect information, without the need to manually select an appropriate set of anchor progressive stages (e.g., age, smile level, etc.). In the age example, given a target age (target age is one of the classes in the continuous effect), a personalized age embedding is derived that considers two aspects of facial aging: (1) a personalized residual age at the current age of the object embedded in the image, which preserves the individual's aging information; and (2) an exemplary facial aging basis at the target age, which encodes the aging pattern shared across the entire group. The computation and training mechanisms are described in detail. The computed target age embedding is then used for final image generation. Experiments on the FFHQ

[15] and CACD2000[5] datasets are presented in detail. These results show significant improvements over the state of the art in all respects, both qualitatively and quantitatively.

[0008] In the context of age, embodiments include a novel approach to self-estimate (e.g., where "self" refers to a model's estimate (e.g., model estimate)) continuous age embeddings and derive personalized age embeddings for face aging tasks by jointly training an age estimator with a generator. Quantitative and qualitative experiments and analysis show that the generated images better preserve personalized information, achieve more precise aging control, and present more fine-grained aging details. The continuous aging method according to embodiments herein generates images with better alignment to the target age and better preserves detailed individual features without the need to manually define appropriate age groups.

[0009] The proposed techniques and methods for estimating personalized age embeddings from related discriminative models can be easily applied to other conditional image-to-image translation tasks without introducing additional complexity. In particular, tasks involving continuous conditional and modeling (e.g., non-smiling to smiling, etc.) can benefit from this setting.

[0010] In one embodiment, a method is provided, comprising: providing a unified age simulation model to generate a new image of a target age of the object from an input image of the object; and generating a new image using the model; wherein the unified age simulation model provides a plurality of corresponding model-estimated age embeddings at each of a plurality of consecutive ages representing continuous aging information, the model-estimated age embeddings being learned by joint training of a generator and an age estimator embedded in an encoder-decoder architecture of the model, the age estimator being configured to determine the model-estimated age of the object in response to features generated from a corresponding encoder by a corresponding input image; and wherein the generator generates a new image using the features generated by the encoder based on the input image, the input image being transformed by corresponding embeddings in the model-estimated age embeddings determined based on the target age of the object and the model-estimated age.

[0011] In one embodiment, the encoder-decoder architecture includes an age estimator to estimate the model-estimated age of objects in the input image.

[0012] In one embodiment, an encoder of the model processes an input image to determine encoder-generated features, and wherein the age estimator processes the encoder-generated features to determine an age estimated by the model.

[0013] In one embodiment, features generated by the encoder are transformed by personalized age embeddings, the personalized age embeddings comprising: a corresponding embedding of the model-estimated age embedding determined according to the model-estimated age; and a corresponding embedding of the model-estimated age embedding determined according to the target age. In one embodiment, the personalized age embeddings comprise: a personalized residual age embedding determined from a plurality of corresponding model-estimated age embeddings in response to the model-estimated age to preserve the subject's identity information; and an exemplary age embedding comprising a corresponding embedding of the model-estimated age embedding determined according to the target age to represent aging patterns shared across the population.

[0014] In one embodiment, a personalized age embedding is applied based on an affine transformation.

[0015] In one embodiment, the generator processes the encoded features transformed by the personalized age embedding to generate new images of the target age.

[0016] In one embodiment, the model is one or both of a deep learning neural network model and a generative adversarial network model.

[0017] In one embodiment, the method includes providing a recommendation interface to obtain recommendations for one or both of products and services.

[0018] In one embodiment, the method includes providing an e-commerce purchasing interface to purchase one or both of the product and service.

[0019] In one embodiment, the method includes receiving an input image and providing a new image for display.

[0020] In one embodiment, the input image and the new image each include a face of a subject.

[0021] According to one embodiment, a method is provided, comprising: providing a unified model to generate a new image of a target class of a continuous effect of an object from an input image of the object; and generating a new image using the model; wherein the model provides a plurality of corresponding model-estimated class embeddings at each of a plurality of target ranges representing continuous effect information, the model-estimated class embeddings being learned by joint training of a generator and an effect estimator embedded in an encoder-decoder architecture of the model, the effect estimator being configured to determine the model-estimated class of the corresponding object in response to features generated from the corresponding encoder of the corresponding input image; and wherein the generator generates a new image using the features generated by the encoder based on the input image, the input image being transformed by corresponding embeddings in the model-estimated class embeddings determined based on the target class of the object and the model-estimated class.

[0022] According to one embodiment, a method is provided, comprising: providing a domain transfer model to transfer an input image to a new image, applying a continuous effect in a continuous manner to transform the input image to a target class of a plurality of continuous classes of the continuous effect using a plurality of corresponding model-estimated class embeddings learned by a model for each continuous class of the continuous effect; and transferring the input image to the new image using the domain transfer model. According to one embodiment, the continuous effect is an aging effect, and the target class is a target age. According to one embodiment, when transferring the input image, the domain transfer model operates to: a) generate an encoded feature of the input image; b) transform the encoded feature using: a personalized residual age embedding, determined from a plurality of corresponding model-estimated class embeddings responsive to a model-estimated age of an object in the input image to preserve the object's identity information; and an exemplary age embedding, including a corresponding embedding from the model-estimated class embedding determined according to the target age to represent aging patterns shared across a population; and c) using the transformed encoded feature to generate a new image. According to one embodiment, the model includes an age estimator to determine the model-estimated age. According to one embodiment, the age estimator comprises a classifier trained together with an encoder (of the model), the encoder configured to produce encoded features, and wherein the age estimator is trained to determine a corresponding model-estimated age of an object in a new image using corresponding encoded features encoded by the encoder. According to one embodiment, model-estimated class embeddings are defined during training of the age estimator together with the encoder, and corresponding ones of the model-estimated class embeddings are associated with corresponding model-estimated ages.

[0023] According to one embodiment, the method includes providing a recommendation for at least one of a product and service associated with the continuous effect. According to one embodiment, the recommendation is generated in response to one or both of a skin analysis of an input image and a user preference input. According to one embodiment, the target age is determined in response to the recommendation. According to one embodiment, the computing device is configured to communicate with an e-commerce service to make the recommendation. According to one embodiment, the computing device is configured to provide an annotated image generated from the input image to present the recommendation. According to one embodiment, the method provides an e-commerce interface to purchase the product, the service, or both. According to one embodiment, the method includes receiving an input image from a camera. According to one embodiment, the continuous effect is an aging effect; the product includes one of a skin rejuvenation product, an anti-aging product, and a cosmetic product; and the service includes one of a skin rejuvenation service, an anti-aging service, and a cosmetic service.

[0024] According to one embodiment, a computing device is provided, comprising a processing unit and a storage device coupled thereto, the storage unit storing instructions which, when executed by the processing unit, configure the computing device to perform a method according to any one of the method embodiments.

[0025] According to one embodiment, there is provided a computer program product comprising a non-transitory storage device storing instructions which, when executed by a processing unit of a computing device, configure the computing device to perform a method according to any one of the method embodiments.

[0026] According to one embodiment, a computing device is provided, comprising a processing unit and a storage device coupled thereto, the storage unit storing instructions that, when executed by the processing unit, configure the computing device to: provide a recommendation for at least one of a product and a service; and provide an age-simulated image, the age-simulated image comprising a new image generated from an input image and a target age, wherein the new image is generated according to any one of the method embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a block diagram of a model architecture according to an embodiment of the present invention, showing a training configuration.

[0028] Figure 2A 、 2B , 2C, 2D, 2E and 2F are image arrays showing input images with results from two models according to the corresponding embodiments of the present invention and results from a model according to the prior art, for six examples, showing more aging details and identity preservation in the results from the two models according to the corresponding embodiments of the present invention.

[0029] Figure 3A 、 3B , 3C and 3D are image arrays showing input images, with results from a model according to an embodiment of the present invention and results from three models according to the prior art, for four examples.

[0030] Figure 4 is an image array showing an input image, with results from a model according to embodiments herein, and zoomed-in facial crops of those results showing more details of aging.

[0031] Figure 5A and 5B is an image array showing input images, with results from a model according to embodiments herein and results from a model according to the prior art, for two examples showing continuous aging starting at age 21 over 4 year age intervals.

[0032] Figure 6A and 6B shows a confusion matrix for continuous aging, comparing results from a model according to embodiments herein with results from a model according to the prior art;

[0033] Figure 7 is an array of images showing linear interpolation between transformed identity encodings from a model according to embodiments herein.

[0034] Figure 8 is an image array showing input images and results from a first model using residual embeddings according to embodiments herein and a second (comparator) model not using residual embeddings according to embodiments herein.

[0035] Figure 9 is a block diagram of a computer system including multiple computing devices according to one embodiment. DETAILED DESCRIPTION

[0036] In particular, facial synthesis including facial aging has been one of the major challenges to significantly improve image fidelity by using generative adversarial networks (GANs). Most existing facial aging methods divide the dataset into several age groups and exploit a group-based training strategy, which inherently lacks the ability to provide continuous aging synthesis with fine control. In one embodiment, a unified network structure is provided that embeds a linear age estimator into a GAN-based model, where the embedded age estimator is jointly trained with the encoder and decoder to estimate the age of a facial image and provide personalized target age embeddings for age progression / regression. The personalized target age embedding is synthesized by combining the personalized residual age embedding of the current age and an exemplary facial aging basis of the target age, where all previous aging bases are derived from the learned weights of the linear age estimator. This formulation brings a unified perspective of estimating age and generating personalized aged faces, where an age embedding can be learned for each individual age model. Qualitative and quantitative evaluations on different datasets further demonstrate a significant improvement in continuous facial aging compared to the state-of-the-art.

[0037] Related work

[0038] - facial aging model

[0039] Traditional methods can be divided into physical model-based methods [3, 42, 34] and prototype-based methods [31, 37, 16, 17]. Physical model-based methods focus on creating models to address specific sub-effects of aging, such as skin wrinkles [42, 2, 3], craniofacial growth [38, 27], muscle structure [34, 28], and facial composition [35, 36]. These methods are generally very complex, which usually requires facial image sequences of the same person at different ages and expert knowledge of aging mechanisms. Prototype-based methods [31, 37, 4] use group-based learning to explore the facial progression problem, where the average face is estimated within each age group. However, personalized aging patterns and identity information are not well preserved in this strategy. In [40, 44, 33], sparse representations of the input image have been used to express personalized facial transformation patterns. Although personalized aging patterns are preserved to some extent by this approach, the synthesized images have quality issues.

[0040] Recently, deep learning methods have been used to simulate personalized aging transformations. Wang et al.

[39] proposed a recursive neural network model that uses a series of recursive forward passes to achieve a smoother transition from young to old. Later GAN-based works [18, 41, 43] have achieved remarkable breakthroughs in image fidelity. Li et al.

[18] designed three subnetworks for local blocks and fused local and global features to obtain smooth synthetic images. IPCGAN

[41] implemented age estimation loss and identity loss on the generated images to achieve good facial aging effects. More efforts have been made to address the problems of age accuracy and identity persistence. Yang et al.

[43] and Liu et al.

[20] introduced modifications to the discriminator loss to guide the output image to a more accurate age. The authors of

[21] improved the image quality of the synthetic images by using wavelet packet transform and multi-facial attribute encoding. However, these methods [41, 43, 20] condition the output image by cascading one-hot vectors representing the target age group. In order to obtain continuous aging conditions, the vectors will be expanded to larger dimensions, which makes training unstable and more complex. In addition, it requires a large number of training images.

[0041] Although some works aiming to interpolate features in the latent space [46, 1, 32] provide directions for supporting continuous aging, they are limited in their ability to produce high-quality images while preserving identity. In

[10] , the authors proposed linearly interpolating feature vectors from adjacent age groups based on group-based training to achieve continuous aging progression. Similarly,

[24] linearly interpolated between two adjacent anchor age embeddings. These methods follow the assumption that the embeddings are linearly aligned between anchors, which makes the determination of anchor age crucial. In this work, a continuous model is proposed to estimate age embeddings without manual operation, while achieving better continuous aging modeling.

[0042] -Generative Adversarial Networks

[0043] Generative adversarial networks [9] have become a popular choice for image-to-image translation tasks. CycleGAN

[47] and Pix2Pix

[14] explored image translation between two domains using unpaired and paired training samples, respectively. Recent work [6, 19] proposed training techniques to achieve multi-domain translation. In [22, 23], the authors first explored conditional image generation as an extension of basic GANs. Later work [7, 26] further showed superiority in many conditional image translation tasks by incorporating conditional translation and injection into the model in a more efficient way.

[0044] - Facial age estimation

[0045] The task of predicting apparent age is a regression problem that estimates a continuous numerical value for each given facial image. Deep Expectation of Appearance Age (DEX)

[30] proposed a method that achieved a mean absolute error (MAE) of 3.25 on MORPH II

[29] by combining classification loss and regression loss. Pan et al.

[25] proposed using mean-variance loss on probability distribution to further improve the MAE on MORPH II to 2.16.

[0046] Model Architecture Implementation

[0047] Figure 1 1 is a diagram of a model architecture 100 according to one embodiment. The model 100 is shown in a training configuration. In this embodiment, an input image 102 and a real image 104 are received for processing, and a pseudo image 106 is generated. The input image includes an image of a subject at a real age or current age (which can be any age). The age is related to the age of the subject in the image, not the age of the image itself. The real image 104 includes an image of another subject at a target age (e.g., the image is a real image that represents features of the target domain (a specific age)). The target age can be any age relative to the actual age / current age, including the same age.

[0048] In this embodiment, the age estimator is jointly trained with the image generator, where E (e.g., 108) is the generated feature (e i ) 110, and C (e.g., 112) is branched for the age estimation task. C produces the output β θ 114, the probability distribution of age over multiple age classes. The Personalized Age Embedding Transform (PAT, Equation (2) 116) is based on two components: 1) the residual aging basis at the current age (e.g., the age estimated by the model determined by C); and 2) the exemplary facial aging basis at the target age 118 (e.g., receiving the target age as input). The PAT produces the output γ φ 120. In one embodiment, the target age is a specific absolute age (e.g., "29" means 29 years old). In one embodiment, the target age is a delta age or age difference (e.g., a positive integer or a negative integer (e.g., "-5")) relative to the current age in the image or other baseline age.

[0049] In this embodiment, the transformed identity code (e) is generated by an affine projective transformation of the feature 110 through operations 122 and 124 using the outputs 120 and 114, respectively (see equation (3)). i ,t i ) (eg, 126) for decoding by G (eg, at 128) to produce pseudo image 106 (output).

[0050] The entire model (100) is trained via the discriminator D (140) using age losses (e.g., 130, 132, and 134), identity loss 136, and adversarial loss 138. As further described below, the second encoder is used 142 and second age estimator 146 (e.g., in a training configuration of model 100).

[0051] Figure 1 Shown in respective even-length dashed boxes are a first grouping 148 and a second grouping 150. The first grouping 148 highlights encoder components according to an embodiment, and the second grouping 150 highlights decoder or generator-side components that are useful in a training environment and are not involved in back-propagation during training.

[0052] like Figure 1 As shown in FIG, according to one embodiment, the model includes four components: 1) identity encoding module E 108; 2) age estimation module C 112; 3) personalized age embedding transformation module PAT 116; and 4) aged face generation module G 128. The encoder network E is used to generate an aging face from a given input image x. i,102 extracts the identity information, where the code 110 is represented as e i =E(x i ). Then, the embedded age estimator C is used to obtain the age probability distribution 114 of the identity code 110. Based on the age distribution estimated by the model and the target age t 118, the identity code e i Finally, the generator G,128 encodes the identity from the transformed PAT (e i ; t) (e.g., 126) decodes the synthesized face (pseudo image 106).

[0053] According to one embodiment, all modules are jointly optimized end-to-end under three objectives, including the mean-variance age loss

[25] (130, 132, and 134) for accurate aging, the L1 reconstruction loss 136 for identity preservation, and the adversarial loss 138 for image realism.

[0054] Unlike many previous face aging works [41, 10], which require a pre-trained age classifier to guide face aging training, model 100 directly obtains model-estimated age embeddings by leveraging a unified framework for simultaneously implementing face aging and age estimation. More advantageously, compared to interpolation-based methods, the embedded age estimator 112 can not only achieve personalized continuous age transformation in a more accurate manner, but also provide guidance for face image generation (e.g., at G 128).

[0055] -formula

[0056] Identity Age Estimation Module (C): In previous works [41, 10], face aging and face age estimation are treated as two independent tasks, where the age estimation model (usually a classifier) ​​is pre-trained separately and then used to guide the generator to achieve natural aging effects. According to the embodiments of this paper, since the two tasks mentioned are inherently related, they can be achieved in a unified structure by sharing the encoder E.

[0057] In the embodiment of model 100, the age estimator C 112 including a global average pooling layer and a fully connected layer is branched from E 108. Finally, the age probability distribution p can be obtained by performing a softmax function. i ∈R K (For example, β θ 114), where K represents the number of age categories. In one embodiment, K=100. i Determine the parameter m i .

[0058] The unified design can provide three advantages. First, it eliminates the need to obtain a well-trained age estimation model in advance. Second, the age estimation of identity encoding helps the model build a more age-specific identity representation. Third, the weights W in the fully connected layer C It is also used as an age embedding basis (the bias term is set to zero), which encodes example face information from the perspective of metric learning. In the comments:

[0059] a j =W C [j] (1)

[0060] Where W C ∈R K×D ,a j ∈R D and D is equal to the channel size of the identity code. It should be understood that the size D is Figure 1 The discriminator D at 140 is not relevant.

[0061] Despite Figure 1 The embodiments described in

[15] are described as aging simulation or aging effects, but other multi-class domain transfers that apply continuous effects are also considered. In a general sense, the age embedding is based on the class embedding, a latent representation of some class, for example, age = 51 or smile degree = 40.

[0062] Personalized Age Embedding Transform (PAT): Facial aging is an inherently challenging and ambiguous task because different facial signs / symptoms age differently for different people at different stages. Therefore, personalization is required when performing facial aging. In model 100, this personalization is characterized by the age probability distribution p i,j ∈R K and exemplary facial aging foundations i,j ∈R D The remaining age embeddings are calculated, where i represents sample i and j∈1,2,…K represents age. To obtain any target age t i The personalized aging foundation of the process is formulated as follows:

[0063]

[0064] The term represents the personalized aging basis of the identity by taking the expected value of the aging basis based on the probability distribution of age. Then, the current (model-estimated) age is subtracted from the personalized aging basis. The residual age embedding is obtained by applying the exemplary facial aging basis of . The residual age embedding retains the personalized factors of identity while removing the main aging factors at the age estimated by the model. The final personalized target age embedding is achieved by targeting the aging The target aging basis is obtained by adding an exemplary facial aging basis to the target age, which encodes the shared age factor of the target age in the entire population. Using personalized target age embedding Then an affine projective transformation is applied to derive the original identity encoding E(x i )=e i The scale and offset coefficients are similar to those of conditional BN[8] and AdaIN

[13] :

[0065]

[0066] In the experiments, no significant performance differences were observed w / w β φ .

[0067] Continuous Aging: As the aging cardinality from the fully connected layers encodes each individual age (e.g., in increments of 1 year, according to one embodiment), continuous aging is naturally supported by choosing an arbitrary target age as input 118. While some previous group-based methods could support continuous aging via linear interpolation in the latent space, careful selection of anchor age groups is required.

[0068] However, the techniques and methods of embodiments explicitly model fine-grained age progression by learning independent aging bases for each age separately (e.g., in classes 1, 2, 3, ... K, where each class is 1 year wide).

[0069] -Target

[0070] According to one embodiment, the design of the target ensures that the synthesized facial image 106 reflects accurate age progression / regression, maintains identity, and looks realistic.

[0071] Mean-variance age loss: The age loss plays two roles in the network: 1) it helps the estimator (C) learn a good aging basis for all ages; and 2) it guides the generator (G) by estimating the age of the generated pseudo images. To achieve these two goals, according to one embodiment, the mean-variance age loss proposed in

[25] is adopted. Given an input image x i and age label yi, the mean-variance loss is defined as:

[0072]

[0073] in, is the mean of the distribution (e.g., from equation (2)), is the variance of the distribution.

[0074] In addition to being more effective than other losses on the age estimation task, the mean-variance loss also satisfies the desire to learn a relatively concentrated age distribution while capturing the age continuity of adjacent aging bases, according to an embodiment. According to an embodiment, the supervised age loss is formulated as:

[0075] Ltrue= L mv (C(E(x)),y) (5)

[0076] To guide facial aging, according to an embodiment, the embedded age estimator 146 is applied to both the transformed identity encoding stage and the generated image stage (as shown in Figure 1 FIG. 2), such that, according to an embodiment:

[0077]

[0078] When the age estimator 146 and the encoder 142 are used for the transformed identity encoding 126 and the pseudo image 106, their corresponding weights are not updated during backpropagation.

[0079] L1 reconstruction loss: Another important aspect is to preserve the identity of the individual, according to an embodiment. By setting the target age to its model-estimated age, a L1 pixel-wise reconstruction loss (e.g., at identity loss 136) is applied to the synthesized face. Specifically, its formulation is:

[0080]

[0081] Experiments are conducted using the cycle-consistency loss proposed in StarGAN [6] to enforce the identity criterion. It has been disclosed that the pixel-wise L1 reconstruction loss is sufficient to achieve the goal without much effort in tuning hyperparameters.

[0082] Adversarial loss: To produce high-fidelity images, according to an embodiment, a GAN loss (e.g., 138) is applied in an unconditioned adversarial training manner. More specifically, according to an embodiment, a PatchGAN

[14] discriminator (140) is adopted and the hinge loss is optimized, with the formulation:

[0083]

[0084] where the data distribution is denoted as x ~ p 数据 (x) and z ~ p 数据 (z). Furthermore:

[0085]

[0086] In experiments, it is observed that the sampled age is equal to or close to the target age t iTraining the discriminator with real examples helps stabilize the learning process.

[0087] According to one embodiment, all objectives are jointly optimized using different balancing coefficients as follows:

[0088]

[0089]

[0090] experiment

[0091] Datasets: The model is evaluated on FFHQ

[15] and CACD2000[5]. FFHQ consists of 70,000 images with a resolution of 1024x1024. Following the data preprocessing procedure of

[24] , images with ids 0-68999 are used as the training set, and images with ids 69000-69999 are used for testing. Based on the facial attributes annotated by Face++, the following images are filtered out: low confidence in distinguishing gender, low confidence in estimating age, wearing sunglasses, extreme poses and angles (Facial Attribute Annotation API: URL www.faceplus.com).

[0092] Because the annotations from

[24] only include age group labels, age label information is obtained from

[45] . To reconcile the age group labels and age labels, images where the age labels are inconsistent with the age group labels are further filtered out. This results in 12,488 male and 13,563 female images for training, and 279 male and 379 female images for testing. CACD2000 consists of 163,446 images where the age ranges from 14 to 62 years old, 10% of which are randomly sampled for evaluation. Images are classified into male and female using Face++, and facial landmarks are extracted using Dlib (Dlib toolkit: URL dlib.net).

[0093] Implementation: Since the aging patterns between males and females are different, two separate models were trained on the FFHQ dataset for 256x256 and 512x512 resolutions. The model architecture was modified based on Cycle-GAN

[47] . In Equation (4), λ mv1 and λ mv2 is set to 0.05 and 0.005. In equation (6), λ 伪1 and λ 伪2 is set to 0.4 and 1. In equation (10), λ 年龄 ,λ idt and λ adv are set to 0.05, 1, and 1 respectively.

[0094] -Qualitative assessment

[0095] Facial aging: Test results on FFHQ are presented and compared with the results of

[24] . The images of

[24] were generated using the code provided by them (Lifespan official code: URL github.com / royorel / Lifespan_Age_Transformation_Synthesis). To illustrate the model performance at different age groups, 6 input examples AF from 4 representative age groups (<30, 30-40, 40-50, 50+) are shown, and results are generated for each group and are presented in

[24] . Figure 2A 、 2B , 2C, 2D, 2E, and 2F are listed in arrays 200, 202, 204, 206, 208, and 210. The input image is in the leftmost column, with the four target ages shifted to the right. The top row shows the lifespan output, and the middle and bottom rows show the output of the model according to an embodiment of the present invention, where the image size of the bottom row is 512×512.

[0096] The target ages for the model (shown in the column to the right of the input image) are chosen to be 25, 35, 45, and 55 years old. Figure 2A 、 2B As can be seen in 2C, 2D, 2E and 2F, the images generated by the model according to the embodiments of this invention result in fewer artifacts and show clearer aging details, such as beard color changes ( Figure 2A and 2C 200 and 204) and wrinkles on different parts of the face (see Figure 2B 、 2C , 2D and 2E examples 202, 204, 206 and 208). Figure 2F The convincing details in example 210 show how well the individual features (mole 210A) are preserved using this model.

[0097] We also used a model trained on FFHQ at a resolution of 256x256 (bottom (fourth) row) to generate images directly on CACD2000 to compare with Figure 3A 、 3B, 3C and 3D examples 300, 302, 304 and 306 are compared with CAAE

[46] (top (first) row), IPCGAN

[41] (second row) and S2 GAN

[10] (third row). Input images 308, 310, 312 and 314 are wrapped in boxes. The images for demonstration are examples presented in

[11] , which is the latest work on CACD2000. For all age groups (listed from left to right as 11-30, 31-40, 41-50 and 50+), the model according to the embodiments of this paper presents more obvious and more subtle aging effects compared to all previous works.

[0098] Aging details: Images generated using models according to embodiments herein convey significant levels of aging details on different parts of the face (e.g., facial crops). Figure 4 In the three examples of the array 400, three magnified facial crops 402, 404, and 406 (bottom row) from the generated image (middle row) are shown, which give a clear and detailed view of the enhanced wrinkles, skin smoothness, beard and eyebrow color variations. The leftmost example 402 shows a true / target age pair of 36 / 55 (i.e., the input image shows an individual of 36 years old, and the generated image shows an image with a target age of 55). The middle example 404 shows a true / target age pair of 33 / 55, and the rightmost example 406 shows a true / target age pair of 66 / 30.

[0099] Continuous aging: Figure 5A and 5B In FIG, image arrays 500 and 502 show some examples of continuous aging results comparing images generated by a model according to an embodiment in the second and fourth rows with images generated by a lifespan model

[24] in the first and third rows. In each array, the input image is displayed in the leftmost column. An age step of 4 is selected for presentation (e.g., 21, 25, 29, ... 65 from left to right). According to one embodiment, a gradual and smooth natural aging process (e.g., changes in wrinkle depth, beard, and facial pigmentation) can be observed from the images generated by the model while retaining individual characteristics. However, interpolation-based methods in lifespan lack the ability to generate images of the target age that are better aligned and cannot retain certain personalized information. In addition, the model according to the embodiment generates more realistic aging effects with minimal artifacts.

[0100] -Quantitative evaluation

[0101] Identity Preservation: Identity preservation is evaluated using the face verification rate metric. Specifically, in order to make a fair comparison with previous work, the evaluation scheme of

[10] is followed on an age group basis. The face verification rate is calculated between all combinations of image pairs, i.e., (test, 10-29), (test, 30-39), ..., (30-39, 40-49), (40-49, 50-59). The face verification score is obtained from face++ and the threshold is set to 76.5 (@FAR=1e-5). The complete results of CACD2000 and FFHQ are presented in Tables 1 and 2, respectively. As shown in the results, the model according to the embodiment achieves the highest face verification rate for both datasets among all candidate datasets, which indicates that it best meets the identity preservation requirement of the task.

[0102]

[0103] Table 1

[0104]

[0105] Table 2

[0106] Aging Accuracy: To evaluate aging accuracy, an unbiased age estimator is used to infer the age of the generated images. To compare with previous group-based methods on CACD2000, images consistent with the CACD2000 age group setting are generated. The age group size is adaptively increased / decreased by a factor of 10 from the actual age of the input image as the target age for generation, i.e., given a current age of 23, images of age group 30-40 are generated using a target age of 33. Without access to the evaluated age estimator of

[10] or its pre-trained model used to evaluate the model according to the embodiments of this paper to perform direct comparison, the age estimation results of Face++ are used for the model according to the embodiments of this paper and one of the models of the accessible previous work IPCGAN

[41] , which is also evaluated in

[10] to show relative comparison. The evaluation of FFHQ follows the same procedure as CACD2000. The evaluation results of CACD2000 and FFHQ are shown in Tables 3 and 4, respectively. As shown in the results, the model evaluated using Face++ according to the embodiments of this paper has a more reasonable mean age in each age group than IPCGAN

[41] and Lifespan

[24] on CACD2000, and has similar performance to Lifespan on FFHQ.

[0107]

[0108] Table 3

[0109] 10-29 30-39 40-49 50+ Lifespan

[24] - 38.4 - 63.8 Our 30.7 38.4 47.7 62.1

[0110] Table 4

[0111] Image Fidelity: For image fidelity, the model according to the embodiments of this paper is evaluated using the Frechet Inception Distance (FID)

[12] metric on both datasets. Similar to the previous image generation setting, the FID scores are calculated on the generated images corresponding to the same age group on CACD2000. For comparison with

[24] on FFHQ, the FID scores are calculated on the generated images, which share the same age group range. The results are shown in Table 5 (FID evaluation: lower is better). On both datasets, the model according to the embodiments of this paper achieves the lowest FID score, which quantitatively demonstrates its superiority in image quality.

[0112] CACD2000 FFHQ CAAE

[46] 44.2 - IPCGAN

[41] 9.1 - S2 GAN

[10] 8.4 - Lifespan

[24] 11.7 26.2 Our 6.7 18.5

[0113] Table 5

[0114] -Model interpretability and ablation studies

[0115] Continuous Aging: To evaluate how well the models according to the examples herein generate synthetic images in a continuous setting, we used an age estimator to predict the age of 1) the model for ages 25 to 65 and 2) the generated pseudo images by linear interpolation between anchored aging bases. The anchor bases were generated by taking the average of each aging base within an age group. Based on the MAE of the estimator, an age step size of 3 was chosen.

[0116] The confusion matrix was calculated based on the aging accuracy used for each method using the age estimator jointly trained on the FFHQ dataset. Figure 6A and 6B The corresponding confusion matrices 600 and 602 indicate that, compared with the pseudo images generated by the linear interpolation method, the pseudo images generated according to the model of this paper exhibit more obvious continuous aging trends with much higher aging accuracy.

[0117] Interpolation between two identities in the latent space: Figure 7 700 , it is further shown that the model according to the embodiments herein also learns disentangled representations of age and identity in the latent space. Figure 7 The figure shows linear interpolation between transformed identity codes for three paired examples, with the real images in the box (leftmost and rightmost columns). From left to right, linear interpolation is performed between the transformed identity codes for two images of the same target age 65. Images for interpolated encoding are generated. As shown, the identities gradually change while maintaining the corresponding ages. Personal features, such as eye color and tooth shape, change smoothly from one person to the next.

[0118] Use of residual embedding: A feature of the model architecture according to embodiments is the formulation of a personalized age embedding that combines both the individual’s personalized aging characteristics and the aging effects shared across the population. To better illustrate and understand the effectiveness of the design, a comparator model is trained without the addition of the residual embedding (i.e., directly applying the exemplary face aging basis for the target age a(i,j) = t,) and compared to the model according to embodiments herein with the addition of the residual embedding. Figure 8 FIG. 8 of the image array 800 shows two examples comparing the results without residual embedding (first and third rows) and with residual embedding (second and fourth rows). The input images are in the leftmost column, and the age ranges of 11-30, 31-40, 41-50, and 50+ are in the columns from left to right. In both examples, more unnatural artifacts and a tendency of the exemplary face modification are observed in the images generated without residual embedding.

[0119] Applications

[0120] In one embodiment, the disclosed techniques and methods include developer-related methods and systems to define (e.g., by tuning / training) a model with a generator for providing image-to-image conversion of age simulation and an age estimator. The generator exhibits continuous control using multiple model-estimated age embeddings across multiple consecutive ages learned by tuning to create a smooth transformation between the original image (input image) and the transformed image (new image). In one embodiment, the images are facial (e.g., of a face). In one embodiment, a personalized age embedding (determined using the target age of the original image and the model-estimated age from multiple model-estimated age embeddings) is used to transform features generated by an encoder from an encoder component of the model.

[0121] In one embodiment, the model for image-to-image conversion according to embodiments herein is incorporated into a computer-implemented method (e.g., application) or computing device or system to provide a virtual reality, augmented reality, and / or modified reality experience. The application is configured to facilitate a user to take a selfie image (or video) using a smartphone or tablet equipped with a camera, and the generator applies the desired effect, e.g., for playback or other presentation through the smartphone or tablet.

[0122] In one embodiment, the generator as taught herein is configured for loading and execution on a commonly available consumer smartphone or tablet (e.g., target device). An example configuration includes a device with the following hardware specifications: CPU E5-2686 v4 @ 2.30GHz, with only 1 core and 1 thread. In one embodiment, the generator is configured to be loaded and executed on computing devices with more resources, including servers, desktops, gaming computers, or other devices with multiple cores and executing in multiple threads. In one embodiment, the generator is provided as a (cloud-based) service.

[0123] In one embodiment, in addition to developer (e.g., for use at training time) and target (for use at inference time) computing device aspects, one of ordinary skill will understand that computer program product aspects are disclosed wherein instructions are stored in a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.) to configure a computing device to perform any method aspects disclosed herein.

[0124] Figure 9 is a block diagram of a computer system 900 according to one embodiment. The computer system 900 includes multiple computing devices (902, 904, 906, 908, 910, and 950), including servers, developer computers (PCs, laptops, etc.), and typical user computers (e.g., PCs, laptops, and smaller form factor (personal) mobile devices, such as smartphones and tablets, etc.). In this embodiment, in accordance with the teachings herein, the computing device 902 provides a network model training environment 912 including hardware and software to define a model for providing a continuously aged image-to-image transformation. Components of the network model training environment 912 include a model trainer component 914 to define and configure, for example, E 108, C 112, PAT 116, G 128, and the like by adjusting the network model training environment 912. 142, 146 and D 140. Components 140, 142, and 146 are constructs used for training, but are not used as runtime components, such as for generating new images in runtime (inference time) applications.

[0125] In this embodiment, for example, Figure 1 The model network architecture 100 is used to perform the adjustments. In this embodiment, a data server (e.g., 904) or other form of computing device stores an image dataset 926 of images for training and other purposes, and is coupled via one or more networks, representatively shown as network 928, which couples any of the computing devices 902, 904, 906, 908, and 910. For example, network 928 is wireless or otherwise, public or otherwise, etc. It will also be understood that system 900 is simplified. At least any service can be implemented by more than one computing device.

[0126] Once trained, the trained model 100 can be further defined to include runtime components as needed and provided as trained model 930. According to the techniques and methods herein, in one embodiment, the trained model 930 can be used in different ways. For example, Figure 9 In one embodiment shown in FIG, trained model 930 is provided as a cloud service 932 or other software as a service (SaaS) provided via cloud server 908. User applications such as augmented reality (AR) applications 934 are defined for use with cloud service 932, which provides an interface to trained model 930. In one embodiment, AR application 934 is provided for distribution (e.g., via download) from an application distribution service 936 provided by server 906.

[0127] Although not shown, in one embodiment, the AR application 934 is developed using an application developer computing device for a specific target device having specific hardware and software (particularly, an operating system configuration, etc.). In one embodiment, the AR application 934 is a native application configured to execute in a specific local environment (e.g., an environment defined for a specific operating system (and / or hardware)). Native applications are typically distributed through an application distribution service 936 configured as an e-commerce "store" operated by a third-party service, although this is not required. In one embodiment, the AR application 920 is a browser-based application, for example, configured to execute in the browser environment of the target user device.

[0128] The AR application 934 is provided for distribution (e.g., download) by a user device (such as the mobile device 910). In one embodiment, the AR application 934 is configured to provide an augmented reality experience to the user (e.g., via an interface). For example, an effect is provided to an image via processing by the inferred time generator 930. The mobile device has a camera (not shown) to capture an image (e.g., captured image 938), which in one embodiment is a still image including a selfie image. The effect is applied to the captured image 938 using image processing techniques that provide image-to-image conversion. An age-simulated image (new image) 940 is defined and displayed on a display device (not shown) of the mobile device 910 to simulate the effect on the captured image 938. The position of the camera can be changed and the effect applied in response to further captured images to simulate augmented reality. It should be understood that the captured image defines the source, input image, or original image, and the aged image defines the new image, the translated or transformed image, or the image to which the effect is applied.

[0129] exist Figure 9In the cloud service example of an embodiment, a captured image 938 is provided to a cloud service 932, where it is processed by a trained model 930 to perform an image-to-image transformation with continuous aging to define an aged image 940. The aged image 940 is transmitted to a mobile device 910 for display, saving (storage), sharing, etc.

[0130] In one embodiment, the AR application 934 provides an interface (not shown) for operating the AR application 934, such as a graphical user interface (GUI), which can be voice-enabled. The interface is configured to enable image capture, communication with the cloud service, and display, saving, and / or sharing of the converted image (e.g., the aged image 940). In one embodiment, the interface is configured to enable the user to provide input to the cloud service, such as defining a target age. In one embodiment, the input includes an age increment. As previously described, in one embodiment, the input includes a product / service selection. For example, a product / service selection is associated with an age increment to update the input image. In one example, the input can be a lifestyle factor, such as smoking rate, sun exposure rate, or other factors that contribute to premature aging. The lifestyle factor can be associated with the age increment applied to the input image.

[0131] exist Figure 9 In an embodiment, the AR application 934 or another (not shown) provides access to a computing device 950 that provides an e-commerce service 952 (e.g., via a communication interface). The e-commerce service 952 includes a recommendation component 954 to provide (personalized) recommendations for products, services, or both. In this embodiment, such products and / or services are anti-aging or skin rejuvenation products and / or services, etc. In one embodiment, for example, such products and / or services are associated with specific skin signs. In one embodiment, images captured from the device 910 are provided to the e-commerce service 952. According to one embodiment, skin sign analysis is performed using deep learning, for example, by a skin sign analyzer model 956. The skin (e.g., facial areas associated with specific skin signs) is analyzed using image processing of a trained model to generate a skin analysis including scores for at least some skin signs. For example, a (dedicated) aging sign estimation model (e.g., a classifier) ​​based on a ResNet

[27] architecture can be used to generate values ​​for individual scores on the image, for example, as previously described for analyzing training set data.

[0132] In this embodiment, skin signs (e.g., their scores) are used to generate personalized recommendations. For example, corresponding products (or services) are associated with one or more skin signs and specific scores (or score ranges) for these signs. In this embodiment, the information is stored in a database (e.g., 960) for use by the e-commerce service 952, for example, via appropriate searches that match the user's data with product and / or service data. In one embodiment, other user data used by the recommendation component 954 includes any of gender, ethnicity, and location data.

[0133] In this embodiment, the skin sign score of the user's captured image is provided from the e-commerce service for display via the AR application 934, such as in the AR application interface. For example, in one embodiment, other means are used to generate or modify the score, such as by applying rules or other code.

[0134] In one embodiment, an annotated image is provided from a captured image of a user (i.e., an input image), for example, where the annotated image includes the input image annotated with any of: a skin sign score, a skin sign description / information associated with such a score, product information associated with such a score, or service information associated with such a score.

[0135] In a non-limiting embodiment, a user receives personalized product recommendations, such as those from an e-commerce service 952. The user selects a specific product or service. The selected product or service is associated with an age increment (which can be a determined rule (e.g., taking into account the subject's actual age, length of product use, other demographic or geographic data, etc.), thereby invoking a modification of the input image. For example, the modification simulates the age of the subject in the input image to produce a new image of the new target age. The input image and the target age determined based on the product or service selection can be provided to the cloud service 932 to receive an aged image (e.g., an example of 940).

[0136] exist Figure 9 In an embodiment, the e-commerce service 952 is configured with a purchase component 958 to facilitate the purchase of products or services. The products or services may include cosmetic products or services or other products or services. Although not shown, the e-commerce service 952 and / or the AR application 934 provide image processing of the captured image to simulate the cosmetic product or service, such as applying cosmetics to the captured image to produce an image with the effect applied.

[0137] Although a captured image is used as the source image for processing in the embodiments described above, in one embodiment, another source image is used (e.g., from a source other than the camera of device 910). One embodiment may use a captured image or another source image. Whether it is a captured image or another image, in one embodiment, such an image is a high-resolution image to improve the user experience because the training model 930 is trained on high-resolution images. Although not shown, in this embodiment, the image used by the skin sign analyzer model 956 is scaled down during analysis. For this analysis, additional image preprocessing is performed.

[0138] In one embodiment, the AR application 934 can coach the user on quality features (i.e., lighting, centering, background, hair occlusion, etc.) to improve performance. In one embodiment, if the image does not meet certain minimum requirements and is not suitable, the AR application 934 rejects the image.

[0139] Despite Figure 9 910 is shown as a mobile device, but in one embodiment, the computing device 910 can have a different form factor as described. Instead of (or in addition to) providing the trained model 930 as a cloud service, it can be hosted and executed locally to a particular computing device with sufficient storage and processing resources.

[0140] Thus, in one embodiment, the AR application 934 performs and method, and the computing device is configured to: provide an interface for receiving an input image; transmit the input image to a recommendation service to receive a skin analysis and a recommendation including at least one recommended product or service responsive to the analysis; provide an interface for presenting an annotated input image, the annotated input image showing the skin analysis and displaying at least one recommendation; provide an interface for selecting a product or service from the recommendations; in response to a selection, generate an age-simulation image using a target age associated with the selected product or service and the input image, and present the age-simulation image via the interface; and provide an interface for purchasing the product or service via an e-commerce service.

[0141] In one embodiment, the AR application communicates for recommendations and provides recommendations without performing skin analysis, for example, based on a user's preference set—eg, selecting recommendations for facial areas or specific skin sign or signs.

[0142] In one embodiment, the AR application 934 generates a second age-simulating image at the second target age—for example, where the second target age is not referenced to use of the recommended product. In one embodiment, the two age-simulating images are presented simultaneously for comparison. Effects (such as makeup and hair effects) can be applied to any of the age-simulating images.

[0143] In one embodiment, the computing device includes a camera, and wherein the processing unit receives a raw image from the camera.

[0144] In one embodiment, the product comprises one of a skin rejuvenation product, an anti-aging product, and a cosmetic product. In one embodiment, the service comprises one of a skin rejuvenation service, an anti-aging service, and a cosmetic service.

[0145] In one embodiment, a computing device, such as mobile device 910, is configured to perform a method according to the computing device aspects described herein. Other aspects will be apparent, such as computer program product aspects.

[0146] In one embodiment, a network model training environment provides a computing device configured to perform a method such as the method configured by adjusting a (GAN-based) age simulation generator.

[0147] In one embodiment, a computing device including a facial effects unit is provided, comprising processing circuitry configured to apply at least one continuous effect to a source image and generate one or more virtual instances of the applied effect image on an e-commerce interface, the facial effects unit utilizing an encoder having a generator and an estimator to simulate the applied continuous effect (e.g., aging), wherein the applied continuous effect has a continuous control over the effect of the corresponding class (e.g., age over an age range, smile degree over a smile range, etc.).

[0148] In one embodiment, a computing device includes a recommendation unit comprising a processing circuit configured to present a recommendation for a product and / or service and receive a selection of the product and / or service, wherein the product and / or service is associated with a target age (e.g., a modifier such as an increment relative to a current age or an absolute age number). The facial effects unit is configured to generate an applied effect image of the target age in response to the selection, thereby simulating the effect of the product and / or service on the source image. In one embodiment, the recommendation unit is configured to obtain the recommendation by calling a skin sign analyzer to determine a current skin sign score using the source image; and determining the product and / or service using the current skin sign score. In one embodiment, the skin sign analyzer is configured to analyze the source image using a deep learning model. In one embodiment, the target age is defined from an aging target modifier associated with the product / service.

[0149] In addition to age-related embodiments, the network structures, methods, and techniques proposed in this paper can also be applied to other multi-class domain transfer tasks to avoid group-based training and achieve more accurate continuous modeling. It should be understood that domain transfer tasks involve converting source images from one domain to another, such as where an effect is applied. The "multi-class" here refers to the progression of different degrees or granularities of continuous effects. In the age example of the continuous effect, K classes were discussed. For a continuous effect related to smile, for example, in one embodiment, the class can represent the degree of smile at a granularity of 1%. In the smile example, the age estimator C and its training complement An estimator adapted (e.g., via training) for smile estimation (e.g., predicting the degree of smile rather than age). The estimator may be used to determine a class embedding estimated by the model at each continuous granularity range (class) of a continuous effect representing continuous effect information.

[0150] Other multi-class effects can be considered (e.g., domain shift), including non-facial effects (e.g., degree of baldness, weight gain, etc.). Thus, the generator is able to generate continuous effect images at the target class from the combined encoder and estimator, where the target is one of the classes of the effect (i.e., granularity range).

[0151] Thus, in one embodiment, a method is provided, comprising providing a unified model to generate a new image of a target class of a continuous effect of an object from an input image of the object. The model provides a plurality of corresponding model-estimated class embeddings at each of a plurality of continuous classes representing continuous effect information. The model-estimated class embeddings are learned by joint training of a generator and an estimator embedded in an encoder-decoder architecture of the model. The estimator is configured to determine the model-estimated class of the corresponding object using features generated by the corresponding encoder in response to the corresponding input image. The generator generates a new image using the features generated by the encoder based on the input image, the input image being transformed by a corresponding embedding in the model-estimated class embedding determined based on the target class.

[0152] In one embodiment, the continuous effect is an aging effect, and the target range is a specific one of the age categories (eg, one of the degrees of the continuous effect), eg, an integer number of years.

[0153] in conclusion

[0154] In this work, a novel approach is introduced to the task of facial aging, with a particular focus on continuous aging aspects. By introducing an age estimation module in a GAN-based generator, a unified framework for learning continuous aging fundamentals is proposed. The designed PAT module further enhances the personalization of the aging fundamentals of example faces, resulting in more natural and realistic facial images. Compared with previous work, experiments show superior performance in aging accuracy, identity preservation, and image fidelity on two datasets, both qualitatively and quantitatively. In addition, the proposed network structure can also be applied to other multi-class domain transfer tasks to avoid group-based training and achieve more accurate continuous modeling. As mentioned earlier, an example is the smile effect applied to faces. Continuous effect estimators (e.g., C112), instead of age estimators, include smile degree effect estimators.

[0155] Actual implementations may include any or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, devices, systems, components, program products, and other combinations of features described herein for performing the functions. A number of embodiments have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. Furthermore, other steps may be provided from the described processes, or steps may be eliminated, and other components may be added to or removed from the described systems. Therefore, other embodiments are within the scope of the appended claims.

[0156] Throughout the description and claims of this specification, the words "comprise" and "comprising" and their variations mean "including but not limited to", and are not intended to (and do not) exclude other components, integers or steps. Throughout this specification, the singular encompasses the plural, unless the context requires otherwise. In particular, where the indefinite article is used, the specification should be understood as contemplating plurality as well as singularity, unless the context requires otherwise.

[0157] Features, integers, characteristics, or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood to be applicable to any other aspect, embodiment or example unless incompatible therewith. All features disclosed herein (including any accompanying claims, abstract and drawings) and / or all steps of any method or process so disclosed may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of any foregoing examples or embodiments. The invention extends to any novel one or any novel combination of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one or any novel combination of the steps of any method or process disclosed.

[0158] References

[0159] In the snow-covered area, a slightly less expensive snowflake is available.

[0160] [1]Grigory Antipov,Moez Baccouche,and Jean-Luc Dugelay.Face agingwith conditional generative adversarial networks.In 2017IEEE InternationalConference on Image Processing(ICIP),pages 2089–2093.IEEE,2017.

[0161] [2]Yosuke Bando,Takaaki Kuratate,and Tomoyuki Nishita.A simple methodfor modeling wrinkles on human skin.In Pacific Conference on Computer Graphics and Applications, pages 166–175.Citeseer,2002.

[0162] [3]Laurence Boissieux,Gergo Kiss,Nadia Magnenat Thalmann,and PremKalra.Simulation of skin aging and wrinkles with cosmetics insight.InComputer Animation and Simulation 2000,pages 15–27.Springer,2000.

[0163] [4]D Michael Burt and David I Perrett.Perception of age in adultcaucasian male faces:Computer graphic manipulation of shape and colourinformation.Proceedings of the Royal Society of London.Series B:BiologicalSciences,259(1355):137–143,1995.

[0164] [5]Bor-Chun Chen,Chu-Song Chen,and Winston H Hsu.Cross-age referencecoding for age-invariant face recognition and retrieval.In Europeanconference on computer vision,pages 768–783.Springer,2014.

[0165] [6]Yunjey Choi,Minje Choi,Munyoung Kim,Jung-Woo Ha,Sunghun Kim,andJaegul Choo.Stargan:Unified generative adversarial networks for multi-domainimage-to-image translation.In Proceedings of the IEEE conference on computervision and pattern recognition,pages 8789–8797,2018.

[0166] [7]Yunjey Choi,Youngjung Uh,Jaejun Yoo,and Jung-Woo Ha.Stargan v2:Diverse image synthesis for multiple domains.In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,pages 8188–8197,2020.

[0167] [8]Harm de Vries,Florian Strub,Jeremie Mary,Hugo Larochelle,OlivierPietquin,and Aaron C Courville.Modulating early visual processing bylanguage.In I.Guyon,U.V.Luxburg,S.Bengio,H.Wallach,R.Fergus,S.Vishwanathan,and R.Garnett,editors,Advances in Neural Information Processing Systems 30,pages 6594–6604.Curran Associates,Inc.,2017.

[0168] [9]Ian J Goodfellow,Jean Pouget-Abadie,Mehdi Mirza,Bing Xu,DavidWarde-Farley,Sherjil Ozair,Aaron Courville,and Yoshua Bengio.Generativeadversarial networks.Advances in neural information processing systems,3(06),2014.

[0169]

[10] Zhenliang He,Meina Kan,Shiguang Shan,and Xilin Chen.S2gan:Shareaging factors across ages and share aging trends among individuals.InProceedings of the IEEE International Conference on Computer Vision,pages9440–9449,2019.

[0170]

[11] Zhenliang He,Wangmeng Zuo,Meina Kan,Shiguang Shan,and XilinChen.Attgan:Facial attribute editing by only changing what you want.IEEETransactions on Image Processing,28(11):5464–5478,2019.

[0171]

[12] Martin Heusel,Hubert Ramsauer,Thomas Unterthiner,BernhardNessler,and Sepp Hochreiter.Gans trained by a two time-scale update ruleconverge to a local nash equilibrium.In Advances in neural informationprocessing systems,pages 6626–6637,2017.

[0172]

[13] Xun Huang and Serge Belongie.Arbitrary style transfer in real-time with adaptive instance normalization.In ICCV,2017.

[0173]

[14] Phillip Isola,Jun-Yan Zhu,Tinghui Zhou,and Alexei A Efros.Image-to-image translation with conditional adversarial networks.CVPR,2017.

[0174]

[15] Tero Karras,Samuli Laine,and Timo Aila.A style-based generatorarchitecture for generative adversarial networks.In Proceedings of the IEEEconference on computer vision and pattern recognition,pages 4401–4410,2019.

[0175]

[16] Ira Kemelmacher-Shlizerman,Supasorn Suwajanakorn,and Steven MSeitz.Illumination-aware age progression.In Proceedings of the IEEEconference on computer vision and pattern recognition,pages 3334–3341,2014.

[0176]

[17] Andreas Lanitis,Christopher J.Taylor,and Timothy F.Cootes.Towardautomatic simulation of aging effects on face images.IEEE Transactions onpattern Analysis and machine Intelligence,24(4):442–455,2002.

[0177]

[18] Peipei Li,Yibo Hu,Qi Li,Ran He,and Zhenan Sun.Global and localconsistent age generative adversarial networks.In 2018 24th InternationalConference on Pattern Recognition(ICPR),pages 1073–1078.IEEE,2018.

[0178]

[19] Ming Liu,Yukang Ding,Min Xia,Xiao Liu,Errui Ding,Wangmeng Zuo,andShilei Wen.Stgan:A unified selective transfer network for arbitrary imageattribute editing.In Proceedings of the IEEE conference on computer visionand pattern recognition,pages 3673–3682,2019.

[0179]

[20] Si Liu,Yao Sun,Defa Zhu,Renda Bao,Wei Wang,Xiangbo Shu,andShuicheng Yan.Face aging with contextual generative adversarial nets.InProceedings of the 25th ACM international conference on Multimedia,pages 82–90,2017.

[0180]

[21] Yunfan Liu,Qi Li,and Zhenan Sun.Attribute-aware face aging withwavelet-based generative adversarial networks.In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,pages 11877–11886,2019.

[0181]

[22] Mehdi Mirza and Simon Osindero.Conditional generative adversarialnets.arXiv preprint arXiv:1411.1784,2014.

[0182]

[23] Augustus Odena,Christopher Olah,and Jonathon Shlens.Conditionalimage synthesis with auxiliary classifier gans.In International conference onmachine learning,pages 2642–2651,2017.

[0183]

[24] Roy Or-El,Soumyadip Sengupta,Ohad Fried,Eli Shechtman,and IraKemelmacher-Shlizerman.Lifespan age transformation synthesis.In Proceedingsof the European Conference on Computer Vision(ECCV),2020.

[0184]

[25] Hongyu Pan,Hu Han,Shiguang Shan,and Xilin Chen.Mean-variance lossfor deep age estimation from a face.In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition,pages 5285–5294,2018.

[0185]

[26] Taesung Park,Ming-Yu Liu,Ting-Chun Wang,and Jun-Yan Zhu.Gaugan:semantic image synthesis with spatially adaptive normalization.In ACMSIGGRAPH 2019 Real-Time Live!2019.

[0186]

[27] Narayanan Ramanathan and Rama Chellappa.Modeling age progressionin young faces.In 2006 IEEE Computer Society Conference on Computer Visionand Pattern Recognition(CVPR’06),volume 1,pages 387–394.IEEE,2006.

[0187]

[28] Narayanan Ramanathan and Rama Chellappa.Modeling shape andtextural variations in aging faces.In 2008 8th IEEE International Conferenceon Automatic Face&Gesture Recognition,pages 1–8.IEEE,2008.

[0188]

[29] Karl Ricanek and Tamirat Tesafaye.Morph:A longitudinal imagedatabase of normal adult age-progression.In 7th International Conference onAutomatic Face and Gesture Recognition(FGR06),pages 341–345.IEEE,2006.

[0189]

[30] Rasmus Rothe,Radu Timofte,and Luc Van Gool.Dex:Deep expectationof apparent age from a single image.In Proceedings of the IEEE internationalconference on computer vision workshops,pages 10–15,2015.

[0190]

[31] Duncan A Rowland and David I Perrett.Manipulating facialappearance through shape and color.IEEE computer graphics and applications,15(5):70–76,1995.

[0191]

[32] Yujun Shen,Jinjin Gu,Xiaoou Tang,and Bolei Zhou.Interpreting thelatent space of gans for semantic face editing.In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,pages 9243–9252,2020.

[0192]

[33] Xiangbo Shu,Jinhui Tang,Hanjiang Lai,Luoqi Liu,and ShuichengYan.Personalized age progression with aging dictionary.In Proceedings of theIEEE international conference on computer vision,pages 3970–3978,2015.

[0193]

[34] Jinli Suo,Xilin Chen,Shiguang Shan,Wen Gao,and Qionghai Dai.Aconcatenational graph evolution aging model.IEEE transactions on patternanalysis and machine intelligence,34(11):2083–2096,2012.

[0194]

[35] Jinli Suo,Feng Min,Songchun Zhu,Shiguang Shan,and Xilin Chen.Amulti-resolution dynamic model for face aging simulation.In 2007 IEEEConference on Computer Vision and Pattern Recognition,pages 1–8.IEEE,2007.

[0195]

[36] Jinli Suo,Song-Chun Zhu,Shiguang Shan,and Xilin Chen.Acompositional and dynamic model for face aging.IEEE Transactions on PatternAnalysis and Machine Intelligence,32(3):385–401,2009.

[0196]

[37] Bernard Tiddeman,Michael Burt,and David Perrett.Prototyping andtransforming facial textures for perception research.IEEE computer graphicsand applications,21(5):42–50,2001.

[0197]

[38] James T Todd,Leonard S Mark,Robert E Shaw,and John BPittenger.The perception of human growth.Scientific American,242(2):132–145,1980.

[0198]

[39] Wei Wang, Zhen Cui, Yan Yan, Jiashi Feng, Shuicheng Yan, Xiangbo Shu, and Nicu Sebe. Recurrent face aging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2378-2386, 2016.

[0199]

[40] Wei Wang, Yan Yan, Stefan Winkler, and Nicu Sebe. Category specific dictionary learning for attribute specific feature selection. IEEE Transactions on Image Processing, 25(3): 1465-1478, 2016.

[0200]

[41] Zongwei Wang, Xu Tang, Weixin Luo, and Shenghua Gao. Face aging with identity-preserved conditional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7939-7947, 2018.

[0201]

[42] Yin Wu, Nadia Magnenat Thalmann, and Daniel Thalmann. A plastic-visco-elastic model for wrinkles in facial animation and skin aging. In Fundamentals of Computer Graphics, pages 201-213. World Scientific, 1994.

[0202]

[43] Hongyu Yang, Di Huang, Yunhong Wang, and Anil K Jain. Learning face age progression: A pyramid architecture of gans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 31-39, 2018.

[0203]

[44] Hongyu Yang, Di Huang, Yunhong Wang, Heng Wang, and Yuanyan Tang. Face aging effect simulation using hidden factor analysis joint sparse representation. IEEE Transactions on Image Processing, 25(6):2493-2507, 2016.

[0204]

[45] Xu Yao, Gilles Puy, Alasdair Newson, Yann Gousseau, and Pierre Hellier. High resolution face age editing. arXiv preprint arXiv:2005.04410, 2020.

[0205]

[46] Zhifei Zhang, Yang Song, and Hairong Qi. Age progression / regression by conditional adversarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5810-5818, 2017.

[0206]

[47] Jun-Yan Zhu,Taesung Park,Phillip Isola,and Alexei AEfros.Unpaired image-to-image translation using cycleconsistent adversarialnetworks.In Proceedings of the IEEE international conference on computervision,pages 2223–2232,2017.

Claims

1. An image processing method, comprising: providing a unified age simulation model to generate new images of a subject at a target age from an input image of the subject; wherein the unified age simulation model provides a plurality of corresponding model-estimated age embeddings at each of a plurality of continuous ages representing continuous aging information, the model-estimated age embeddings being learned by joint training of a generator and an age estimator embedded in an encoder-decoder architecture of the model, the age estimator being configured to determine a model-estimated age of a subject in response to features generated from a corresponding encoder in response to a corresponding input image; and wherein the generator generates the new image from the input image using features generated by the encoder, the input image being transformed by a corresponding embedding in the model-estimated age embedding determined based on the target age and the model-estimated age of the subject.

2. The image processing method according to claim 1, wherein: The encoder-decoder architecture includes the age estimator to estimate a model-estimated age of the object in the input image.

3. The image processing method according to claim 2, wherein: An encoder of the model processes the input image to determine encoder-generated features, and wherein the age estimator processes the encoder-generated features to determine an age estimated by the model.

4. The image processing method according to claim 2, wherein: Transforming the features generated by the encoder by a personalized age embedding, wherein the personalized age embedding comprises: a. a corresponding embedding of the model-estimated age embedding determined according to the model-estimated age; and b. The corresponding embedding of the model-estimated age embedding determined according to the target age.

5. The image processing method according to claim 4, wherein: The personalized age embedding includes: a. in response to the model-estimated age, a personalized residual age embedding determined from the plurality of corresponding model-estimated age embeddings to preserve identity information of the subject; and b. An exemplary age embedding comprising the corresponding embedding of the model-estimated age embedding determined from the target age to represent aging patterns shared across the population.

6. The image processing method according to claim 4 or 5, wherein: Apply the personalized age embedding according to the affine transformation.

7. The image processing method according to claim 4 or 5, wherein: The generator processes features generated by the encoder through the personalized age embedding transform to generate the new image of the target age.

8. The image processing method according to any one of claims 1 to 5, wherein: The model is one or both of a deep learning neural network model and a generative adversarial network model.

9. The image processing method according to any one of claims 1 to 5, comprising providing a recommendation interface to obtain recommendations for one or both of products and services.

10. The image processing method according to any one of claims 1 to 5, comprising providing an e-commerce purchase interface for purchasing one or both of the product and service.

11. An image processing method according to any one of claims 1 to 5, comprising receiving the input image and providing the new image for display.

12. The image processing method according to any one of claims 1 to 5, wherein: Each of the input image and the new image includes a face of the subject.

13. An image processing method, comprising: Providing a unified model to generate, from an input image of an object, a new image of a target class that is a continuous representation of the object; wherein the model provides a plurality of corresponding model-estimated class embeddings at each of a plurality of continuous classes representing continuous effect information, the model-estimated class embeddings being learned by joint training of a generator and a class estimator embedded in an encoder-decoder architecture of the model, the class estimator being configured to determine the model-estimated class of the corresponding object in response to features generated from the corresponding encoder in response to the corresponding input image; and wherein the generator generates the new image from the input image using the features generated by the encoder, the input image being transformed by a corresponding class embedding in the model-estimated class embedding determined according to the target class of the object and the model-estimated class.

14. An image processing method, comprising: providing a domain transfer model to transfer an input image to a new image, applying a continuous effect to transform the input to a target class for a plurality of continuous classes of the continuous effect using a plurality of corresponding model-estimated class embeddings learned by the model for each of the plurality of continuous classes of the continuous effect; as well as The input image is transferred to the new image using the domain transfer model.

15. The image processing method according to claim 14, wherein: The continuous effect includes an aging effect, the plurality of corresponding model-estimated class embeddings include corresponding model-estimated age embeddings, and the target class includes a target age.

16. The image processing method according to claim 15, wherein: When transferring the input image, the domain transfer model operates to: generating an encoding feature of the input image; The encoded features are converted using the following features: a personalized residual age embedding determined from the plurality of corresponding model-estimated age embeddings in response to the model-estimated age of the subject in the input image to preserve identity information of the subject; as well as exemplary age embeddings, including corresponding embeddings of the model-estimated residual age embeddings determined from the target age to represent aging patterns shared across the population; and The new image is generated using the transformed encoded features.

17. The image processing method according to claim 16, wherein: The model includes an age estimator to determine an age estimated by the model.

18. The image processing method according to claim 17, wherein: The age estimator comprises a classifier trained with an encoder, the encoder configured to produce the encoded features, and wherein the age estimator is trained to determine a corresponding model-estimated age of an object in a new image using corresponding encoded features encoded by the encoder.

19. The image processing method according to claim 18, wherein: The model-estimated age embeddings are defined during training of the age estimator together with the encoder, associating corresponding ones of the model-estimated age embeddings with the corresponding model-estimated ages.

20. An image processing method according to any one of claims 14 to 19, comprising providing a recommendation for at least one of products and services associated with the continuous effect.

21. The image processing method according to claim 20, wherein: The recommendation is generated in response to one or both of a skin analysis of the input image and a user preference input.

22. The image processing method according to claim 20, wherein: A target age is determined in response to the recommendation.

23. The image processing method of claim 20, comprising providing an annotation image generated from the input image to present the recommendation.

24. The image processing method of claim 20, comprising providing an e-commerce interface for purchasing products, services, or both.

25. The image processing method according to claim 20, wherein: The continuous effect is an aging effect; the product includes one of a skin rejuvenation product, an anti-aging product, and a cosmetic product; and the service includes one of a skin rejuvenation service, an anti-aging service, and a cosmetic service.

26. A computing device comprising a processing unit and a storage device coupled thereto, the storage unit storing instructions which, when executed by the processing unit, configure the computing device to perform the method according to any one of the preceding method claims.

27. A computer program product comprising a non-transitory storage device storing instructions which, when executed by a processing unit of a computing device, configure the computing device to perform the image processing method according to any one of claims 1 to 26.

28. A computing device comprising a processing unit and a storage device coupled thereto, the storage unit storing instructions that, when executed by the processing unit, configure the computing device to: a. provide a recommendation for at least one of the products and services; and b. Providing an age-simulated image, the age-simulated image comprising a new image generated from the input image and the target age, wherein, The new image is generated according to any one of claims 1 to 26.

Images