Systems and methods for facial image manipulation

Abstract

A method, performed on a computer, for manipulating one or more attributes of a face depicted in an input image. For example, the attributes may comprise a facial expression of the face, an orientation of the face, a lighting arrangement (e.g. the location(s) of the light source(s) that light the face depicted in the image, the intensity of the lights, the colour of the lights, etc.), eye gaze direction(s), presence of eyewear (e.g. glasses, goggles, sunglasses, etc.) and / or the like.

Description

SYSTEMS AND METHODS FOR FACIAL IMAGE MANIPULATIONReference to Related Applications

[0001] This application claims priority from, and for the purposes of the United States of America the benefit of 35 USC 119 in relation to, Unites States patent application No. 63 / 584495 filed 21 September 2023 which is hereby incorporated herein by reference.Technical Field

[0002] This application is directed to systems and methods for computer simulation of faces. More particularly, this application is directed to systems and methods for dynamic manipulation of one or more aspects of an image depicting a face of a person or character such as, for example, the facial expression, the head orientation, the lighting, the eye gaze direct, the presence of eyewear, the identity of the person or character depicted and / or the like.Background

[0003] There is a desire to manipulate of one or more aspects of an image depicting a face of a person or character such as, for example, the facial expression, the head orientation, the lighting, the eye gaze direction, the presence of eyewear, the identity of the person or character depicted and / or the like.

[0004] There is also a desire to manipulate one or more aspects of an image of a face (e.g. the head orientation, the facial expression, the lighting, the identity of the person or character and / or the like) while preserving remaining aspects of the original image. For example, it may be desirable to change the head orientation in an image of a face without changing the facial expression, identity or lighting.

[0005] In some circumstances, there may be desirability that such manipulation occurs smoothly (e.g. over successive frames of video images) to the perception of a viewer.

[0006] In some circumstances, it may be desirable that such manipulation be temporally consistent (i.e. without perceptible artifacts) in the context of a series of image frames associated with video or animation.

[0007] The foregoing examples of the related art and limitations related thereto are intended to be illustrative and not exclusive. Other limitations of the related art will become apparent to those of skill in the art upon a reading of the specification and a study of the drawings.Summary

[0008] The following embodiments and aspects thereof are described and illustrated in conjunction with systems, tools and methods which are meant to be exemplary and illustrative, not limiting in scope. In various embodiments, one or more of the abovedescribed problems have been reduced or eliminated, while other embodiments are directed to other improvements.

[0009] One aspect of the invention provides a method for manipulating one or more attributes (e.g. facial expression, head orientation and / or lighting orientation) of a depiction of a face in an input image. The method may be performed on a computer. The method may comprise preparing training data. Preparing the training data may comprise preparing one or more non-CG image subsets each comprising non-CG images of a face of a non-CG identity and all having at least one common non-CG attribute and preparing one or more CG image subsets each comprising CG images of a face of a CG identity and all having at least one common CG attribute. The method may comprise training a model comprising first and second autoencoders by training the first autoencoder with the one or more non-CG image subsets by, for each of the one or more non-CG image subsets, causing a first encoder of the first autoencoder to create first latent codes, each representative of one non- CG image of a current non-CG image subset, replacing first components of the first latent codes with first mean values to create first modified latent codes, causing a first decoder of the first autoencoder to create reconstructed non-CG images based on the first modified latent codes and updating trainable parameters of the first autoencoder based at least in part on the reconstructed non-CG images and the first mean values. The method may comprise training the second autoencoder with the one or more CG image subsets by, for each of the one or more CG image subsets, causing a second encoder of the second autoencoder to create second latent codes, each representative of one CG image of a current CG image subset wherein the first encoder and the second encoder comprise a shared set of encoder trainable neural-network parameters, replacing second componentsof the second latent codes with second mean values to create second modified latent codes, causing a second decoder of the second autoencoder to create reconstructed CG images based on the second modified latent codes and updating trainable parameters of the second autoencoder based at least in part on the reconstructed CG images and the second mean values. The method may comprise receiving an input image depicting an input face of one of the non-CG identity and the CG identity, receiving donor latent code components representative of one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation). The method may comprise processing the input image with the first encoder or the second encoder to obtain unmodified latent code, replacing one or more components of the unmodified latent code with the donor latent code components to obtain modified input latent code and employing a decoder of the model to infer an output image based on the modified input latent code wherein the output image corresponds to the input image except in that the output image depicts the desired attributes (e.g. facial expression, head orientation and / or lighting orientation).

[0010] In some embodiments, employing a decoder of the model to infer an output image based on the modified input latent code comprises employing the first decoder to infer an output non-CG image based on the modified input latent code wherein the non-CG output image corresponds to the input image except in that the non-CG output image depicts the desired attributes (e.g. facial expression, head orientation and / or lighting orientation).

[0011] In some embodiments, employing a decoder of the model to infer an output image based on the modified input latent code comprises employing the second decoder to infer an output CG image based on the modified input latent code wherein the CG output image corresponds to the input image except in that the CG output image depicts the desired attributes (e.g. facial expression, head orientation and / or lighting orientation).

[0012] In some embodiments, the input image comprises a CG input image. In some embodiments, the input image comprises a non-CG input image.

[0013] In some embodiments, in the one or more non-CG image subsets comprises a plurality of non-CG image subsets and the plurality of non-CG image subsets comprises one or more controlled lighting (“CL”) image subsets each comprising CL images of a face of the non-CG identity and all having at least one common CL attribute and one or more real image subsets each comprising real images of a face of the non-CG identity and all having at least one common real attribute.

[0014] In some embodiments, preparing the one or more CG image subsets comprisesrendering the CG images from a CG animation rig and based at least in part on a desired distribution of facial expressions and head orientations. In some embodiments, preparing the one or more CG image subsets comprises rendering a first set of CG images from a CG animation rig and based at least in part on a desired distribution of facial expressions and head orientations and then re-lighting the first set of CG images to obtain the CG images. In some embodiments, re-lighting the first set of CG images to obtain the CG images comprises re-lighting the first set of CG images based at least in part on spherical harmonics data.

[0015] In some embodiments, preparing the one or more non-CG image subsets comprises cropping and aligning input non-CG images and preparing the one or more CG image subsets comprises cropping and aligning input CG images.

[0016] In some embodiments, preparing the one or more non-CG image subsets comprises sorting the non-CG input images into single variable non-CG subsets wherein only a single attribute (e.g. facial expression, head orientation or lighting) varies, multi-variable non-CG subsets wherein at least two attributes (e.g. facial expression, head orientation and / or lighting) varies and / or variable non-CG subsets wherein attributes (e.g. facial expression, head orientation and / or lighting orientation) are randomly distributed and preparing the one or more CG image subsets comprises sorting the CG input images into single variable CG subsets wherein only a single attribute (e.g. facial expression, head orientation or lighting) varies, multi-variable CG subsets wherein at least two attributes (e.g. facial expression, head orientation and / or lighting) varies and / or variable CG subsets wherein attributes (e.g. facial expression, head orientation and / or lighting orientation) are randomly distributed.

[0017] In some embodiments, the method comprises choosing the first components and the second components based at least in part on a desired format of the unmodified latent code wherein the desired format divides the unmodified latent code into subsets of components each corresponding to a particular attribute (e.g. facial expression, head orientation or lighting).

[0018] In some embodiments, the method comprises determining the first mean values by, for each of the first components determining a mean of values of a corresponding component across the first latent codes and determining the second mean values by, for each of the second components determining a mean of values of a corresponding component across the second latent codes.

[0019] In some embodiments, the first decoder and the second decoder each share one ormore identity independent layers of trainable parameters and updating the trainable parameters of the first autoencoder based at least in part on the reconstructed non-CG images and the first mean values comprises updating the one or more identity independent layers of trainable parameters of the second decoder based at least in part on the reconstructed non-CG images and the first mean values and updating the trainable parameters of the second autoencoder based at least in part on the reconstructed CG images and the second mean values comprises updating the one or more identity independent layers of trainable parameters of the first decoder based at least in part on the reconstructed CG images and the second mean values.

[0020] In some embodiments, updating the trainable parameters of the first autoencoder based at least in part on the reconstructed non-CG images and the first mean values comprises also updating the second encoder based at least in part on the reconstructed non-CG images and the first mean values and updating the trainable parameters of the second autoencoder based at least in part on the reconstructed CG images and the second mean values comprises also updating the first encoder based at least in part on the reconstructed CG images and the second mean values.

[0021] In some embodiments, updating the trainable parameters of the first autoencoder based at least in part on the reconstructed non-CG images and the first mean values comprises determining a first image loss function based on a comparison of the reconstructed non-CG images to their corresponding non-CG images and determining a first target loss function based on a comparison of the first mean values and the corresponding components of the first latent code and updating the trainable parameters of the first autoencoder based at least in part on the first image loss function and the first target loss function and updating the trainable parameters of the second autoencoder based at least in part on the reconstructed CG images and the second mean values comprises determining a second image loss function based on a comparison of the reconstructed CG images to their corresponding CG images and determining a second target loss function based on a comparison of the second mean values and the corresponding components of the second latent code and updating the trainable parameters of the second autoencoder based at least in part on the second image loss function and the second target loss function.

[0022] In some embodiments, receiving the donor latent code components representative of the one or more desired attributes (e.g. facial expression, head orientation and / or lightingorientation) comprises obtaining the donor latent code components representative of the one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation) by processing a donor image with the first encoder or the second encoder to obtain donor latent code and the donor latent code components are extracted from the donor latent code.

[0023] In some embodiments, the donor image comprises a CG donor image. In some embodiments, the donor image comprises a non-CG donor image.

[0024] In some embodiments, receiving the donor latent code components representative of the one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation) comprises obtaining the donor latent code components representative of the one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation) by training a second model to map a relationship between descriptions of a first attribute (e.g. facial expression, head orientation or lighting orientation)and corresponding training latent code components and inputting a description of at least one of the one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation) into the second model to obtain at least some of the donor latent code components.

[0025] In some embodiments, the method comprises obtaining the corresponding training latent code components by processing images representative of the descriptions of the first attribute (e.g. facial expression, head orientation or lighting orientation)with the first encoder.

[0026] In some embodiments, the images representative of the descriptions of the first attribute (e.g. facial expression, head orientation or lighting orientation)are obtained from amongst the non-CG image subsets or the CG image subsets.

[0027] In some embodiments, receiving the donor latent code components representative of the one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation) comprises obtaining donor latent code components representative of a lighting arrangement by training a second model to map a relationship between lighting arrangement descriptions and corresponding training latent code components and inputting a lighting description into the second model to obtain at least some of the donor latent code components.

[0028] In some embodiments, the method comprises obtaining the corresponding training latent code components by processing training CL images which vary only by lighting arrangement with the first encoder and wherein the lighting arrangement descriptionscomprise descriptions of the training CL images. In some embodiments, the lighting arrangement descriptions comprises panoramic light maps.

[0029] In some embodiments, preparing the training data comprises preparing one or more second non-CG image subsets each comprising second non-CG images of a second face of a second non-CG identity and all having at least one common second non-CG attribute and preparing one or more second CG image subsets each comprising second CG images of a second face of a second CG identity and all having at least one common second CG attribute. The model may comprise third and fourth autoencoders and training the model comprises training the third autoencoder with the one or more second non-CG image subsets by, for each of the one or more second non-CG image subsets, causing a third encoder of the third autoencoder to create third latent codes, each representative of one second non-CG image of a current second non-CG image subset wherein the first encoder and the third encoder comprise the shared set of encoder trainable neural-network parameters, replacing third components of the third latent codes with third mean values to create third modified latent codes, causing a third decoder of the third autoencoder to create reconstructed second non-CG images based on the third modified latent codes and updating trainable parameters of the first autoencoder based at least in part on the second reconstructed non-CG images and the third mean values. Training the model may also comprise training the fourth autoencoder with the one or more second CG image subsets by, for each of the one or more second CG image subsets, causing a fourth encoder of the fourth autoencoder to create fourth latent codes, each representative of one second CG image of a current second CG image subset wherein the first encoder and the fourth encoder comprise the shared set of encoder trainable neural-network parameters, replacing fourth components of the fourth latent codes with fourth mean values to create fourth modified latent codes, causing a fourth decoder of the fourth autoencoder to create second reconstructed CG images based on the fourth modified latent codes and updating trainable parameters of the fourth autoencoder based at least in part on the reconstructed second CG images and the fourth mean values.

[0030] In some embodiments, employing a decoder of the model to infer an output image based on the modified input latent code comprises employing the third decoder to infer an output non-CG image of the second identity based on the modified input latent code wherein the non-CG output image corresponds to the input image except in that the non-CG output image of the second identity depicts the desired attributes (e.g. facial expression,head orientation and / or lighting orientation).

[0031] In some embodiments, employing a decoder of the model to infer an output image based on the modified input latent code comprises employing the fourth decoder to infer an output CG image of the second identity based on the modified input latent code wherein the CG output image corresponds to the input image except in that the CG output image of the second identity depicts the desired attributes (e.g. facial expression, head orientation and / or lighting orientation).

[0032] In some embodiments, the input image comprises a CG input image of the first identity. In some embodiments, the input image comprises a CG input image of the second identity. In some embodiments, the input image comprises a non-CG input image of the first identity. In some embodiments, the input image comprises a non-CG input image of the second identity.

[0033] Another aspect of the invention provides a method for training a model for manipulating one or more attributes (e.g. facial expression, head orientation and / or lighting orientation) of a depiction of a face in an input image. The method may be performed on a computer. The method may comprise preparing training data. Preparing the training data may comprise preparing one or more non-CG image subsets each comprising non-CG images of a face of a non-CG identity and all having at least one common non-CG attribute (e.g. facial expression, head orientation or lighting orientation), preparing one or more CG image subsets each comprising CG images of a face of a CG identity and all having at least one common CG attribute (e.g. facial expression, head orientation or lighting orientation). The method may comprise training a model comprising first and second autoencoders by training the first autoencoder with the one or more non-CG image subsets by, for each of the one or more non-CG image subsets, causing a first encoder of the first autoencoder to create first latent codes, each representative of one non-CG image of a current non-CG image subset, replacing first components of the first latent codes with first mean values to create first modified latent codes, causing a first decoder of the first autoencoder to create reconstructed non-CG images based on the first modified latent codes and updating trainable parameters of the first autoencoder based at least in part on the reconstructed non-CG images and the first mean values. The method may comprise training the second autoencoder with the one or more CG image subsets by, for each of the one or more CG image subsets, causing a second encoder of the second autoencoder to create second latent codes, each representative of one CG image of a current CG image subset whereinthe first encoder and the second encoder comprise a shared set of encoder trainable neural-network parameters, replacing second components of the second latent codes with second mean values to create second modified latent codes, causing a second decoder of the second autoencoder to create reconstructed CG images based on the second modified latent codes and updating trainable parameters of the second autoencoder based at least in part on the reconstructed CG images and the second mean values.

[0034] In some embodiments, preparing the training data comprises preparing one or more second non-CG image subsets each comprising second non-CG images of a second face of a second non-CG identity and all having at least one common second non-CG attribute (e.g. facial expression, head orientation or lighting orientation), preparing one or more second CG image subsets each comprising second CG images of a second face of a second CG identity and all having at least one common second CG attribute (e.g. facial expression, head orientation or lighting orientation), wherein the model comprises third and fourth autoencoders. Training the model may comprise, training the third autoencoder with the one or more second non-CG image subsets by, for each of the one or more second non-CG image subsets, causing a third encoder of the third autoencoder to create third latent codes, each representative of one second non-CG image of a current second non-CG image subset wherein the first encoder and the third encoder comprise the shared set of encoder trainable neural-network parameters, replacing third components of the third latent codes with third mean values to create third modified latent codes, causing a third decoder of the third autoencoder to create reconstructed second non-CG images based on the third modified latent codes and updating trainable parameters of the first autoencoder based at least in part on the second reconstructed non-CG images and the third mean values. The method may comprise training the fourth autoencoder with the one or more second CG image subsets by, for each of the one or more second CG image subsets, causing a fourth encoder of the fourth autoencoder to create fourth latent codes, each representative of one second CG image of a current second CG image subset wherein the first encoder and the fourth encoder comprise the shared set of encoder trainable neural-network parameters; replacing fourth components of the fourth latent codes with fourth mean values to create fourth modified latent codes, causing a fourth decoder of the fourth autoencoder to create second reconstructed CG images based on the fourth modified latent codes and updating trainable parameters of the fourth autoencoder based at least in part on the reconstructed second CG images and the fourth mean values.

[0035] In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the drawings and by study of the following detailed descriptions.

[0036] It is emphasized that the invention relates to all combinations of the above features and any other features described herein, even if these are recited in different claims or recited to depend from different claims.Brief Description of the Drawings

[0037] Exemplary embodiments are illustrated in referenced figures of the drawings. It is intended that the embodiments and figures disclosed herein are to be considered illustrative rather than restrictive.

[0038] Figure 1 is a broad schematic depiction of a method for manipulating one or more attributes of a face depicted in an image according to a particular embodiment.

[0039] Figure 2 depicts an exemplary system for performing one or more methods described herein (e.g. the method of Figure 1 ) according to a particular embodiment.

[0040] Figure 3 is a schematic depiction of a method for preparing the input data for the Figure 1 method according to a particular embodiment.

[0041] Figure 4 is a schematic depiction of a method of face cropping and alignment according to a particular embodiment.

[0042] Figure 5 is a schematic depiction of a method for training a model that may be used to implement the training of the model of the Figure 1 method according to a particular embodiment.

[0043] Figure 6 is a schematic depiction of a model that may be used as the model of the Figure 1 method according to a particular embodiment.

[0044] Figure 7A is a schematic depiction of a portion of a method for training a portion of a model that may be used to implement the training of the model of the Figure 1 method according to a particular embodiment. Figure 7B is a schematic depiction of another of a portion of a method for training a portion of a model that may be used to implement the training of the model of the Figure 1 method according to a particular embodiment. Figure7C is a schematic depiction of a portion of a method for training another portion of a model that may be used to implement the training of the model of the Figure 1 method according to a particular embodiment.

[0045] Figure 8 is a schematic depiction of latent code generated by an encoder of a model that may be used to implement the training of the model of the Figure 1 method according to a particular embodiment.

[0046] Figure 9 is a schematic depiction of a method for modifying latent code generated by an encoder of a model that may be used to implement the training of the model of the Figure 1 method according to a particular embodiment.

[0047] Figure 10 is a schematic depiction of a portion of a method for training a model that may be used to implement at least a portion of the training of the model of the Figure 1 method according to a particular embodiment.

[0048] Figure 11 is a schematic depiction of another method for training a model that may be used to implement the training of the model of the Figure 1 method according to a particular embodiment.

[0049] Figure 12 is a schematic depiction of a method for obtaining donor latent code that may be used as part of the Figure 1 method according to a particular embodiment.

[0050] Figure 13 is a schematic depiction of another method for obtaining donor latent code that may be used as part of the Figure 1 method according to a particular embodiment.

[0051] Figure 14 is a broad schematic depiction of a method for manipulating one or more attributes and / or the identity of a face depicted in an image according to a particular embodiment.

[0052] Figure 15 is a schematic depiction of a model that may be used as the model of the Figure 14 method according to a particular embodiment.Description

[0053] Throughout the following description specific details are set forth in order to provide a more thorough understanding to persons skilled in the art. However, well known elements may not have been shown or described in detail to avoid unnecessarily obscuring thedisclosure. Accordingly, the description and drawings are to be regarded in an illustrative, rather than a restrictive, sense.

[0054] One aspect of the invention provides a method, performed on a computer, for manipulating one or more attributes of a face depicted in an input image. For example, the attributes may comprise a facial expression of the face, an orientation of the face, a lighting arrangement (e.g. the location(s) of the light source(s) that light the face depicted in the image, the intensity of the lights, the colour of the lights, etc.), eye gaze direction(s), presence of eyewear (e.g. glasses, goggles, sunglasses, etc.) and / or the like. The method may comprise preparing training data. Preparing the training data may comprise preparing one or more non-CG image subsets each comprising non-CG images of a face of a non-CG identity and all having at least one common non-CG attribute and preparing one or more CG image subsets each comprising CG images of a face of a CG identity and all having at least one common CG attribute. The method may comprise training a model comprising first and second autoencoders by training the first autoencoder with the one or more non-CG image subsets by, for each of the one or more non-CG image subsets, causing a first encoder of the first autoencoder to create first latent codes, each representative of one non- CG image of a current non-CG image subset, replacing first components of the first latent codes with first mean values to create first modified latent codes, causing a first decoder of the first autoencoder to create reconstructed non-CG images based on the first modified latent codes and updating trainable parameters of the first autoencoder based at least in part on the reconstructed non-CG images and the first mean values. The method may comprise training the second autoencoder with the one or more CG image subsets by, for each of the one or more CG image subsets, causing a second encoder of the second autoencoder to create second latent codes, each representative of one CG image of a current CG image subset wherein the first encoder and the second encoder comprise a shared set of encoder trainable neural-network parameters, replacing second components of the second latent codes with second mean values to create second modified latent codes, causing a second decoder of the second autoencoder to create reconstructed CG images based on the second modified latent codes and updating trainable parameters of the second autoencoder based at least in part on the reconstructed CG images and the second mean values. The method may comprise receiving an input image depicting an input face of one of the non-CG identity and the CG identity, receiving donor latent codecomponents representative of one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation). The method may comprise processing the input image with the first encoder or the second encoder to obtain unmodified latent code, replacing one or more components of the unmodified latent code with the donor latent code components to obtain modified input latent code and employing a decoder of the model to infer an output image based on the modified input latent code wherein the output image corresponds to the input image except in that the output image depicts the desired attributes (e.g. facial expression, head orientation and / or lighting orientation).

[0055] Figure 1 is a broad schematic depiction of a method 10 for manipulating one or more attributes of a face depicted in an input image 42 according to a particular embodiment. Method 10 may be logically divided into a training portion 10A and an inference portion 10B.

[0056] Training portion 10A receives, as input, input data 12. Input data 12 may comprise a plurality of non-CG images (e.g. photographs of an actor) and a plurality of CG images (or a CG model or rig employable to create CG images or a CG animation from which CG images can be obtained).

[0057] Input data 12 may comprise controlled lighting (“CL”) images 12A. CL images 12A may comprise a plurality of images (e.g. frames of video) that exhibit the face of their corresponding identity in different controlled lighting conditions for each of one or more facial expressions and / or one or more head orientations. CL images 12A may be obtained using any suitable technique. CL images 12A may comprise so-called one-light-at-a-time images. CL images 12A may be captured by a single camera (e.g. a high resolution camera). In some embodiments, each image of CL images 12A may represent an actor lit by a different light or combination of lights (i.e. it is not strictly necessary that only one light is turned on when an image of CL images 12A is captured). In a particular example, an actor is positioned in front of an array of lights and, for a particular facial expression and head orientation, images are captured as the lights of the array of lights are turned on one at a time such that CL images 12A comprise a first set of images comprising one image of the actor for each light in the array. This may be repeated such that CL images 12A comprise additional sets of images for one or more additional head orientations and one or more additional facial expressions. For example, where an array of 172 lights is employed, CL images 12A could comprise 172 images (each lit with a different light of the array of lights) of a first facial expression with a first head orientation, 172 images (each lit with adifferent light of the array of lights) of a second facial expression with the first head orientation, 172 images (each lit with a different light of the array of lights) of the first facial expression with a second head orientation, etc.

[0058] Input data 12 may comprise CL data 12B. CL data 12B may comprise information corresponding to CL images 12A which identifies the lighting arrangement for each image of CL images 12A. For example, CL data 12B may identify which lights of a light array were on and which were off when each image of the CL images 12A was captured. CL data 12B may include location information which describes where each light of a light array is located (e.g. relative to the actor in CL images 12A). CL data 12B may be in any suitable form such as one or more databases, a plurality of panoramic light maps, etc.

[0059] Input data 12 may comprise real actor images 12C (also referred to herein as real images 12C). Real images 12C may comprise a plurality of non-CG images (e.g. frames of video) that exhibit the face of their corresponding identity (e.g. the same identity as CL images 12A). Real actor images 12C may be obtained using any suitable technique. Real actor images 12C may be obtained using a green screen, but this is not mandatory. Real actor images 12C may be obtained with constant or variable lighting conditions. By way of non-limiting example, real actor images 12C may comprise video footage of an actor executing a performance or a set of disjoint images of an actor executing a performance, and / or the like. Real actor images 12C may be purposely created for method 10 or may be obtained from a previous performance by the actor. Real images 12C may include a performance by the actor with particular range of motion (ROM) exercises and / or visemes.

[0060] In some embodiments, input data 12 only comprises one set of non-CG images (e.g. either CL images 12A or real images 12C).

[0061] Input data 12D may comprise CG model 12D. CG model 12D may comprise an animation rig or the like which may be employed to produce a plurality of CG images 12D’ (e.g. frames of video) that exhibit the face of a CG character (which may have the same identity as the identity depicted in CL images 12A and real images 12C or which may be of an entirely different identity). CG images 12D’ may be obtained from CG model 12D using any suitable rendering technique. By way of non-limiting example, CG images 12D’ may comprise rendered CG animation images corresponding to a CG character in the form of successive animation frames or disjoint images, and / or the like. In some embodiments, CGmodel 12D is replaced with CG images 12D’ such that CG images 12D’ are input directly into method 10. While CG images 12D’ are described herein as being rendered from a rig or the like, this is not strictly necessary. For example, CG images 12D’ could be individually created by one or more artists.

[0062] It is currently preferable (but not necessary) that real actor images 12C and CG images 12D’ have somewhat similar distributions of facial expressions and head orientations. Such similar distributions can be obtained by asking the actor (from whom real actor images 12C are obtained) to perform particular range of motion (ROM) exercises and / or visemes and by generating corresponding ROM poses (frames) using the CG character from which CG images 12D’ are obtained. In some embodiments, CG images 12D’ are prepared from CG model 12D with a similar distribution of facial expressions and head orientations according to blocks 210, 220 and 230 of method 200, as discussed further herein.

[0063] Input data 12 may optionally comprise lighting data 12E which comprises data representative of various lighting arrangements. Such lighting arrangements could be random or such lighting arrangements may be chosen so as to be representative of various natural environments (e.g. spherical harmonics lighting data).

[0064] CL images 12A and real images 12C may be non-CG images of the same identity. In the context herein, the term “identity” may refer to the identity of a human actor, the identify of a CG model that is a likeness of a human character or the identity of a CG character generally. In some embodiments, CG model 12D is employable to generate CG images 12D’ of a CG version of the identity of CL images 12A and real images 12C (e.g. a CG identity that looks like and / or shares characteristics with the identity of CL images 12A and real images 12C). However, this is not required and the identity of CG model 12D and / or depicted in CG images 12A may be different from the identity of the human actor in CL images 12A and real images 12C.

[0065] Training portion 10A of method 10 comprises training a neural-network-based model 32. Method 10 starts at block 20 which may involve preparation of input data 12. As described in more detail herein, training data preparation at block 20 may comprise processing input data 12 to provide training data 22 comprising subsets of training images that will be used during training at block 30. Block 20 training data preparation maycomprise any suitable training data preparation methods. For example, training data 22 may be prepared according to method 200 discussed further herein.

[0066] Method 10 then proceeds to block 30 which involves training of model 32. Training at block 30 may comprise unsupervised training of model 32. Trained model 32 (once trained at block 30) can be used to manipulate one or more attributes (e.g. facial expression, head orientation and / or lighting) of a face depicted in an input image 42. That is, trained model 32 can translate an image of a face into a corresponding image wherein one or more attributes (e.g. facial expression, head orientation and / or lighting) of the face are selectively modified while the remaining attributes are unchanged. Block 30 training may comprise any suitable training methods. For example, block 30 training may employ training method 400 and / or training method 900, discussed further herein.

[0067] Method 10 then proceeds to inference portion 10B. Inference portion 10B may be performed once for each input image 42. That is, where it is desired to employ method 10 for video input comprising a plurality of image frames, inference portion 10B may be performed once for each of the plurality of image frames of the video (each such image frame being a input image 42). For each instance of inference portion 10B, inference portion 10B receives, as input, an input image 42 depicting a face having a facial expression, a head orientation and a lighting arrangement (collectively attributes 4) and donor latent code 52 representative of desired attributes 4’ (e.g. desired facial expression 4A’, desired head orientation 4B’ and / or desired lighting 4C’). For simplicity, method 10 is only described herein for a limited number of attributes (e.g. facial expression, head orientation and lighting arrangement) but it should be understood that method 10 could additionally or alternatively be employed for other attributes such as, for example, eye gaze direction(s), presence of eyewear, etc.

[0068] Inference portion 10B starts at block 40 which receives, as input, input image 42 and processes input image 42 with an encoder (e.g. encoder 504, as discussed further herein) of trained model 32 to obtain latent code 44 representative of input image 42. In some embodiments, latent code 44 comprises a vector having a plurality of components.

[0069] Inference portion 10B then proceeds to block 50 which receives, as input, latent code 44 and donor latent code 52. Donor latent code 52 may be obtained at block 60 based on desired attributes 4’ by any suitable method such as, for example, methods 700 and / or800 described herein. Donor latent code 52 may comprise one or more donor latent code components representative of one or more desired attributes 4’ for output image 82 of method 10 (e.g. a desired facial expression 4A’, a desired head orientation 4B’ and / or a desired lighting 4C’).

[0070] At block 50, one or more components of latent code 44 are replaced with one or more components of donor latent code 52 to create modified latent code 54. Determining which component(s) of latent code 44 to replace with components of donor latent code 52 may be based at least in part on a format of latent code 44 (e.g. latent code format 432, as discussed further herein) which may be determined or set during training block 30 of method 10. For example, where donor latent code 52 is representative of desired lighting 4C’, the components of latent code 44 representative of lighting would be replaced by corresponding components of donor latent code 52.

[0071] Inference portion 10B then proceeds to block 80 which receives, as input, modified latent code 54. Block 80 employs a decoder (e.g. non-CG image decoder 506A or CG- image decoder 506B, as discussed further herein) of trained model 32 to process modified latent code 54 to thereby create output image 82 depicting a face having desired attributes 4’ while preserving any remaining attributes of attributes 4.

[0072] Some aspects of the invention provide a system 100 (an example embodiment of which is shown in Figure 2) for performing one or more methods described herein (e.g. method 10 of Figure 1 ) or portions thereof. System 100 may comprise a processor 102, a memory module 104, an input module 106, and an output module 108. Memory module 104 may store one or more of the networks and / or representations described herein. Processor 102 may receive (via input module 106) input data 12 and may store these inputs in memory module 104. Processor 102 may perform method 10 to train model 32 at block 30 as described herein, and store trained model 32 in memory module 104. Processor 102 may receive input image 42 (via input module 106) for example and may store such data in memory module 104. Processor 102 may receive desired attributes 4’ (via input module 106) for example and may store such data in memory module 104. Processor 102 may receive donor latent code 52 (via input module 106) for example and may store such data in memory module 104. Processor 102 may, at block 40, use trained model 32 (or a portion thereof such as an encoder thereof) to create latent code 44 based at least in part on input image 42. Processor 102 may, at block 50, modify latent code 44 with donor latent code 52to create modified latent code 54. Processor 102 may, at block 80, process modified latent code 54 using trained model 32 (or at least a portion thereof such as an identity-specific decoder) to create output image 82 which may in turn be stored in memory module 104 and / or be outputted via output module 108.

[0073] Figure 3 is a schematic depiction of a method 200 that may be used to implement the block 20 training data preparation of method 10 according to a particular embodiment. Method 200 may be performed in an automated manner by processor 102 of system 100 (Figure 2) or a similar processor or system. Method 200 receives, as input, input data 12. As discussed above, input data 12 may comprise CL images 12A, CL data 12B real images 12C, CG model 12D and optionally lighting data 12E. In some embodiments, input data 12 may comprise CG images 12D’ rather than CG model 12D thereby allowing method 200 to skip blocks 210, 220 and 230.

[0074] Method 200 may start at block 210 which receives, as input, real images 12C. At block 210, a distribution 212 of face orientations and facial expressions depicted in real images 12C is determined. Distribution 212 may be obtained at block 210 with off-the-shelf software such as, for example, Hopenet head pose estimation software or the like.

[0075] Method 200 then proceeds to block 220 which receives, as input, CG model 12D and distribution 212. At block 220, CG model 12D is employed to create rendered images 222 at least in part based on distribution 212. Rendered images 222 may depict the face of a CG character (which may correspond in identity to the identity of real images 12C but this is not required). The distribution of face orientations and facial expressions of rendered images 222 may substantially match distribution 212 of real images 12C. In some embodiments, rendered images 222 are rendered with variable lighting conditions such that some images of rendered images 222 have different lighting than others. In some embodiments, rendered images 222 are rendered with constant lighting conditions across all rendered images 222 (e.g. to reduce computational expense). Where rendered images 222 are rendered with constant lighting conditions, it may be desirable to replicate one or more of rendered images 222 under different lighting conditions at block 230 thereby effectively increasing the total number of rendered images 222 available to other parts of method 200.

[0076] Method 200 may optionally proceed to block 230 where one or more of rendered images 222 are replicated with new lighting conditions (e.g. re-lit). Rendered images 222 may be re-lit according to any suitable method. In some embodiments, rendered images 222 may be re-lit based at least in part on lighting data 12E. For example, in some embodiments, block 230 receives, as input, one or more rendered images 222 and lighting data 12E. Lighting data 12E may comprise spherical harmonics lighting data. At block 230, one or more rendered images 222 may each be re-lit under different lighting conditions based at least in part on lighting data 12E using methods known in the art to thereby create CG images 12D’ which comprise images depicting the face of a CG character with a distribution of face orientations and facial expressions that matches distribution 212 of real images 12C wherein one or more of the images from among CG images 12D’ is replicated in different lighting conditions. As compared to rendering rendered images 222 with variable lighting conditions at block 220, employing the block 230 step may involve less computational expense.

[0077] In some embodiments, such as where method 10 directly receives CG images 12D’, blocks 210, 220 may be skipped. In such cases, block 230 may or may not be employed, as desired.

[0078] Method 200 then proceeds to block 240 which receives, as input, CL images 12A, real images 12C and CG images 12D’. At block 240, CL images 12A, real images 12C and CG images 12D’ may be processed to crop the face depicted in each image and align the face in a canonical pose (e.g. by using detected landmarks). Block 240 may output CL face images 242A, aligned real face images 242B and aligned CG face images 242C (also referred to herein as CL faces 242A, real faces 242B and CG faces 242C, respectively). Block 240 may employ any suitable cropping and / or alignment methods for preparing cropped canonical front head poses from CL images 12A, real images 12C and CG images 12D’. In some embodiments, block 240 may use method 300, described further herein.

[0079] Method 200 may then proceed to optional block 245 which receives, as input, CL faces 242A, real faces 242B and CG faces 242C. At optional block 245, CL faces 242A, real faces 242B and CG faces 242C may undergo one or more augmentations. Non-limiting examples of augmentations that may be applied at the optional block 245 include: affine transformations such as random translation (e.g. less than a maximum of 5%, 10% some other configurable threshold of image size), rotation (e.g. less than a maximum 5°, 10° orsome other configurable threshold of rotation) and / or uniform scaling (e.g. less than 5%, 10% some other configurable threshold in scale); random color augmentation (e.g. color tinting); random contrast augmentation; random exposure augmentation; random brightness augmentation; random tint augmentation; lighting augmentation; background augmentation; augmentations in clothing and accessories; augmentations in facial hair and / or the like. In some embodiments, the augmentation at block 245 may comprise grid distortion, wherein the input images are distorted by 2D warp vectors defined for each pixel. The warp vectors may be computed by first creating a grid of coordinates with random number of columns / rows (e.g. 2,4,8 or 16), followed by random shifts on the cell coordinates (some percentage (e.g. 24%) of the cell size) and lastly, up-sampling the grid to match the image resolution. These image augmentations are described, for example, in Buslaev A. et al. Albumentations: Fast and Flexible Image Augmentations. Information. 2020; 11 (2):125., which is hereby incorporated herein by reference. In some embodiments, the augmentation at block 245 may comprise other types of additional or alternative image augmentations, such as other types of grid distortions, elastic transforms and piecewise affine transformations.

[0080] Method 200 then proceeds to block 250 which receives, as input, CL faces 242A, real faces 242B and CG faces 242C (for simplicity, it should be understood that where block 245 has occurred, the terms CL faces 242A, real faces 242B and CG faces 242C refer to the augmented faces output from block 245). At block 250, CL faces 242A, real faces 242B and CG faces 242C are sorted into a plurality of CL image subsets 252A, a plurality of real image subsets 252B and a plurality of CG image subsets 252C, respectively according to one or more attributes and / or augmentations of the faces depicted therein.

[0081] At block 250, CL faces 242A may be sorted into CL image subsets 252A of one or more of three types: single variable subsets, multi-variable subsets and variable subsets. Single variable subsets may comprise subsets in which each image in a subset has only one attribute and / or augmentation which is different between each image (e.g. differing facial expressions, differing head orientations or differing lighting arrangements) while the remaining attributes and / or augmentations are common between each image in the subset. For example, a first CL image subset 252A may comprise eight CL faces 242A having the same facial expressions, the same head orientation but wherein each of the eight CL faces 242A have different lighting arrangements or different lighting augmentations (e.g. randombrightness, contrast or color tinting augmentations). Multi-variable subsets may comprise subsets in which each image in a subset has more than attribute or augmentation which is different between each image (e.g. differing facial expressions, differing head orientations or differing lighting arrangements) and only one attribute or augmentation which is common between each image in the subset. For example, a first CL image subset 252A may comprise eight CL faces 242A having the same facial expressions, but each of the eight CL faces 242A have different head orientations and lighting arrangements. Variable subsets, on the other hand, may comprise subsets in which there is a random assortment of attributes and / or augmentations amongst the images of the subset such that some images may share some attributes while others do not. Where two non-CG image facial expressions are described as being common or the same, it should be understood that there may still be some variability between such facial expressions (e.g. because it is difficult for a human to re-create a facial expression exactly). Likewise, where two non-CG image head orientations are described as being common or the same, it should be understood that there may still be some variability between such head orientations (e.g. because it is difficult for a human to re-create a head orientation exactly). On the other hand, for CG images, facial expressions, head orientations, eye gaze, etc. can be recreated without variability. In this way, employing CG images (e.g. CG image subsets 252C) in addition to non-CG images (e.g. CL image subsets 252A and real image subsets 252B), may improve the training of model 32.

[0082] At block 250, CL faces 242A may be sorted based at least in part on CL data 12B provided with CL images 12A which identifies the lighting arrangement (e.g. which light(s) of the light array are on / off) for each image of CL images 12A. For example, CL Faces 242A may be sorted into groupings having the same lighting arrangement but different facial expressions and / or head orientations. In some embodiments, human judgment may be employed to sort CL faces 242A according to head orientation and / or facial expression. In some embodiments, CL faces 242A may be sorted automatically (e.g. by suitable software which identifies head orientation and / or facial expression) according to head orientation and / or facial expression. As it is difficult for an actor to re-create identical facial expressions at different head orientations or identical head orientations with different facial expressions, it may be the case that CL image subsets 252A include more single variable subsets wherelighting is the variable attribute (as compared to single variable subsets where other attributes are the variable attribute).

[0083] At block 250, real faces 242B may likewise be sorted into real image subsets 252B of one or more of three types: single variable subsets, multi-variable subsets and variable subsets. In some embodiments, human judgment may be employed to sort real faces 242B according to lighting arrangement, head orientation and / or facial expression. In some embodiments, CL faces 242A may be sorted automatically (e.g. by suitable software which identifies head orientation and / or facial expression) according to lighting arrangement, head orientation and / or facial expression. For example, where a lighting arrangement in a captured performance from which real faces 242B are obtained is static, then it could be assumed that images where the actor has a similar head orientation have similar lighting conditions. As precise lighting arrangements of real images 12C may be unknown and / or it may be difficult to find real faces 242B which only vary by a single attribute and / or which only have a single common attribute, it may be the case that real image subsets 252B include more variable subsets than single variable subsets and / or multi-variable subsets.

[0084] At block 250, CG faces 242C may likewise be sorted into CG image subsets 252C of one or more of three types: single variable subsets, multi-variable subsets and variable subsets. In some embodiments, human judgment may be employed to sort CG faces 242C according to lighting arrangement, head orientation and / or facial expression. In some embodiments, CG faces 242C may be sorted automatically (e.g. by suitable software) according to lighting arrangement, head orientation and / or facial expression. Since it may be possible to precisely control the attributes of each CG face 242C, it may be possible to provide any desired distribution of single variable, multi-variable and variable subsets with accurate consistency between common attributes. For example, where a single CG image 252C was re-lit at block 230, it would be possible to create a single variable CG image subset 252C where the facial expression and head orientation are identical while the lighting arrangement changes in a controlled manner. Similarly, because of the control available during rendering at block 220, it would be possible to create a multi-variable CG image subset 252C where the facial expression is identical but the head orientation and lighting arrangements vary (which would be difficult to achieve with real images 12C). As such, the inclusion of CG images 12D’ in addition to the non-CG images (e.g. CL Images 12A and real images 12C) may facilitate the creation of effective training data 22 and / orallow for better training of model 32 as compared to training model 32 without CG images 12D’.

[0085] To facilitate training (e.g. at block 30 of method 10), block 250 may also output CL subset data 254A, real subset data 254B and CG subset data 254C corresponding to CL image subsets 252A, real image subsets 252B and CG image subsets 252C, respectively. CL subset data 254A may, for each CL image subset 252A, identify the type of subset (e.g. single variable, multi-variable or variable) and which attributes (e.g. facial expression, head orientation and lighting arrangement) vary and / or which don’t. Real subset data 254B may, for each real image subset 252B, identify the type of subset (e.g. single variable, multivariable or variable) and which attributes (e.g. facial expression, head orientation and lighting arrangement) vary and / or which don’t. CG subset data 254C may, for each CG image subset 252C, identify the type of subset (e.g. single variable, multi-variable or variable) and which attributes (e.g. facial expression, head orientation and lighting arrangement) vary and / or which don’t. Together, CL subset data 254A, real subset data 254B and CG subset data 254C and CL image subsets 252A, real image subsets 252B and CG image subsets 252C may be referred to as training data 256. In some embodiments, training data 256 is used as training data 22 for method 10.

[0086] Figure 4 is a schematic depiction of a method 300 that may be used to implement the block 240 cropping and alignment of method 200 of Figure 3 according to a particular embodiment. Method 300 may be performed in an automated manner by processor 102 of system 100 (Figure 2) or a similar processor or system. Method 300 may be notionally separated into three branches. For example, method 300 may be understood to have one branch 301 A for CL images 12A, one branch 301 B for real images 12C, and one branch 301 C for CG images 12D’. Each branch 301 of method 300 receives, as input, a corresponding set of training images and produces, as output, corresponding aligned face images (i.e. one aligned face image for each input training image). For brevity, branch 301 A of method 300 (corresponding to CL images 12A) is described in detail and the corresponding data preparation branches 301 B and 301 C for real images 12C and CG images 12D’ will be understood to be analogous.

[0087] Branch 301 A comprises an alignment block 302 which is performed once for each image of CL images 12A to generate corresponding aligned CL face images 242A (also referred to herein as CL faces 242A). For each frame / image of CL images 12A, alignmentblock 302 starts at block 306 which comprises performing a face detection operation to determine a bounding box in the current frame which includes the identity’s face. There are numerous face detection techniques known in the art that may be used at block 306. One suitable non-limiting technique, is that disclosed by Bulat et al. 2017. How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks). In International Conference on Computer Vision., which is hereby incorporated herein by reference. In some embodiments, where CL images 12A comprise sequences of images wherein for each sequence, the head orientation (and, in some cases, facial expression) of the actor do not change, the block 302 steps are employed for a first image of that sequence (e.g. where the actor is lit from the front) and then the same alignment is used for the remaining images of that sequence. This may facilitate achieving accurate alignment in subsequent images of the sequence where the lighting conditions (e.g. where the face is mostly not illuminated) may make alignment more difficult and / or may reduce computing expenses / time.

[0088] Alignment block 302 then proceeds to block 308 which involves applying a 2D landmark detection process within the bounding box determined at block 306 to find fiducial points on the face. There are numerous facial landmark detection techniques known in the art that may be used at block 308. One suitable non-limiting technique, is that disclosed by Bulat et al. discussed above. In some embodiments, the 2D landmarks (fiducial points) of interest at block 308 include landmarks from the eyebrows, eyes and / or nose.

[0089] Alignment block 302 then proceeds to block 310 which involves computing and applying a 2D affine transformation that will align the face to a canonical front head pose. Suitable non-limiting techniques for this block 310 process are described in: Shinji Umeyama. 1991. Least-Squares Estimation of Transformation Parameters Between Two Point Patterns. IEEE Trans. Pattern Anal. Mach. Intell. 13, 4 (1991 ), 376-380; and.Naruniec et al. 2020. High-Resolution Neural Face Swapping for Visual Effects. Computer Graphics Forum 39, 4 (2020), 173-184; both of which are hereby incorporated herein by reference. The output of the block 310 process is a cropped canonical front head pose (referred to herein as aligned CL face images 242A) corresponding to the current image CL images 12A. Alignment block 302 may be repeated for each image of CL images 12A.

[0090] Figure 4 shows branches 301 A, 301 B, 301 C corresponding to CL images 12A, real images 12C and CG images 12D’ for generating aligned CL face images 242A, aligned realface images 242B and aligned CG face images 242C respectively. Each branch 301 may be implemented in a manner analogous to that of branch 301 A described above.

[0091] In some embodiments, aligned CL face images 242A, aligned real face images 242B and aligned CG face images 242C may comprise (or may be converted, using suitable upsampling or downsampling techniques, to) 512x512 pixel images of a face of their corresponding identity with three channels (e.g. red (R), green (G), blue (B)) per pixel, although other image resolutions and other numbers of per-pixel channels are possible. This upsampling or downsampling may be used so that aligned CL face images 242A, aligned real face images 242B and aligned CG face images 242C have resolutions corresponding to the configurations of trained model 32.

[0092] Figure 5 is a schematic depiction of a training method 400 that may be used to implement the block 30 model training of method 10 according to a particular embodiment. Training scheme 400 may be performed in an automated manner by processor 102 of system 100 (Figure 2) or a similar processor or system.

[0093] Training method 400 trains autoencoders 502 (described in more detail below) to each receive an image and to re-construct the received image. Training method 400 receives, as input, training data 22 (e.g. training data 256 comprising CL image subsets 252A, real image subsets 252B and CG image subsets 252C and corresponding CL subset data 254A, real subset data 254B and CG subset data 254C).

[0094] Training method 400 may involve training autoencoders 502, having the architecture shown in Figure 6. In general, autoencoders, are a type of neural network which comprise encoders that compress their input into latent code and decoders that decompress the latent code in an effort to reconstruct the original input. Autoencoders 502 depicted in the illustrated embodiment of the Figure 5 training method and the Figure 6 architecture comprise a first autoencoder 502A comprising an encoder 504 and a first (non-CG image) decoder 506A and a second autoencoder 502B comprising encoder 504 and a second (CG- image) decoder 506B. First (non-CG image) decoder 506A and second (CG-image) decoder 506B may collectively be referred to herein as decoders 506. In some embodiments, block 80 of method 10 (Figure 1 ) only employs first autoencoder 502A while second autoencoder 502B is used instead to better train encoder 504 of first autoencoder 502A at block 30 of method 10 to thereby improve the output of first autoencoder 502A. Assuch, in some embodiments, trained model 32 comprises autoencoder 502A (and not autoencoder 502B) but for the purposes of this description, its is assumed, for generality, that model 32 comprises both autoencoder 502A and autoencoder 502B.

[0095] Optionally, one or more initial decoder layers 506-1 of first (non-CG image) decoder 506A share the same trainable parameters (i.e. are the same) as one or more initial layers 506-1 of second (CG-image) decoder 506B. That is, one or more initial decoder layers 506- 1 of the Figure 6 embodiment are constrained to be common (share the same trainable parameters) as between autoencoders 502. Further, encoder 504 is common between autoencoders 502. This commonality of encoder 504 and the one or more initial decoder layers 506-1 is shown schematically in Figure 6 by shading. Further, the data compression of autoencoder 502 is schematically illustrated in Figure 6 by its shape - that is, encoder 504 is shown in Figure 6 as getting narrower (in height) from right to left as data is compressed and decoders 506 are shown as getting wider (in height) from right to left as the latent code is decompressed.

[0096] Apart from its one or more shared initial layers 506-1 , each decoder 506A, 506B is unique (comprises trainable parameters that are unique to it). As described in more detail below, first (non-CG image) decoder 506A is trained to reconstruct reconstructed CL faces 562A from CL image subsets 252A and reconstructed real faces 562B from real image subsets 252B while second (CG-image) decoder 506B is trained to construct reconstructed CG faces 562B from CG image subsets 252C. In some embodiments, reconstructed CL faces 562A, reconstructed real faces 562B and reconstructed CG faces 562C (collectively reconstructed faces 562) may comprise (and image decoders 506 may output) 512x512 pixel images with three channels (e.g. red (R), green (G), blue (B)) per pixel, although other image resolutions and other numbers of per-pixel channels are possible.

[0097] Returning to Figure 5, method 400 may start at any one of blocks 410, 420 or 430. In some embodiments, blocks 410, 420 and / or 430 occur in parallel. In some embodiments, blocks 410, 420 and / or 430 occur in series.

[0098] At block 410, method 400 involves initializing a counter variable / '. The counter variable / is used to perform n iterations of batch loop 405 where n is the total number of subsets from amongst CL image subsets 252A, real image subsets 252B and CG image subsets 252C. In other words, method 400 may involve one iteration of batch loop 405 foreach of the subsets from amongst CL image subsets 252A, real image subsets 252B and CG image subsets 252C. In the illustrated embodiment, the counter variable / is set to i=1 at block 410.

[0099] At block 420, method 400 involves initializing trainable parameters 505 of encoder 504, trainable parameters 507A of first (non-CG image) decoder 506A and trainable parameters 507B of second (CG-image) decoder 506B. Trainable parameters 505, 507A, 507B may comprise any trainable parameters (e.g. weights, bias parameters and / or the like). In some embodiments, block 410 may randomly initialize trainable parameters 505 of encoder 504, trainable parameters 507A of first (non-CG image) decoder 506A and trainable parameters 507B of second (CG-image) decoder 506B. In some embodiments, other techniques (such as assigning some prescribed values) may be employed to set trainable parameters 505 of encoder 504, trainable parameters 507A of first (non-CG image) decoder 506A and trainable parameters 507B of second (CG-image) decoder 506B.

[0100] At block 430, method 400 involves initializing the latent code format 432. Latent codes 512 generated by encoder 504 (see Figure 6) may comprise a vector having a plurality of components 512A (see Figure 8). When training of model 32 begins, components 512A of latent code 512 may be “entangled” such that it may be difficult or impossible to modify a single attribute 4 (e.g. facial expression, head orientation, lighting arrangement) represented by latent code 512 without undesirably affecting other attributes 4, since many of the components 512A of latent code 512 may directly or indirectly affect multiple attributes 4. However to facilitate manipulation of attributes 4 of input images 42 according to inference portion 10B of method 10 (Figure 1 ), training method 400 (as described in more detail below) encourages disentanglement of components 512A by encouraging organization of components 512A of latent code 512 according to a desired latent code format 432.

[0101] At block 430, a desired latent code format 432 may be set by a user (e.g. by trial and error) or automatically chosen. In some embodiments, latent code format 432 is dynamic such that it varies throughout the training of model 32 based on one or more objectives (e.g. latent code format 432 may be learned while training model 32). Latent code format 432 may depend on a number of factors. For example, latent code format 432 may depend on one or more of: the number of attributes to be partitioned or disentangled within latent code format 432, the types of attributes to be partitioned or disentangled withinlatent code format 432 (e.g. facial expression may be better represented by a larger portion of latent code as compared to head orientation), the extent of desired disentanglement of individual attributes to be partitioned or disentangled within latent code format 432 (e.g. it may be desirable to allocate more components of latent code format 432 to the arrangement of the lighting when disentangling the colour or intensity of the lighting in addition the location(s) of the lighting).

[0102] Latent code format 432 divides at least some components 512A of latent code 512 into subsets 512B wherein each subset 512B is desired to be representative of a particular attribute 4. For example, latent code format 432 may divide latent code 512 into a first subset 512B-1 of components 512A of latent code 512 desired to be representative of facial expression, a second subset 512B-2 of components 512A of latent code 512 desired to be representative of head orientation and a third subset 512B-3 of components 512A of latent code 512 desired to be representative of the lighting arrangement, as schematically depicted in Figure 8. While Figure 8 depicts subsets 512B-1 , 512B-2 and 512B-3 of elements 512A as being contiguous, this is not mandatory.

[0103] Each subset 512B of latent code 512 may be further divided at block 430. For example, subset 512A-2 (representative of head orientation) may be divided into one or more components 512A representative of head azimuth and one or more components 512A representative of head elevation. This further division of components 512A within subsets 512B is not mandatory.

[0104] Returning to Figure 5, method 400 then proceeds to block 440 which involves choosing a subset from amongst CL image subsets 252A, real image subsets 252B and CG image subsets 252C to be the / thsubset 442, where / is a number between 1 and n, and where n is the total number of subsets from amongst CL image subsets 252A, real image subsets 252B and CG image subsets 252C. The / hsubset 442 may be selected by any suitable method. For example, the / thsubset 442 may be chosen by a user or according to an algorithm. In some embodiments, the / hsubset 442 is selected randomly from amongst CL image subsets 252A, real image subsets 252B and CG image subsets 252C that have not yet been chosen at block 440 during previous batch loops 405 of method 400 (if any such batch loops 405 have occurred). In some embodiments, / *hsubset 442 is semirandomly chosen (e.g. block 440 alternates between choosing CL image subsets 252A, realimage subsets 252B and CG image subsets 252C but picks randomly from within each of CL image subsets 252A, real image subsets 252B and CG image subsets 252C).

[0105] Block 440 outputs / thsubset 442 comprising k images 442A, where k is at least one. In some embodiments, k is consistent for each / thsubset 442, from i= 1 to i=n, but this is not mandatory and some subsets 442 may comprise more or fewer images. It should be noted that the k images 442A are of the form of cropped and aligned CL faces 242A, real faces 242B, and CG faces 242C optionally augmented at block 245 (see discussion of data preparation method 200 (Figure 3) described above). Block 440 also outputs / thsubset data 444 (from amongst CL subset data 254A, real subset data 254B and CG subset data 254C) corresponding to / hsubset 442.

[0106] Method 400 then proceeds to block 450 which involves employing one of autoencoders 502 to reconstruct the images 442A of / thsubset 442. Where / thsubset 442 was chosen from amongst CL image subsets 252A or real image subsets 252B, first autoencoder 502A may be employed (see Figure 6) and where / ’hsubset 442 was chosen from amongst CG image subsets 252C, second autoencoder 502C may be employed (see Figure 6). Block 450 may output intermediate training data 452. Block 450 may employ method 500, described further herein.

[0107] Method 400 then proceeds to block 460 where (referring to Figure 6): shared trainable parameters 505 of encoder 504 and shared trainable parameters 507A, 507B of decoders 506 (e.g. shared trainable parameters 507A, 507B from one or more initial decoder layers 506-1 of decoders 506) are updated based at least in part on intermediate training data 452 outputted from block 450; and either: (1 ) trainable parameters 507A of portion 506-2A of first (non-CG image) decoder 506A are updated based at least in part on intermediate training data 452 obtained from block 450 (e.g. if / ’hsubset 442 was chosen from amongst CL image subsets 252A, or real image subsets 252B); or (2) trainable parameters 507B of portion 506-2B of second (CG-image) decoder 506B are updated based at least in part on intermediate training data 452 obtained from block 450 (e.g. if / hsubset 442 was chosen from amongst CG image subsets 252C). Such trainable parameters of autoencoders 502 may be updated using any suitable machine learning method (e.g. back propagation). In some embodiments, such trainable parameters of autoencoders 502 may be updated according to method 600, described further herein.

[0108] Once block 460 is complete, method 400 proceeds to inquiry 470 to determine if counter / ' reaches i=n. Once, counter / ' reaches i=n, then autoencoders 502 have been trained with all of the subsets of images (e.g. CL image subsets 252A, real image subsets 252B and CG image subsets 252C) and method 400 is complete. If counter / ' is less than n, then method 400 increments counter / 'and returns to block 440 to choose a new Ithsubset 442. In this way, blocks 440, 450 and 460 are repeated until inquiry 470 determines that counter / ' reaches i=n.

[0109] Figures 7A, 7B and 7C (collectively Figure 7) schematically depict a method 500 for reconstructing images with encoders 502. Method 500 may be used to implement the block 450 reconstruction of images of method 400 according to a particular embodiment. Method 500 may be performed in an automated manner by processor 102 of system 100 (Figure 2) or a similar processor or system.

[0110] Method 500 may be notionally separated into three branches 501 A, 501 B, 501 C. Where the Ithsubset 442 is chosen (e.g. at block 440 of method 400) from amongst CL image subsets 252A, branch 501 A of method 500 may be employed at block 450 of method 400. Where the ithsubset 442 is chosen (e.g. at block 440 of method 400) from amongst real image subsets 252B, branch 501 B of method 500 may be employed at block 450 of method 400. Where the Ithsubset 442 is chosen (e.g. at block 440 of method 400) from amongst CG image subsets 252C, branch 50BA of method 500 may be employed at block 450 of method 400. For brevity, branch 501 A of method 500 is described in detail and the corresponding branches 501 B, 501 C will be understood to be analogous, unless otherwise indicated. For example, first branch 501 A involves employing first autoencoder 502A to reconstruct CL image subsets 252A as reconstructed CL faces 562A; second branch 501 B involves employing first autoencoder 502A to reconstruct real image subsets 252B as reconstructed real faces 562B; and third branch 501 B involves employing second autoencoder 502B to reconstruct CG image subsets 252C as reconstructed CG faces 562C. Together reconstructed CL faces 562A, reconstructed real faces 562B and reconstructed CG faces 562C may be referred to as reconstructed faces 562.

[0111] Branch 501 A commences at block 510 which receives, as input, Ithsubset 442 comprising k images 442A from amongst CL image subsets 252A. For purposes of illustration, branch 501 A is described in detail for an instance where Ithsubset 442 comprises a first CL image subset 252A-1 comprising k images.

[0112] When the / ( images of first CL image subset 252A-1 are fed to encoder 504 at block 510, encoder 504 compresses each image into latent codes 512 (e.g. k latent codes 512, one for each of the k images of first CL image subset 252A-1 ). The latent code 512 for each image of first CL image subset 252A-1 may be in the form of a vector comprising a plurality of components 512A.

[0113] Unlike typical training schemes where latent codes 512 are then fed directly into a decoder to reconstruct the images of first CL image subset 252A-1 , latent codes 512 are processed according to blocks 530, 540 and 550 before being fed to first (non-CG image) decoder 506A at block 560, as discussed further herein.

[0114] Branch 501 A proceeds from block 510 to block 530 which receives, as input, ithsubset data 444 (see Figure 5). The ithsubset data 444 may comprise, in the illustrated embodiment, first CL subset data 254A-1 corresponding to first CL image subset 252A-1 and latent code format 432. At block 530, target components 532 of latent codes 512 are identified based at least in part on first CL subset data 254A-1 . Target latent code components 532 are components 512A of latent codes 512 that are representative (or, more precisely, are selected to be representative) of the attributes that are constant within a subset of images (e.g. first CL image subset 252A-1 ). In one example, CL subset data 254A-1 indicates that first CL image subset 252A-1 comprises a single variable subset, wherein only the lighting arrangement changes from image to image. In this example, target latent code components 532 may comprise the latent code components 512A of first subset 512B-1 and second subset 512B-2 (e.g. the subsets 512B of latent code components 512A representative of facial expression and head orientation) - see Figure 8. As another example, where Ithsubset 442 comprises a multi-variable CL image subset 252A wherein only the lighting arrangement is constant, target latent code components 532 would comprise latent code components 512A of third subset 512B-3 (e.g. the subset 512B of components 512A representative of lighting arrangement) - see Figure 8. On the other hand, where ithsubset 442 comprises a variable CL image subset 252A (e.g. in which there are no constant attributes throughout the entire subset of CL faces 242A), there are no target latent code components 532 and blocks 530 to 552 of method 500 (discussed further herein) may be skipped in which case the k latent codes 512 output by encoder 504 are fed directly to decoder 506 at block 560.

[0115] Branch 501 A then proceeds to block 540 which receives, as input, target latent code components 532 and latent codes 512 for each of the k images of first CL image subset 552A-1 . At block 540, a mean value for each of target latent code components 532 is determined as the mean of the values, across each of the k images of first CL image subset 552A-1 , of the corresponding component 512A of latent codes 512. These mean target component values for each target component 532 are together output as mean target component values 542.

[0116] Branch 501 A then proceeds to block 550 which receives, as input, mean target component values 542 and latent codes 512. At block 550, the values of target components 532 of each of the k latent codes 512 are substituted with the mean target component values 542 obtained at block 540 to create k modified latent codes 552, as shown in Figure 9.

[0117] Branch 501 A then proceeds to block 560 which receives, as input, modified latent codes 552 and feeds modified latent codes 552 into first (non-CG image) decoder 506A. First (non-CG image) decoder 506A attempts (and is trained to) reconstruct each of the k CL faces 242A of first CL subset 252A-1 based on modified latent codes 552. These attempts are output by first (non-CG image) decoder 506A as reconstructed CL faces 562A.

[0118] Branch 501 A may output reconstructed CL faces 562A, the values of target latent code components 532 of each of the k latent codes 512 and mean target component values 542 together as intermediate training data 452 (see Figure 5).

[0119] Figure 10 is a schematic depiction of a method 600 for updating trainable parameters 505 of encoder 504 and trainable parameters 507A, 507B of decoders 506 based at least in part on intermediate training data 452 according to a particular embodiment. Method 600 may be used to implement block 460 of method 400 (Figure 5). Method 600 may be performed by processor 102 of system 100 (Figure 2).

[0120] Method 600 may involve the use of a number of loss functions (also known as objective functions or criterion functions) which are computed during method 600 to determine trainable parameters 505 of encoder 504 and trainable parameters 507A, 507B of decoders 506 to thereby generate trained model 32. In the illustrated embodiment, method 600 has two types of loss functions.

[0121] The first loss function is an image loss (IL) function, which compares reconstructed faces 562 of the Ithsubset 442 output from block 450 (see also reconstructed faces 562 of Figures 7A, 7B, 7C) to the corresponding input faces of the Ithsubset 442. In the described example where ithsubset 442 comprises a first CL image subset 252A-1 , reconstructed CL faces 562A are compared to CL faces 242A of first CL image subset 252A-1 .

[0122] The second loss function is a target loss (TL) function, which compares mean target component values 542 to the values of target latent code components 532 for each of the k latent codes 512 generated by encoder 504 based on the Ithsubset 442 at block 510 See Figures 7A, 7B, 7C). In the described instance where ithsubset 442 comprises first CL image subset 252A-1 , mean target component values 542 are compared to the values of target components 532 for each of the k latent codes 512 generated by encoder 504 based on the first CL image subset 252A-1 .

[0123] In general, the IL criterion functions that are used for IL function evaluations may comprise a number of terms that are representative of differences between the input images and reconstructed images. Similarly, the TL criterion functions that are used for TL function evaluations may comprise one or more terms that are representative of differences between the target component values 532 and mean values 542. In one particular embodiment, IL function evaluations comprise least absolute deviation (L1 norm) and structural similarity index measure (SSIM) criterion functions. Other additional or alternative criterion functions could be included in IL function evaluations. In one particular embodiment, TL function evaluations comprise least absolute deviation (L1 norm) criterion function. Other additional or alternative criterion functions could be included in TL function evaluations.

[0124] Method 600 starts at block 610 which involves determining the losses (e.g. IL losses 612 and TL losses 614) corresponding to Ithsubset 442. For the Ithsubset 442, block 610 may be performed k times for each of the k images in the ithsubset 442.

[0125] As discussed above, in some embodiments, for each of the k images of Ithsubset 442 (K=1 , 2, ... k), the IL loss 612, £IL Kcomprises L1 norm (least absolute deviation) and SSIM (structural similarity index measure) terms, in which case the IL loss £ILKfor theimage may have the form lL,K= a^IL,Ll,K + b lL,SSIM,K ( 1 )where £IL:L1:Kis the image loss L1 norm loss function for the K ' image corresponding to Ithsubset 442, £IL:SSIM,Kisthe image loss SSIM loss function for the K*himage corresponding to Ithsubset 442 and a, b are configurable (e.g. user configurable or preconfigured) weight parameters. In some embodiments, the SSIM loss function £ILISSIM,K may comprise those described in Wang et al. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600-612, which is hereby incorporated herein by reference. In some embodiments, additional or alternative loss terms may be used as a part of IL loss 612. By way of non-limiting example, such additional or alternative loss terms may include some of those used in adversarial networks such as PatchGAN (e.g. as disclosed, for example, in Isola et al. 2017. Image-to-lmage Translation with Conditional Adversarial Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5967s5976. https: / / doi.org / 10 (which is hereby incorporated herein by reference) and / or the like, perceptual loss terms (also known as VGG loss terms) as described, for example, in Johnson et al. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. arXiv:1603.08155 (which is hereby incorporated herein by reference) and / or other types of loss terms. Such additional or alternative loss terms may be incorporated into equation (1 ) and may be assigned suitable (user configurable or preconfigured) weights. In some embodiments, different loss terms may additionally or alternatively be used for one or more different iterations of method 600.

[0126] In some embodiments, for each of the k images of Ithsubset 442 (K=1 , 2, ... k), the TL loss 614 comprises an L1 norm (least absolute deviation) term, in which case the TL loss £TLiKfor theimage may have the form: TL,K= a- "TL,Ll,K (2) where £TL,L1 >Kis the image loss L1 norm loss function for theimage corresponding to ithsubset 442 and a is a configurable (e.g. user configurable or preconfigured) weight parameter. In some embodiments, additional or alternative loss terms may be used as a part of TL loss 614. Such additional or alternative loss terms may be incorporated into equation (2) and may be assigned suitable (user configurable or preconfigured) weights. In some embodiments, different loss terms may additionally or alternatively be used for one or more different iterations of method 600.

[0127] After the image loss 612 is determined for each of the k images of ithsubset 442 (K=1 , 2, ... k) corresponding to ithsubset 442 atblock 610, the total image loss 622 corresponding to Ithsubset 442 (which may be referred to herein as the total Ithimage loss 622) may be determined at block 620 by accumulating (e.g. adding and / or averaging) the image losses 612 for each of the k images of Ithsubset 442 over the k images of ithsubset 442 to determine the total image loss 622 corresponding to Ithsubset 442. Both the L1 norm term £IL:L1:Kand the SSIM term £IL:SSIM,K can be aggregated and / or averaged over the k images of Ithsubset 442 to obtain total Ithimage loss 622 for the Ithsubset 442 The aggregation or averaging may occur at block 610 such that block 620 is not mandatory.

[0128] Similarly, after the target loss 614 is determined for each of the k images of ithsubset 442 (K=1 , 2, ... k) corresponding to Ithsubset 442 at block 610, the total Ithtarget loss 624 corresponding to Ithsubset 442 may be determined at block 620 by accumulating (e.g. adding and / or averaging) the target losses for each of the k images of Ithsubset 442 over the k images of Ithsubset 442 to determine the total ithtarget loss 624 corresponding to Ithsubset 442. The aggregation or averaging may occur at block 610 such that block 620 is not mandatory.

[0129] Method 600 then proceeds to blocks 630 and 640, which may occur in parallel or in series (in any order). Block 630 involves determining encoder parameter loss gradients 632 for Ithsubset 442. Determining encoder parameter loss gradients 632 at block 630 comprises computing partial derivatives of total ithIL losses 622 and total IthTL losses 624 with respect to each of trainable encoder parameters 505. Encoder parameter loss gradients 632 may be stored as part of block 630.

[0130] Block 640 involves determining decoder parameter loss gradients 642 for Ithsubset 442. Determining decoder parameter loss gradients 642 at block 630 comprises computing partial derivatives of total IthIL losses 622 with respect to each of the trainable parameters for a particular decoder 506 corresponding to the Ithsubset 442. For example, where Ithsubset 442 comprises first CL images subset 552A-1 , block 630 comprises computing partial derivatives of IL losses 622 for first (real image) decoder 506A with respect to each of trainable parameters 507A. Decoder loss gradients 642 may be stored as part of block 640.

[0131] Method 600 then proceeds to block 650 which involves using the encoder parameter gradients 632 together with a learning rate parameter (which is a pre-set or configurable(e.g. user-configurable) to update trainable parameters 505 of encoder 504, thereby obtaining updated trainable parameters 505 and using the decoder parameter gradients 642 together with a learning rate parameter (which is a pre-set or configurable (e.g. user- configurable) to update the trainable parameters 507A of first (non-CG image) decoder 506A (or trainable parameters 507B of second (CG-image) decoder 506B, as the case may be), thereby obtaining updated trainable parameters 507A (or updated trainable parameters 507B, as the case may be depending on whether the Ithsubset 442 comprises a CL or real images or whether the Ithsubset 442 comprises CG images). For a given parameter p, the block 650 parameter update may comprise implementing functionality of the form:9J (3)Pnew Poldagp where pnewis the updated parameter value, poidis the existing parameter value prior to block 284, a is the learning rate andis the applicable gradient for the parameter p. In some embodiments, block 650 may involve use of a suitable optimization algorithm together with its meta-parameters to update trainable parameters 505 and trainable parameters 507A (or trainable parameters 505 and trainable parameters 507B, as the case may be). One non-limiting example of such an optimization algorithm is the so-called Adam optimization technique, with its meta-parameters described, for example, in Kingma et al. 2014a. Auto-Encoding Variational Bayes. In 2ndInternational Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.)., which is hereby incorporated herein by reference. In some embodiments, the meta-parameters of this Adam optimization technique may comprise= 0.5, / ?2= 0.999 and learning rate of a = 5e-5.

[0132] Method 600 may conclude at block 660 which involves updating autoencoders 502 to incorporate updated trainable parameters 505, 507A, 507B and outputting updated autoencoders 502.

[0133] In the alternative to or in addition to training method 400, a cyclic loss training scheme may be employed for the block 30 model training of method 10. For example, Figure 11 is a schematic depiction of a method 900 for a cyclic loss training scheme that may be used for at least a portion of the block 30 model training of method 10. Method 900may be performed in an automated manner by processor 102 of system 100 (Figure 2) or a similar processor or system.

[0134] Method 900 may receive, as input, one or more of: at least a portion of CL faces 242A, at least a portion of real faces 242B and at least a portion of CG faces 242C. Such faces may be augmented (e.g. at block 245 of method 200) but this is not mandatory.

[0135] Method 900 starts at block 910 which involves choosing an image 912 or a batch of images 912 from among CL faces 242A, real faces 242B and CG faces 242C. Where a batch is chosen at block 910, it may be that each image of the batch may be a non-CG image (e.g. chosen from among CL faces 242A and real faces 242B) or that each image of the batch may be a CG face 242C. For simplicity, the steps of method 900 are described herein for a single image 912 but it should be understood that steps of method 900 could be implemented for a batch of images 912 instead.

[0136] At block 920, image 912 is processed through encoder 504 to generate latent code 922 representative of image 912.

[0137] At block 930, a subset 512B of components 512A of latent code 922 (which represents an instance of a particular attribute) is replaced or swapped with a corresponding replacement subset 934 which represents a different instance of the same attribute to create a swapped latent code 932. Replacement subset 934 may be any suitable replacement subset. Replacement subset 934 may be chosen randomly from among a number of suitable replacement subsets. Replacement subset 934 may come from latent code generated based on a non-CG image or a CG image (independently of whether or not image 912 is a non-CG image or a CG image). In some embodiments, replacement subset 934 is obtained from latent code representative of one of the other images 912 in the batch chosen at block 910. In some embodiments, more than one subset 512B of latent code 922 is replaced at block 930 (e.g. more than one attribute represented by latent code 922 is modified at block 930).

[0138] At block 940, swapped latent code 932 is processed by a suitable decoder 506 to obtain a swapped image 942. For example, where image 912 is a non-CG image (e.g. chosen from CL faces 242A or real faces 242B), first decoder 506A may be employed whereas where image 912 is a CG image (e.g. chosen from among CG faces 242C) then second decoder 506B may be employed.

[0139] At block 950, swapped image 942 is processed through encoder 504 to generate reprocessed latent code 952 representative of swapped image 942.

[0140] Since swapped image 942 was generated based on swapped latent code 932, reprocessed latent code 952 should be identical or near to identical to swapped latent code 932. As such, by comparing re-processed latent code 952 to swapped latent code 932, a loss may be determined and such loss may be used to update autoencoders 502. For example, the loss determined by comparing re-processed latent code 952 to swapped latent code 932 may be used to update autoencoders 502 by the same or similar steps as set out in blocks 620, 630, 640, 650 and 660 of method 600 described herein.

[0141] Figure 12 is a schematic depiction of a method 700 for obtaining at least a portion of donor latent code 52 representative of one or more desired attributes 4’ (e.g. desired facial expressions 4A’, desired head orientation 4B’ and / or desired lighting 4C’) according to a particular embodiment. Method 700 may be used to implement block 60 of method 10 (see Figure 1 ). Method 700 may be performed in an automated manner by processor 102 of system 100 (Figure 2) or a similar processor or system.

[0142] Method 700 may receive, as input, a donor image 705. Donor image 705 may depict a face with one or more desired attributes 4’ that are desired to be applied to input image 42 as part of inference portion 10B of method 10 (see Figure 1 ). For example donor image 705 may depict a desired facial expression 4A’, a desired head orientation 4B’ and / or a desired lighting arrangement 4C’. Donor image 705 may be obtained in any suitable way. In some embodiments, donor image 705 is a non-CG image (e.g. a photograph of an actor) while in other embodiments, donor image 705 is computer generated such that desired attributes 4’ can be easily controlled (e.g. using a suitable CG model, CG rig and / or the like).

[0143] Method 700 starts at block 710 which involves processing donor image 705 with encoder 504 of trained model 32 to obtain latent code 712 representative of donor image 705 (i.e. a latent code 712 which encodes the desired attributes 4’ (e.g. desired facial expressions 4A’, desired head orientation 4B’ and / or desired lighting 4G’) to be applied to input image 42.

[0144] Method 700 then proceeds to block 720 which extracts at least a portion of latent code 712 representative of the desired attributes 4’ of donor image 705. The extracted portion of latent code 712 is output as donor latent code 52. The portion of latent code 712to extract in block 720 may be identified based on a format of latent code (e.g. latent code format 432) which (as discussed above) sets out which components of latent code 712 are representative of which desired attributes 4’. In this way, the components of latent code 712 representative of desired attributes 4’ (e.g. desired facial expressions 4A’, desired head orientation 4B’ and / or desired lighting 4C’) may be identified and output as donor latent code 52.

[0145] For example, where the desired head orientation 4B’ depicted in donor image 705 comprises the only desired attribute 4’ of donor image 705, donor image 705 is processed at block 710 with encoder 504 of model 32 to obtain latent code 712. The components of latent code 712 representative of desired head orientation 4B’ are then identified based on the known format of latent code 712 (e.g. based on latent code format 432) and output as donor latent code 52.

[0146] As discussed above in connection with method 10 (Figure 1 ), block 50 of inference portion 10B uses donor latent code 52 (e.g. output from method 700) to replace corresponding components of latent code 44 encoded from input image 42, thereby obtaining modified latent code 54 and block 80 of inference portion 10B decodes modified latent code 54 to generate output image 82 having the desired attributes 4’.

[0147] If desired, method 700 can be repeated with additional donor images 705 each having additional desired attributes 4’ and each generating corresponding donor latent codes 52. The donor latent codes 52 from each donor image 705 may then together be employed at block 50 of method 10 (see Figure 1 ) to modify latent code 44 encoded from input image 42, thereby allowing desired attributes 4’ from different donor images 705 to be used to modify input image 42 within a single iteration of inference portion 10B of method 10. For example, in this way, it would be possible to modify input image 42 to include a desired lighting arrangement 4C’ of a first donor image 705 and a desired head orientation 4B’ of a second donor image 705 without having to run inference portion 10B of method 10 multiple times.

[0148] Alternatively, method 10 could be run multiple times. For example, inference portion 10B of method 10 could be run once for the input image 42 and first donor image 705 having a first desired attribute 4’ to obtain a first output image 82 having the first desired attribute and then inference portion 10B could be run another time with first output image 82as its input in the second iteration and a second donor image 705 having a second desired attribute 4’ as input to thereby output a second output image 82 comprising the first input image 42 but modified with the first and second desired attributes 4’.

[0149] Figure 13 is a schematic depiction of a method 800 for obtaining donor latent code 52 representative of a desired attribute 4’ (e.g. desired facial expression 4A’, desired head orientation 4B’ or desired lighting 4C’) according to a particular embodiment. Method 800 may be used to implement block 60 of method 10 (see Figure 1 ). Method 800 may be performed in an automated manner by processor 102 of system 100 (Figure 2) or a similar processor or system.

[0150] Method 800 may be logically divided into a training data preparation portion 800A, a training portion 800B and an inference portion 800C. For simplicity, method 800 is described herein for obtaining donor latent code 52 representative of a desired lighting arrangement 4C’. However, it should be understood that method 800 could be employed to obtain donor latent code representative of other desired attributes 4’ with minor modifications.

[0151] Training data preparation portion 800A receives, as input, CL faces 242A and CL data 12B. As part of training data preparation portion 800A, CL faces 242Aare processed according to blocks 810, 820 and 830 and CL data 12B is optionally processed according to block 840. Block 840 may occur before, during, or after the occurrence of blocks 810, 820 and / or 830 but for convenience, block 840 is described herein after blocks 810, 820 and 830.

[0152] Block 810 receives, as input, CL faces 242A (or at least a portion thereof) and chooses a subset 812 from CL faces 242Awherein each image of the subset only differs by lighting (e.g. the facial expression and head orientation is maintained as constant but the lighting changes). If CL faces 242Aonly include images that vary by lighting (e.g. the facial expression and head orientation are maintained constant throughout all CL faces 242A), then subset 812 may comprise all images of CL faces 242A.

[0153] At block 820, the images of CL subset 812 are processed with encoder 504 of model 32 to obtain latent codes 822 representative of each of the images of CL subset 812. In some embodiments, where such latent codes have previously been obtained (e.g. atanother time during method 10), then blocks 810 and 820 may be skipped and latent codes 822 may be provided directly to method 800 at block 830.

[0154] At block 830, the components of latent codes 822 representing lighting are extracted from latent codes 822 and output from block 830 as lighting latent code components 832. The components of latent codes 822 representative of lighting may be identified based on the known format of latent code 822 (e.g. based on latent code format 432) used to train encoder 504.

[0155] Block 840 receives, as input, CL data 12B (or at least a portion thereof). Block 840 may receive the entirety of CL data 12B in which case the CL data 12B corresponding to CL subset 812 may be extracted or block 840 may receive only CL data 12B corresponding to CL subset 812. At block 840, representations of CL data 12B may be optionally be prepared in a desired format. For example, CL data 12B may be converted to panoramic light maps (e.g. HDR panoramic light maps). Alternatively, if CL data 12B is in an acceptable format, block 840 may be skipped and method 800 may employ CL data 12B directly at block 850.

[0156] Where method 800 is employed to obtain donor latent code 52 representative of desired facial expressions 4A’, training data preparation portion 800A of method 800 may differ from the illustrated embodiment of Figure 13. For example, CL subset 812 may be replaced with a set of images in which only the facial expressions vary (while the head orientation and light arrangement remain constant). These images may comprise CG images as they are relatively easy to control, but could alternatively comprise non-CG images (e.g. photographs of an actor). These images are then processed at block 820 with encoder 504 and the facial expression latent code components (e.g. the components of latent codes 822 representative of facial expressions) may be extracted in place of lighting latent code components 832. CL data 12B may be replaced with descriptions of the facial expressions represented in the images used at block 820.

[0157] Likewise, where method 800 is employed to obtain donor latent code 52 representative of desired head orientations 4B’, training data preparation portion 800A of method 800 may differ from the illustrated embodiment of Figure 13. For example, CL subset 812 may be replaced with a set of images in which only the head orientations vary (while the facial expressions and light arrangement remain constant). These images may comprise CG images as they are relatively easy to control but could alternatively comprisenon-CG images (e.g. photographs of an actor). These images are then processed at block 820 with encoder 504 and the head orientation latent code components (e.g. the components of latent codes 822 representative of head orientations) may be extracted in place of lighting latent code components 832. CL data 12B may be replaced with descriptions of the head orientations (e.g. elevation and azimuth) represented in the images used at block 820.

[0158] Method 800 then proceeds to training portion 800B. Training portion 800B comprises block 850 for training a model 852 to map a relationship between lighting latent code components 832 and lighting representations 842 (or to map facial expression latent code components and facial expression descriptions or to map head orientation latent code components and head orientation descriptions). Model 852 may comprise any suitable type of model for mapping a relationship (e.g. a linear relationship) between lighting latent code components 832 and lighting representations 842 (or facial expression latent code components and facial expression descriptions or head orientation latent code components and head orientation descriptions). For example, model 852 may comprise a machine learning model such as, for example, a variational autoencoder, a convolutional neural network, a de-noising diffusion model or a vision transformer. Model 852 may be trained by any suitable method. In some embodiments, training model 852 at block 850 comprises supervised training as a relationship between lighting latent code components 832 and lighting representations 842 is known.

[0159] Once model 852 is trained at block 850, method 800 then proceeds to inference portion 800C. Inference portion 800C receives, as input at block 860, a desired lighting condition 862 (e.g. desired lighting 4C’) which may be represented in or converted to the same format as lighting representations 842 and processes the desired lighting condition 862 with model 852 to obtain donor latent code 464 (e.g. which may be used as donor latent code 52 or a portion thereof at block 50 of method 10 - see Figure 1 ).

[0160] By employing method 800, a user can avoid needing to create or find donor images 705 with specific desired attributes 4’ for method 700, which can be costly (in time, money and / or computational expense) and / or may be difficult if donor images 705 with the exact desired attributes 4’ are not available. Instead, by using method 800, a user can merely specify any desired lighting arrangement (or facial expression or head orientation, etc.) for use at block 60 of inference portion 10B of method 10.

[0161] While method 10 and sub-methods 200, 300, 400, 500, 600, 700, 800 have been described herein with a single human input identity (e.g. depicted in CL images 12A and real images 12C) for training portion 10A and inference portion 10B (such that input data 12, input image 42 and output image 82 all depict the same identity), this is not mandatory. Instead, method 10 and sub-methods 200, 300, 400, 500, 600, 700, 800 may be modified to also allow for a change of identity of the person or character of input image 42 (in addition to changing one or more attributes 4 such as facial expression, head orientation and / or lighting) when generating output image 82 as part of inference method 10B. For example, such a change of identity may be achieved by incorporating one or more methods of Patent Cooperation Treaty Application No. PCT / CA2022 / 051478 entitled System and Method for Dynamic Neural Face Morphing, which is hereby incorporated herein by reference.

[0162] Figure 14 is a broad schematic depiction of a method 10’ for manipulating one or more attributes 4 (e.g. one or more of facial expression, head orientation and lighting) and the identity of a face depicted in an input image 42 according to a particular embodiment. Method 10’ may be performed in an automated manner by processor 102 of system 100 (Figure 2) or a similar processor or system. While method 10’ is depicted and described for two different identities, it should be understood that this is for simplicity and method 10’ can be expanded for any suitable number of different identities, as desired. Method 10’, like method 10, may be logically divided into a training portion 10A’ and an inference portion 10B’. Method 10’ is substantially similar to method 10 (Figure 1 ) except, for example, as follows.

[0163] Training portion 10A’ is substantially similar to training portion 10A except as follows. Instead of employing only input data 12 for a single human identity (e.g. CL images 12A and real images 12C all depict the same human identity), training portion 10A’ also incorporates second input data 12-2 for a second human identity. Input data 12-2 may be substantially similar to input data 12 (e.g. input data 12-2 may comprise second CL images, second CL data, second real images, second CG images and lighting information). However, the non- CG images of input data 12 depict the second human identity. In some embodiments, input data 12-2 may not include second CL images and / or second CG images. Input data 12-2 may be prepared at block 20 of method 10’ in a substantially similar manner as input data 12.

[0164] Since there is second input data 12-2 for a second identity, training portion 10A’ incorporates an additional autoencoder 502A-2 for non-CG images (e.g. second CL images and second real images) as shown in Figure 15. Autoencoder 502A-2 is substantially similar to autoencoder 502A (e.g. autoencoder 502A-2 also shares encoder 504) but autoencoder 502A-2 employs a decoder 506A-2 which is specific to the second identity.

[0165] Where second input data 12-2 includes second CG images, training portion 10A’ may optionally incorporate an additional autoencoder 502B-2 for CG images (e.g. second GC images) as shown in Figure 15. Autoencoder 502B-2 is substantially similar to autoencoder 502B (e.g. autoencoder 502B-2 also shares encoder 504) but autoencoder 502B-2 employs a decoder 506B-2 which is specific to the second CG identity.

[0166] At block 30, autoencoder 502A is trained with CL image subsets 252A and real image subsets 252B and shared layers from amongst the other autoencoders 502 are updated accordingly. Similarly, autoencoder 502B is trained with CG image subsets 252C and shared layers from amongst the other autoencoders 502 are updated accordingly. Similarly, autoencoder 502A-2 is trained with CL image subsets and real image subsets from second input data 12-2 and shared layers from amongst the other autoencoders 502 are updated accordingly. Similarly, where present, autoencoder 502B-2 is trained with CG image subsets from second input data 12-2 and shared layers from amongst the other autoencoders 502 are updated accordingly.

[0167] Inference portion 10B’ is substantially similar to inference portion 10B except as follows. Inference portion 10B’ comprises additional block 70 which, receives, as input, a desired identity 4D’ for output image 82’. In some embodiments, block 70 only occurs when desired identity 4D’ is different from the identity of input image 42. At block 70, an identityspecific portion of trained model 32 (e.g. an identity-specific autoencoder 502) corresponding to desired identity 4D’ may be chosen to be employed at block 80. For example, where desired identity 4D’ is the first identity, autoencoder 502A may be chosen at block 70 and employed at block 80 to produce an output image 82’ with the first identity. Alternatively, where desired identify 4D’ comprises the second identity, second autoencoder 502A-2 may be chosen at block 70 and employed at block 80 to produce an output image 82’ with the second identity. It will be appreciated that the technique of method 10’ can be expanded to more than 2 identities.

[0168] Since method 10’ may be employed without second CL images and / or without second CG images, method 10’ may be employed to allow manipulation of one or more attributes in images of a second human identity even where second CL images and / or such second CG images are not available or even where it is undesirable to spend the time and / or money to obtain such second CL images and / or such second CG images. This can be advantageous where it is desirable to employ method 10 for a particular actor but there is insufficient training data 22 since method 10’ can be employed for that actor with minimal training data specific to that actor as long as there is sufficient training data for another identity.

[0169] In the preceding description, facial attributes which are manipulated include facial expressions, head orientation, lighting conditions and identity. In other embodiments, other additional or alternative facial attributes may be manipulable using analogous techniques. Such other facial attributes may include eye gaze direction(s) or the presence (or absence) of eyewear (e.g. glasses, goggles, sunglasses, etc.).Interpretation of Terms

[0170] Unless the context clearly requires otherwise, throughout the description and the claims:• “comprise”, “comprising”, and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to”;• “connected”, “coupled”, or any variant thereof, means any connection or coupling, either direct or indirect, between two or more elements; the coupling or connection between the elements can be physical, logical, or a combination thereof;• “herein”, “above”, “below”, and words of similar import, when used to describe this specification, shall refer to this specification as a whole, and not to any particular portions of this specification;• “or”, in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list;• the singular forms “a”, “an”, and “the” also include the meaning of any appropriateplural forms.

[0171] Words that indicate directions such as “vertical”, “transverse”, “horizontal”, “upward”, “downward”, “forward”, “backward”, “inward”, “outward”, “vertical”, “transverse”, “left”, “right”, “front”, “back”, “top”, “bottom”, “below”, “above”, “under”, and the like, used in this description and any accompanying claims (where present), depend on the specific orientation of the apparatus described and illustrated. The subject matter described herein may assume various alternative orientations. Accordingly, these directional terms are not strictly defined and should not be interpreted narrowly.

[0172] Embodiments of the invention may be implemented using specifically designed hardware, configurable hardware, programmable data processors configured by the provision of software (which may optionally comprise “firmware”) capable of executing on the data processors, special purpose computers or data processors that are specifically programmed, configured, or constructed to perform one or more steps in a method as explained in detail herein and / or combinations of two or more of these. Examples of specifically designed hardware are: logic circuits, application-specific integrated circuits (“ASICs”), large scale integrated circuits (“LSIs”), very large scale integrated circuits (“VLSIs”), and the like. Examples of configurable hardware are: one or more programmable logic devices such as programmable array logic (“PALs”), programmable logic arrays (“PLAs”), and field programmable gate arrays (“FPGAs”)). Examples of programmable data processors are: microprocessors, digital signal processors (“DSPs”), embedded processors, graphics processors, math co-processors, general purpose computers, server computers, cloud computers, mainframe computers, computer workstations, and the like. For example, one or more data processors in a control circuit for a device may implement methods as described herein by executing software instructions in a program memory accessible to the processors.

[0173] Processing may be centralized or distributed. Where processing is distributed, information including software and / or data may be kept centrally or distributed. Such information may be exchanged between different functional units by way of a communications network, such as a Local Area Network (LAN), Wide Area Network (WAN), or the Internet, wired or wireless data links, electromagnetic signals, or other data communication channel.

[0174] For example, while processes or blocks are presented in a given order, alternative examples may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternative or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, or may be performed at different times.

[0175] In addition, while elements are at times shown as being performed sequentially, they may instead be performed simultaneously or in different sequences. It is therefore intended that the following claims are interpreted to include all such variations as are within their intended scope.

[0176] Software and other modules may reside on servers, workstations, personal computers, tablet computers, image data encoders, image data decoders, PDAs, colorgrading tools, video projectors, audio-visual receivers, displays (such as televisions), digital cinema projectors, media players, and other devices suitable for the purposes described herein. Those skilled in the relevant art will appreciate that aspects of the system can be practiced with other communications, data processing, or computer system configurations, including: Internet appliances, hand-held devices (including personal digital assistants (PDAs)), wearable computers, all manner of cellular or mobile phones, multi-processor systems, microprocessor-based or programmable consumer electronics (e.g., video projectors, audio-visual receivers, displays, such as televisions, and the like), set-top boxes, color-grading tools, network PCs, mini-computers, mainframe computers, and the like.

[0177] The invention may also be provided in the form of a program product. The program product may comprise any non-transitory medium which carries a set of computer-readable instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of forms. The program product may comprise, for example, non- transitory media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, EPROMs, hardwired or preprogrammed chips (e.g., EEPROM semiconductor chips), nanotechnology memory, or the like. The computer- readable signals on the program product may optionally be compressed or encrypted.

[0178] In some embodiments, the invention may be implemented in software. For greater clarity, “software” includes any instructions executed on a processor, and may include (but is not limited to) firmware, resident software, microcode, and the like. Both processing hardware and software may be centralized or distributed (or a combination thereof), in whole or in part, as known to those skilled in the art. For example, software and other modules may be accessible via local memory, via a network, via a browser or other application in a distributed computing context, or via other means suitable for the purposes described above.

[0179] Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a “means”) should be interpreted as including as equivalents of that component any component which performs the function of the described component (i.e., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated exemplary embodiments of the invention.

[0180] Specific examples of systems, methods and apparatus have been described herein for purposes of illustration. These are only examples. The technology provided herein can be applied to systems other than the example systems described above. Many alterations, modifications, additions, omissions, and permutations are possible within the practice of this invention. This invention includes variations on described embodiments that would be apparent to the skilled addressee, including variations obtained by: replacing features, elements and / or acts with equivalent features, elements and / or acts; mixing and matching of features, elements and / or acts from different embodiments; combining features, elements and / or acts from embodiments as described herein with features, elements and / or acts of other technology; and / or omitting combining features, elements and / or acts from described embodiments.

[0181] Various features are described herein as being present in “some embodiments”. Such features are not mandatory and may not be present in all embodiments. Embodiments of the invention may include zero, any one or any combination of two or more of such features. This is limited only to the extent that certain ones of such features are incompatible with other ones of such features in the sense that it would be impossible for a person of ordinary skill in the art to construct a practical embodiment that combines suchincompatible features. Consequently, the description that “some embodiments” possess feature A and “some embodiments” possess feature B should be interpreted as an express indication that the inventors also contemplate embodiments which combine features A and B (unless the description states otherwise or features A and B are fundamentally incompatible).

[0182] It is therefore intended that the following appended claims and claims hereafter introduced are interpreted to include all such modifications, permutations, additions, omissions, and sub-combinations as may reasonably be inferred. The scope of the claims should not be limited by the preferred embodiments set forth in the examples, but should be given the broadest interpretation consistent with the description as a whole.

Claims

CLAIMS:1 . A method, performed on a computer, for manipulating one or more attributes (e.g. facial expression, head orientation and / or lighting orientation) of a depiction of a face in an input image, the method comprising: preparing training data comprising: preparing one or more non-CG image subsets each comprising non- CG images of a face of a non-CG identity and all having at least one common non-CG attribute; preparing one or more CG image subsets each comprising CG images of a face of a CG identity and all having at least one common CG attribute; training a model comprising first and second autoencoders by: training the first autoencoder with the one or more non-CG image subsets by, for each of the one or more non-CG image subsets: causing a first encoder of the first autoencoder to create first latent codes, each representative of one non-CG image of a current non-CG image subset; replacing first components of the first latent codes with first mean values to create first modified latent codes; causing a first decoder of the first autoencoder to create reconstructed non-CG images based on the first modified latent codes; and updating trainable parameters of the first autoencoder based at least in part on the reconstructed non-CG images and the first mean values; and training the second autoencoder with the one or more CG image subsets by, for each of the one or more CG image subsets: causing a second encoder of the second autoencoder to create second latent codes, each representative of one CG image of a current CG image subset wherein the first encoder and the second encoder comprise a shared set of encoder trainable neural-network parameters;replacing second components of the second latent codes with second mean values to create second modified latent codes; causing a second decoder of the second autoencoder to create reconstructed CG images based on the second modified latent codes; and updating trainable parameters of the second autoencoder based at least in part on the reconstructed CG images and the second mean values; and receiving an input image depicting an input face of one of the non-CG identity and the CG identity; receiving donor latent code components representative of one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation); processing the input image with the first encoder or the second encoder to obtain unmodified latent code; replacing one or more components of the unmodified latent code with the donor latent code components to obtain modified input latent code; and employing a decoder of the model to infer an output image based on the modified input latent code wherein the output image corresponds to the input image except in that the output image depicts the desired attributes (e.g. facial expression, head orientation and / or lighting orientation).

2. A method according to claim 1 or any other claim herein wherein employing a decoder of the model to infer an output image based on the modified input latent code comprises employing the first decoder to infer an output non-CG image based on the modified input latent code wherein the non-CG output image corresponds to the input image except in that the non-CG output image depicts the desired attributes (e.g. facial expression, head orientation and / or lighting orientation).

3. A method according to claim 1 or any other claim herein wherein employing a decoder of the model to infer an output image based on the modified input latent code comprises employing the second decoder to infer an output CG image based on the modified input latent code wherein the CG output image corresponds to theinput image except in that the CG output image depicts the desired attributes (e.g. facial expression, head orientation and / or lighting orientation).

4. A method according to any one of claims 1 to 3 or any other claim herein wherein the input image comprises a CG input image.

5. A method according to any one of claims 1 to 3 or any other claim herein wherein the input image comprises a non-CG input image.

6. A method according to any one of claims 1 to 5 or any other claim herein wherein the one or more non-CG image subsets comprises a plurality of non-CG image subsets and the plurality of non-CG image subsets comprises: one or more controlled lighting (“CL”) image subsets each comprising CL images of a face of the non-CG identity and all having at least one common CL attribute; and one or more real image subsets each comprising real images of a face of the non-CG identity and all having at least one common real attribute.

7. A method according to any one of claims 1 to 6 or any other claim herein wherein preparing the one or more CG image subsets comprises rendering the CG images from a CG animation rig and based at least in part on a desired distribution of facial expressions and head orientations.

8. A method according to any one of claims 1 to 7 or any other claim herein wherein preparing the one or more CG image subsets comprises rendering a first set of CG images from a CG animation rig and based at least in part on a desired distribution of facial expressions and head orientations and then re-lighting the first set of CG images to obtain the CG images.

9. A method according to claim 8 or any other claim herein wherein re-lighting the first set of CG images to obtain the CG images comprises re-lighting the first set of CG images based at least in part on spherical harmonics data.

10. A method according to any one of claims 1 to 9 or any other claim herein wherein:preparing the one or more non-CG image subsets comprises cropping and aligning input non-CG images; and preparing the one or more CG image subsets comprises cropping and aligning input CG images.

11. A method according to claim 10 or any other claim herein wherein: preparing the one or more non-CG image subsets comprises sorting the non-CG input images into single variable non-CG subsets wherein only a single attribute (e.g. facial expression, head orientation or lighting) varies, multi-variable non-CG subsets wherein at least two attributes (e.g. facial expression, head orientation and / or lighting) varies and / or variable non-CG subsets wherein attributes (e.g. facial expression, head orientation and / or lighting orientation) are randomly distributed; and preparing the one or more CG image subsets comprises sorting the CG input images into single variable CG subsets wherein only a single attribute (e.g. facial expression, head orientation or lighting) varies, multi-variable CG subsets wherein at least two attributes (e.g. facial expression, head orientation and / or lighting) varies and / or variable CG subsets wherein attributes (e.g. facial expression, head orientation and / or lighting orientation) are randomly distributed.

12. A method according to any one of claim 1 to 11 or any other claim herein comprising choosing the first components and the second components based at least in part on a desired format of the unmodified latent code wherein the desired format divides the unmodified latent code into subsets of components each corresponding to a particular attribute (e.g. facial expression, head orientation or lighting).

13. A method according to any one of claims 1 to 12 or any other claim herein comprising: determining the first mean values by, for each of the first components determining a mean of values of a corresponding component across the first latent codes; and determining the second mean values by, for each of the second components determining a mean of values of a corresponding component across the second latent codes.

14. A method according to any one of claims 1 to 13 or any other claim herein wherein the first decoder and the second decoder each share one or more identity independent layers of trainable parameters; and wherein: updating the trainable parameters of the first autoencoder based at least in part on the reconstructed non-CG images and the first mean values comprises updating the one or more identity independent layers of trainable parameters of the second decoder based at least in part on the reconstructed non-CG images and the first mean values; and updating the trainable parameters of the second autoencoder based at least in part on the reconstructed CG images and the second mean values comprises updating the one or more identity independent layers of trainable parameters of the first decoder based at least in part on the reconstructed CG images and the second mean values.

15. A method according to any one of claims 1 to 14 or any other claim herein wherein updating the trainable parameters of the first autoencoder based at least in part on the reconstructed non-CG images and the first mean values comprises also updating the second encoder based at least in part on the reconstructed non-CG images and the first mean values; and updating the trainable parameters of the second autoencoder based at least in part on the reconstructed CG images and the second mean values comprises also updating the first encoder based at least in part on the reconstructed CG images and the second mean values.

16. A method according to any one of claims 1 to 15 or any other claim herein wherein: updating the trainable parameters of the first autoencoder based at least in part on the reconstructed non-CG images and the first mean values comprises determining a first image loss function based on a comparison of the reconstructed non-CG images to their corresponding non-CG images and determining a first target loss function based on a comparison of the first mean values and the corresponding components of the first latent code and updating the trainable parameters of the first autoencoder based at least in part on the first image loss function and the first targetloss function; and updating the trainable parameters of the second autoencoder based at least in part on the reconstructed CG images and the second mean values comprises determining a second image loss function based on a comparison of the reconstructed CG images to their corresponding CG images and determining a second target loss function based on a comparison of the second mean values and the corresponding components of the second latent code and updating the trainable parameters of the second autoencoder based at least in part on the second image loss function and the second target loss function.

17. A method according to any one of claims 1 to 16 or any other claim herein wherein receiving the donor latent code components representative of the one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation) comprises obtaining the donor latent code components representative of the one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation) by processing a donor image with the first encoder or the second encoder to obtain donor latent code and the donor latent code components are extracted from the donor latent code.

18. A method according to claim 17 or any other claim herein wherein the donor image comprises a CG donor image.

19. A method according to claim 17 or any other claim herein wherein the donor image comprises a non-CG donor image.

20. A method according to any one of claims 1 to 19 or any other claim herein wherein receiving the donor latent code components representative of the one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation) comprises obtaining the donor latent code components representative of the one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation) by: training a second model to map a relationship between descriptions of a first attribute (e.g. facial expression, head orientation or lighting orientation)andcorresponding training latent code components; and inputting a description of at least one of the one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation) into the second model to obtain at least some of the donor latent code components.21 . A method according to claim 20 or any other claim herein comprising obtaining the corresponding training latent code components by processing images representative of the descriptions of the first attribute (e.g. facial expression, head orientation or lighting orientation)with the first encoder.

22. A method according to claim 20 wherein the images representative of the descriptions of the first attribute (e.g. facial expression, head orientation or lighting orientation)are obtained from amongst the non-CG image subsets or the CG image subsets.

23. A method according to any one of claims 1 to 22 or any other claim herein wherein receiving the donor latent code components representative of the one or more desired attributes (e.g. facial expression, head orientation and / or lighting orientation) comprises obtaining donor latent code components representative of a lighting arrangement by: training a second model to map a relationship between lighting arrangement descriptions and corresponding training latent code components; and inputting a lighting description into the second model to obtain at least some of the donor latent code components.

24. A method according to claim 23 or any other claim herein comprising obtaining the corresponding training latent code components by processing training CL images which vary only by lighting arrangement with the first encoder and wherein the lighting arrangement descriptions comprise descriptions of the training CL images.

25. A method according to any one of claims 23 and 24 or any other claim herein wherein the lighting arrangement descriptions comprises panoramic light maps.

26. A method according to any one of claims 1 to 25 or any other claim herein wherein preparing the training data comprises: preparing one or more second non-CG image subsets each comprising second non-CG images of a second face of a second non-CG identity and all having at least one common second non-CG attribute; preparing one or more second CG image subsets each comprising second CG images of a second face of a second CG identity and all having at least one common second CG attribute; wherein the model comprises third and fourth autoencoders and training the model comprises: training the third autoencoder with the one or more second non-CG image subsets by, for each of the one or more second non-CG image subsets: causing a third encoder of the third autoencoder to create third latent codes, each representative of one second non-CG image of a current second non-CG image subset wherein the first encoder and the third encoder comprise the shared set of encoder trainable neural-network parameters; replacing third components of the third latent codes with third mean values to create third modified latent codes; causing a third decoder of the third autoencoder to create reconstructed second non-CG images based on the third modified latent codes; and updating trainable parameters of the first autoencoder based at least in part on the second reconstructed non-CG images and the third mean values; and training the fourth autoencoder with the one or more second CG image subsets by, for each of the one or more second CG image subsets: causing a fourth encoder of the fourth autoencoder to create fourth latent codes, each representative of one second CG image of a current second CG image subset wherein the first encoder and the fourth encoder comprise the shared set of encoder trainable neural- network parameters; replacing fourth components of the fourth latent codes withfourth mean values to create fourth modified latent codes; causing a fourth decoder of the fourth autoencoder to create second reconstructed CG images based on the fourth modified latent codes; and updating trainable parameters of the fourth autoencoder based at least in part on the reconstructed second CG images and the fourth mean values.

27. A method according to claim 26 or any other claim herein wherein employing a decoder of the model to infer an output image based on the modified input latent code comprises employing the third decoder to infer an output non-CG image of the second identity based on the modified input latent code wherein the non-CG output image corresponds to the input image except in that the non-CG output image of the second identity depicts the desired attributes (e.g. facial expression, head orientation and / or lighting orientation).

28. A method according to claim 26 or any other claim herein wherein employing a decoder of the model to infer an output image based on the modified input latent code comprises employing the fourth decoder to infer an output CG image of the second identity based on the modified input latent code wherein the CG output image corresponds to the input image except in that the CG output image of the second identity depicts the desired attributes (e.g. facial expression, head orientation and / or lighting orientation).

29. A method according to any one of claims 26 to 28 or any other claim herein wherein the input image comprises a CG input image of the first identity.

30. A method according to any one of claims 26 to 28 or any other claim herein wherein the input image comprises a CG input image of the second identity.31 . A method according to any one of claims 26 to 28 or any other claim herein wherein the input image comprises a non-CG input image of the first identity.

32. A method according to any one of claims 26 to 28 or any other claim herein wherein the input image comprises a non-CG input image of the second identity.

33. A method, performed on a computer, for training a model for manipulating one or more attributes (e.g. facial expression, head orientation and / or lighting orientation) of a depiction of a face in an input image, the method comprising: preparing training data comprising: preparing one or more non-CG image subsets each comprising non- CG images of a face of a non-CG identity and all having at least one common non-CG attribute (e.g. facial expression, head orientation or lighting orientation); preparing one or more CG image subsets each comprising CG images of a face of a CG identity and all having at least one common CG attribute (e.g. facial expression, head orientation or lighting orientation); training a model comprising first and second autoencoders by: training the first autoencoder with the one or more non-CG image subsets by, for each of the one or more non-CG image subsets: causing a first encoder of the first autoencoder to create first latent codes, each representative of one non-CG image of a current non-CG image subset; replacing first components of the first latent codes with first mean values to create first modified latent codes; causing a first decoder of the first autoencoder to create reconstructed non-CG images based on the first modified latent codes; and updating trainable parameters of the first autoencoder based at least in part on the reconstructed non-CG images and the first mean values; and training the second autoencoder with the one or more CG image subsets by, for each of the one or more CG image subsets: causing a second encoder of the second autoencoder to create second latent codes, each representative of one CG image of a current CG image subset wherein the first encoder and the secondencoder comprise a shared set of encoder trainable neural-network parameters; replacing second components of the second latent codes with second mean values to create second modified latent codes; causing a second decoder of the second autoencoder to create reconstructed CG images based on the second modified latent codes; and updating trainable parameters of the second autoencoder based at least in part on the reconstructed CG images and the second mean values.

34. A method according to claim 29 or any other claim herein wherein preparing the training data comprises: preparing one or more second non-CG image subsets each comprising second non-CG images of a second face of a second non-CG identity and all having at least one common second non-CG attribute (e.g. facial expression, head orientation or lighting orientation); preparing one or more second CG image subsets each comprising second CG images of a second face of a second CG identity and all having at least one common second CG attribute (e.g. facial expression, head orientation or lighting orientation); wherein the model comprises third and fourth autoencoders and training the model comprises: training the third autoencoder with the one or more second non-CG image subsets by, for each of the one or more second non-CG image subsets: causing a third encoder of the third autoencoder to create third latent codes, each representative of one second non-CG image of a current second non-CG image subset wherein the first encoder and the third encoder comprise the shared set of encoder trainable neural-network parameters; replacing third components of the third latent codes with third mean values to create third modified latent codes;causing a third decoder of the third autoencoder to create reconstructed second non-CG images based on the third modified latent codes; and updating trainable parameters of the first autoencoder based at least in part on the second reconstructed non-CG images and the third mean values; and training the fourth autoencoder with the one or more second CG image subsets by, for each of the one or more second CG image subsets: causing a fourth encoder of the fourth autoencoder to create fourth latent codes, each representative of one second CG image of a current second CG image subset wherein the first encoder and the fourth encoder comprise the shared set of encoder trainable neural- network parameters; replacing fourth components of the fourth latent codes with fourth mean values to create fourth modified latent codes; causing a fourth decoder of the fourth autoencoder to create second reconstructed CG images based on the fourth modified latent codes; and updating trainable parameters of the fourth autoencoder based at least in part on the reconstructed second CG images and the fourth mean values.

35. The method according to any one of claims 33 and 34 comprising any of the features, combinations of features and / or sub-combinations of features of any of claims 1 to 32 or any other claim herein, particularly those features, combinations of features and / or sub-combinations of features relating to training the face-morphing model.

36. A system comprising one or more processors, the one or more processors configured to perform any of the methods of claims 1 to 35.

Images