Method and apparatus for privacy-enhanced data item editing

Abstract

It is desirable having edited an image of a person to e.g. change pose, expression or illumination, that the person is still recognisable from the edited image and this application relates to facial i

Description

Field

[001] The present application generally relates to a method and apparatus for a more secure and privacy-conscious way to edit items of data. For example, the present techniques enable privacy-enhanced modification of images depicting human faces. This may be particularly useful when generating training datasets fortraining machine learning, ML, models, where the datasets are based on modifying or transforming original data. Background

[002] Image generation and editing are open problems of computer vision. Three techniques have been popularly employed for these tasks: variational autoencoders (VAEs), generative-adversarial networks (GANs) and diffusion models. Among them, GANs still remain as the most popular one, predominantly because of their photo-realistic and high-fidelity results coupled with faster inference times. GANs can synthesize high-resolution images from a random noise input, and generate photo-realistic results for human faces. Consequently, distinguishing outputs of GAN models from real images is not simple for human perception.

[003] Low-dimensional features in GAN spaces (i.e. the spaces of feature representations learned by GAN models) derive coarse and fine details of synthesized images. Intuitively, image contents can be edited by modifying these features. However, the contents embodied in representations residing on GAN spaces cannot be directly controlled for editing generated images with desired visual attributes by users, because of its entangled nature. This drawback limits the usability of GANs as a reliable generative tool. For this purpose, GAN spaces are disentangled to edit the contents with desired visual attributes of synthesized images while keeping the other target attributes unchanged.

[004] Since GAN models cannot guarantee users to manipulate contents of generated images explicitly, GAN features should be transferred to more controllable forms for image manipulation. Numerous methods compute plausible linear subspaces of features in GAN spaces for image manipulation in an unsupervised manner. Their major limitation is that they cannot truly decompose non-linear GAN spaces using linear projection methods. With additional supervision on image attributes, nonlinear models that learn to change the semantic content on predetermined concept sets are incorporated. Here, GAN features are moved along concept-specific features with nonlinear operations. However, labeling costs can be infeasible when the concept size and precision of labels are considered. Pseudo-labels emerge as a viable solution to provide the necessary supervision.

[005] The applicant has therefore identified the need for a more robust way to decompose representations of data items. Summary

[006] In a first approach of the present techniques, there is provided a computer-implemented method for a computer-implemented method for generating modified images from input images using a trained machine learning, ML, model, the method comprising: obtaining an image depicting a human face; using the trained ML model to: determine, in vector space, an identity representation for the human face in the obtained image, the identity representation encoding at least one specific identifying characteristic of the human face; determine, in vector space, at least one attribute representation for the human face in the obtained image, each attribute representation encoding an attribute of the human face in the obtained image; modify, in vector space, one or more of the at least one attribute representation; condition, in vector space, each modified attribute representation and each un-modified attribute representation with the identity representation; and generate a modified image using each conditioned modified attribute representation and each conditioned unmodified attribute representation.

[007] Typically, machine learning, ML, models can be used to generate modified images from input images. It is sometimes desirable to modify the features of human faces in images using such ML models. For example, a person may want to modify their face in an image to, for instance, change their facial expression. For instance, a person may wish to change a frown into a smile, or to change the illumination of their face, or to change their pose (e.g. from facing slightly left to facing slightly right). To do so, the ML model needs to understand which features can be modified. However, simply modifying a person’s mouth in an image to change it from a frown into a smile would not necessarily change the overall facial expression correctly. This is because people’s eyes and other parts of their face may change when they smile. Therefore, changing only one part of an image of a face may not correctly alter the facial expression.

[008] It is also desirable to ensure that a person’s face retains its defining characteristics after the editing. That is, it is desirable that the person is still recognisable from an edited image that has been edited to e.g. change pose, expression or illumination. Many existing techniques assume that facial features are independent of the features of a face which encode facial identity, but this is not true. Thus, existing techniques generate edited images in which the person’s facial identity has been altered as well as, for example, their expression.

[009] Advantageously, the present techniques enable modified images of faces to be generated by decoupling features of the image that represent facial identity / facial identifying characteristics, from features of the image which represent other attributes of the face in the image.

[010] The features representing facial identity or facial identifying characteristics are those which relate to the person’s identity. For example, the eye colour, eye shape, nose shape, forehead size, distance between eyes, mouth size, etc. may be features that relate to a person’s identity. Herein, the term “identity representation” is used to mean a representation, in vector space, which encodes at least one specific facial identity feature or facial identifying characteristic.

[011] The features representing other attributes of a face in an image may be those which are to be modified. For example, these other attributes may include pose of the face, illumination of the face, and so on. These features of the image do not help to determine the identity of the person in the image. Herein, the term “attribute representation” is used to mean a representation, in vector space, which encodes an attribute of the face in the image that is not an identifying feature or characteristic of the face.

[012] As noted above at least one attribute representation may be determined for the face in the image. When multiple attribute representations are determined, all of these do not necessarily have to be modified - it is possible for only some of the multiple attribute representations to be modified. Thus, the term “one or more of the at least one attribute representation” used above indicates that when more than one attribute representation is determined, it is possible to modify a subset of them.

[013] As noted above, after at least one attribute representation has been modified, it is necessary to condition each modified attribute representation and each un-modified attribute representation (if any) with the identity representation. That is, because the attribute representation(s) has been decoupled from the identity representation, it is now necessary to re-couple them so that the modification is applied to the actual face in the image. This recoupling is referred to as “conditioning”.

[014] The trained ML model may be or comprise an encoder-decoder neural network. Encoder-decoder neural networks or models typically comprise separate, fully-connected neural networks for the encoder and the decoder. Typically, an encoder-decoder model contains several encoders and several decoders.

[015] Thus, the step of determining at least one identity representation may comprise using a first encoder of the trained ML model. In some cases, the using a first encoder of the trained ML model may comprise using a transformer-based neural network trained to identify facial identity concepts. In other words, the first encoder may be a transformer-based model.

[016] Consequently, the step of determining at least one attribute representation may comprise using a set of second encoders of the trained ML model, where each second encoder of the set generates a representation of a specific attribute of the human face. Thus, one or more encoders of the model may be trained to determine / identify a specific attribute of the human face in an image that is not an identifying feature of the human face. The number of second encoders in the ML model may depend on how many specific attributes are required to be identified for the purpose of modifying images. For example, if it is only desirable to modify pose, then a single second encoder may be sufficient, but if it is desirable to modify pose, illumination, expression, and so on, then multiple second encoders may be required. In some cases, using a set of second encoders may comprise using a set of second transformerbased neural networks, where each second transformer-based neural network is trained to identify a specific attribute of human faces in images. In other words, each second encoder may be a transformer-based model.

[017] Generally speaking, the step of determining the at least one attribute representation may comprise determining any general feature(s) of faces that is not an identifying feature or characteristic of the face. More specifically, the step of determining the at least one attribute representation may comprise determining one or more of: a pose parameter relating to a head pose of the human face in the obtained image; an illumination parameter relating to illumination of the human face in the obtained image; and an expression parameter relating to a facial expression of the human face in the obtained image. It will be understood that these are non-limiting examples of attribute representations.

[018] Attention is a machine learning mechanism that directs an ML model to prioritise (or attend to) the part or parts of some data that is considered most relevant. Attention mechanisms often involve a queries, keys and values, and sequences of tokens. The tokens, or token embeddings, are fixed-width elements of a sequence. A query or query vector represents the information a token in the sequence is seeking. A key or key vector represents the information contained within each token. Alignment between query and key is used to determine attention weights. A value or value vector applies the attention-weighted information from the key vectors. Contributions from keys that are strongly-aligned with a query are weighted more heavily, while contributions from keys that are not relevant to a query are weighted closer to zero.

[019] The conditioning mechanism to re-couple the attribute representations with the identity representation may involve using an attention mechanism. This is useful because it ensures that identifying characteristics are not altered by the modification of the non-identifying characteristics.

[020] Thus, for each modified attribute representation and each un-modified attribute representation, the conditioning may comprise: using the identity representation as a key and a value in an attention mechanism; and using the modified attribute representation or unmodified attribute representation as a query in the attention mechanism.

[021] Generating a modified image may comprise concatenating each conditioned modified attribute representation and each conditioned un-modified attribute representation. In this way, the human face in the image is reconstructed using the conditioned attribute representations.

[022] The method may further comprise: obtaining a second identity representation of a second human face, the second human face being different to the human face in the obtained image; wherein conditioning each modified attribute representation and each un-modified attribute representation with the identity representation comprises using the second identity representation instead of the determined identity representation. In other words, it is possible to take the attribute representations from one image of a face, modify them, and then apply them to a different image of a face. This enables attributes such as pose or illumination from one image of a face to be applied to another image of a face. Crucially, no identifying features of the first human face is included within the attribute representations, and so no identifying features can be added to the image of the second human face or used for any other purpose. This advantage arises from the decoupling of the identifying features from the non-identifying features.

[023] More generally, as explained in more detail below, the present techniques consider a data item to be formed of two components - data content (e.g. facial identifying features) and semantic concepts (e.g. head pose). The decomposition methods of the present techniques enables the semantic concepts to be decoupled from the content, in vector space. Thus, the decoupling takes place in vector space, acting on the representation of the data item, rather than the original data item itself. Advantageously, the decomposition method means that manipulation / modification of semantic concepts does not interfere with or alter the data content itself, which provides improved data privacy particularly in cases where it is desirable to transfer the semantic concepts obtained from one data item to another data item.

[024] In a second approach of the present techniques, there is provided an apparatus for generating modified images from input images using a trained machine learning, ML, model, the apparatus comprising: a display; and at least one processor coupled to memory, for: obtaining an image depicting a human face; using the trained ML model to: determine, in vector space, an identity representation for the human face in the obtained image, the identity representation encoding at least one specific identifying characteristic of the human face; determine, in vector space, at least one attribute representation for the human face in the obtained image, each attribute representation encoding an attribute of the human face in the obtained image; modify, in vector space, one or more of the at least one attribute representation; condition, in vector space, each modified attribute representation and each un-modified attribute representation with the identity representation; and generate a modified image using each conditioned modified attribute representation and each conditioned unmodified attribute representation, and output on the display.

[025] The features described above with respect to the first approach apply equally to the second approach and therefore, for the sake of conciseness, are not repeated.

[026] As noted above, the apparatus comprises at least one processor and memory. The memory may store instructions that, when executed by the at least one processor individually or collectively, cause the at least one processor to perform the above-described steps.

[027] The apparatus may be a smart device. The apparatus may be a smartphone. A smartphone is an example of a smart device. The apparatus may be a smart appliance. A smart appliance is another example of a smart device. An example of a smart appliance is a smart television (TV), a smart fridge, a smart oven, a smart vacuum cleaner, a smart robotic device, and so on. More generally, the apparatus may be a constrained-resource device, but which has the minimum hardware capabilities to use a trained ML model and / or to train an ML model. The apparatus may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuum cleaner, smart lawn mower, smart oven, etc). It will be understood that this is a non-exhaustive and non-limiting list of example apparatuses.

[028] The apparatus may further comprise an image capture device for capturing images, and a storage storing captured images. Thus, obtaining an image depicting a human face may comprise obtaining an image from the captured images in the storage.

[029] The at least one processor may be further configured to: provide a graphical user interface, GUI, on the display; and request, via the graphical user interface, information on at least one attribute representation to be modified, and how the at least one attribute representation is to be modified. For example, the GUI may comprise a list of attribute representations that are determined for the obtained image, and how they can be modified. Thus, prior to the modifying, the method may comprise receiving user input on how to modify at least one attribute representation.

[030] In a third approach of the present techniques, there is provided a computer-implemented method fortraining a machine learning, ML, model to generate modified images from input images, the method comprising: obtaining a first training dataset comprising a plurality of target images depicting human faces; obtaining a second training dataset comprising a plurality of source images depicting human faces; and training the ML model using each image of the first and second training datasets, by: determining, in vector space, a target identity representation for the human face in a target image from the first training dataset, the identity representation encoding at least one specific identifying characteristic of the human face; determining, in vector space, at least one attribute representation for the human face in the target image, each attribute representation encoding an attribute of the human face in the source image; determining, in vector space, a source identity representation for the human face in a source image from the second training dataset; determining, in vector space, at least one attribute representation for the human face in the source image, each attribute representation encoding an attribute of the human face in the source image; replacing, in vector space, the determined at least one attribute representation for the human face in the source image with the determined at least one attribute representation for the human face in the target image; conditioning, in vector space, each replaced attribute representation of the source image with the source identity representation; generating a modified version of the source image using each conditioned modified attribute representation; and training the ML model to: maximise an editing precision score, the editing precision score defining how well the at least one attribute is transferred, during the replacing, from the target image to the modified version of the source image, and maximise an identity preservation score, the identity preservation score defining how well the identity representation for the human face in the source image is preserved, during the replacing, in the modified version of the source image.

[031] Thus, training the model comprises transferring attribute representations from one, target image to another, source image, without transferring the identity representation from the target image (thereby preserving privacy), and without modifying the identity representation in the source image.

[032] Determining at least one identity representation may comprise using a first encoder of the ML model. As noted above with respect to the first approach, using a first encoder of the ML model may comprise using a transformer-based neural network to identify identifying characteristics of human faces.

[033] Similarly, determining at least one attribute representation may comprise using a set of second encoders of the ML model, where each second encoder of the set generates a representation of a specific attribute of the human face. Using a set of second encoders may comprise using a set of second transformer-based neural networks, where each second transformer-based neural network identifies a specific attribute of human faces in images.

[034] Determining the at least one attribute representation may comprise determining one or more of: a pose parameter relating to a head pose of the human face; an illumination parameter relating to illumination of the human face; and an expression parameter relating to a facial expression of the human face.

[035] The training method may further comprise: calculating the identity preservation score by calculating a cosine similarity between the human face in the modified version of the source image and the human face in the source image. That is, by comparing the modified source image and the original source image, it is possible to determine how similar the human face between the images. Ideally, as noted above, no identity features from the target image have been transferred to the modified source image, and no identity features of the source image have been altered in the modified source image. The more similar the cosine similarity, the more the identity of the human face in the source image has been preserved in the modified source image.

[036] The method may further comprise: calculating the editing precision score by calculating a cosine similarity between each replaced attribute representation in the modified version of the source image with a corresponding determined attribute representation in the target image. Similarly, the more similar each attribute representation is in the modified source image to each corresponding attribute representation in the target image, the better the attribute representation has been transferred to the modified source image. The term “corresponding attribute representation” is used to mean the same attribute representation type. That is, if pose is replaced in the modified source image, then the pose in the modified source image is compared with the pose in the target image.

[037] Up to now, the present techniques have been described with reference to images. However, the present techniques can be used to generate modified versions of other data types, such as music, audio, and text.

[038] Thus, in a fourth approach of the present techniques, there is provided a computer-implemented method for generating modified data items from input data items using a trained machine learning, ML, model, the method comprising: obtaining a data item for modification; using the trained ML model to: determine, in vector space, an identity representation in the obtained data item, the identity representation encoding at least one specific identifying characteristic of content of the obtained data item; determine, in vector space, at least one attribute representation in the obtained data item, each attribute representation encoding an attribute of the content of the obtained data item; modify, in vector space, one or more of the at least one attribute representation; condition, in vector space, each modified attribute representation and each un-modified attribute representation with the identity representation; and generate a modified data item using each conditioned modified attribute representation and each conditioned un-modified attribute representation.

[039] The features described above with respect to the first approach apply equally to the fourth approach and therefore, for the sake of conciseness, are not repeated.

[040] The obtained data item may be an audio data item comprising the voice of a human speaker. In this case, determining the identity representation may comprise determining at least one specific identifying characteristic of the voice of the human speaker, and determining the at least one attribute representation comprises determining one or both of: an emotional style, and a language style.

[041] The obtained data item may be a text data item comprising information associated with or representative of a human author. In this case, determining the identity representation may comprise determining at least one specific identifying characteristic of the content of the text data item, and determining the at least one attribute representation comprises determining any one or more of: style, sentiment, and tense.

[042] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

[043] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[044] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[045] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[046] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[047] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[048] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[049] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[050] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[051] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[052] The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[053] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Brief description of the drawings

[054] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[055] Figure 1A is a schematic diagram illustrating how prior art techniques assume a representation comprises multiple independent semantic concepts;

[056] Figure 1B is a schematic diagram illustrating how the present techniques decouple semantic concepts from content of a representation;

[057] Figure 2 is a schematic diagram of an M L model architecture of the present techniques;

[058] Figure 3A shows robustness of the present techniques;

[059] Figure 3B shows improved privacy of the present techniques;

[060] Figure 4A shows a table comparing training time and memory usage of the present techniques (“ours”) against state-of-the-art baselines;

[061] Figure 4B is a table showing identity preservation scores measured by cosine similarlity;

[062] Figure 4G is a table showing editing precision scores with mean square error;

[063] Figure 4D is a table showing identity and editing scores under different configurations to demonstrate impact of the regularisation coefficient;

[064] Figure 4E shows results of an ablation study to analyse the impact of ICAM in terms of identity preservation;

[065] Figure 5 shows sample results when transferring pose, illumuniation and expression, and all three of these attributes simultaneously, from random target faces to the input (source) faces;

[066] Figure 6 shows a visual comparison of the present techniques (“ours”) with a state-of-the-art baseline;

[067] Figure 7 is a flowchart of example steps for generating modified images from input images using a trained machine learning, ML, model;

[068] Figure 8 is a flowchart of example steps for training a machine learning, ML, model to generate modified images from input images; and

[069] Figure 9 is a block diagram of an apparatus for generating modified images from input images using a trained machine learning, ML, model. Detailed description of the drawings

[070] Broadly speaking, embodiments of the present techniques provide a method and apparatus for a more secure and privacy-conscious way to edit items of data. For example, the present techniques enable privacy-enhanced modification of images depicting human faces. This may be particularly useful when generating training datasets for training machine learning, ML, models, where the datasets are based on modifying or transforming original data.

[071] Generative models can synthesize diverse and photo-realistic images that have demonstrated remarkable success in computer vision. Notably, Generative Adversarial Networks (GANs) trained for faces (i.e., StyleGAN2) can be considered as a powerful image generation pipeline. However, the entangled space of features learned by GANs restricts precise control of modifying the content of generated images. The present techniques introduce a framework designed to enhance the control for face editing in image generation by disentanglement of feature spaces of GANs (a.k.a. GAN spaces).

[072] In this way, the present techniques enable better control for modification of face concepts such as pose, expression, and illumination. For this purpose, the present techniques provide a framework that first learns multiple latent spaces for parameterization of face concepts learned by 3D Morphable face models. Then, it employs a Identity-Conditioned Attention Mechanism (ICAM) to decouple face identity representations from the parameterized face concepts. Moreover, it adapts variational autoencoders to model the hierarchical structure of GAN features by incorporating transformer networks for end-to-end optimization of model parameters. Results show that ICAM achieves state-of-the-art identity preservation and editing precision accuracy on benchmark datasets with improved training time and memory usage.

[073] The present techniques provide a novel method to manipulate GAN spaces for face editing. For this purpose, an encoder-decoder network model based on transformers is provided. First, a GAN space of features X is decomposed to independent latent spaces. Here, each i = 0,...,N has distinct face concepts such as pose, illumination, and expression, while D includes facial identity code. Compared to the state-of-the-art baseline, the present techniques provide a formulation that optimizes a unified network model for face editing rather than two separate models (i.e., decomposer and controller) as illustrated in Figures 1A and 1B.

[074] Figure 1A shows how, in the state-of-the-art methods, a GAN space X of features is first decomposed into an intermediate latent space Q by learning a mapping to a face parameter space J’ (decomposition step). Next, a GAN space controller is trained to edit the GAN space X (controller step). In other words, a representation (%) can be decomposed into multiple semantic concepts 0L, where i = 0,..., N., where these concepts control the structure of a representation by implementing p(x)~p(0o,0v ...,0N). Here, a vector representation (%) can be an audio feature for retrieval / search or a visual feature for face editing / search or a text feature for writing style editing, for example. Prior art techniques assume that each concept is independent from each others. Hence, the decomposition is employed and visually interpreted by p(0o,01, ...,0N)~p(0o)p(01) ...p(0Ny However, these prior methods oversimplify the problem by ignoring the dependencies, resulting a information leakage from the data content into semantic concepts in the learning step.

[075] As a result, existing techniques cannot handle two well-known cases - robustness and privacy. The present techniques provide a method that is robust to these cases.

[076] Robustness: The limitation breaks the robustness of prior-art against Adversarial Attacks. Adversarial attacks are ones that add imperceptible random noise to input images to disturb the content for machine learning-based models.

[077] Privacy: The limitation also introduces weakness to Membership Inference Attack. Membership Inference Attack is the process of determining whether a sample comes from the training dataset of a trained ML model or not. Simply, by reverse engineering, determining samples used in training step of a model.

[078] Existing techniques for representation decomposition are sensitive to adversarial attacks since they cannot decouple the data content from semantic concepts. To this end, controlling these concepts can be negatively affected by adversarial attacks since concepts and content alter simultaneously by adversarial attacks. For this purpose, the present techniques decouple the data content from semantic concepts so that controlling the concepts becomes robust to adversarial attacks with improved privacy.

[079] Figure 1B shows how, in contrast, in the present unified framework, decomposition and controller steps are optimized altogether. Furthermore, the present framework conditions face identity space 2) to face parameter spaces J’ in the reconstruction step, which forces the parameter spaces to learn generic and exclusive codes of face concepts disentangled from the face identities.

[080] In other words, the present method addresses the problem by introducing a data content feature d. Later, the present model conditions the content over other semantic concepts implementing the following decomposition of representations: £(00,0!, ..., QN, d)~p(0Q\d)p(0A\d) ...p(0N\d)p(d). This decomposition methods enable the concepts to be decoupled from the content of a representation, which means that the manipulation of concepts does not interfere with content for improved privacy. To this end, robust, private, and independent concepts can be obtained without affecting others under adversarial attacks.

[081] Ultimately, the present framework is trained by a reconstruction loss regularized by the VAE loss. This offers two advantages: 1) Since encoding and decoding steps are optimized together in an end-to-end training pipeline, estimated codes can converge to a stable solution faster. 2) Since the present framework’s outputs are assessed at the feature-level instead of the pixel-level, the present framework is more computationally efficient with faster training time and less memory requirement. Furthermore, the present Identity-Conditioned Attention Mechanism (ICAM) enables facial concepts to be conditioned by face identities using a neural network model. The present techniques introduce identity representation to each face parameter space, which facilitates capturing better decoupled codes related to face parameters in the encoder.

[082] Some advantages of the present techniques include: • To enhance a decoder model by controlling its dependency on facial identity, Identity-Conditioned Attention Mechanism (ICAM) is used. In this model, a facial identity code is conditioned lately over face parameters. The goal of this model is to decouple the face identity representations from face parameters (i.e. parameterized face concepts) in the encoder. Ultimately, this ensures that the manipulation of face concepts does not interfere with facial identity. • VAE-based formulation is leveraged with transformer networks to map the features in GAN spaces onto distinct face parameter spaces. This decomposition facilitates independent control over face parameters for face editing. Moreover, the use of VAE-based formulation allows integration of the decomposition (i.e., corresponds to the encoder model) and controller (i.e., corresponds to the decoder model) models into a unified model, which makes the unified model efficient and stable in the training step. Qualitative and quantitative analyses employing the present framework have been performed on benchmark datasets. The experimental results demonstrate the superiority of the present framework over the state-of-the art baseline.

[083] The present techniques are now described in detail with reference to modifying images of faces. However, it will be understood that this is one example use case of the present techniques, and is used to illustrate how the present techniques work. The present techniques may be used to generate modified versions of other data types that include content and semantic concepts, such as music, audio, speech, and text.

[084] In the present techniques, the face manipulation task is formulated by projecting face concepts onto distinct face parameter spaces. For this purpose, a face identity space 2) is constructed by learning from data, and the distribution of face parameters is conditioned with facial identities in the space 2). To this end, each face parameter can be edited simultaneously or independently to edit GAN features x, and synthesize a photo-realistic image using a pre-trained GAN model 6(-). Note that only latent features x of GAN models are edited, and the trainable components of GAN models are not updated in our framework. An architecture of our framework is illustrated in Figure 2.

[085] Figure 2 shows the overall architecture of the present framework. In the training step, a GAN feature x is projected onto multiple face parameter codes p; (a reparameterization technique denoted by J\T(-,-) and implemented by an VAE is utilized) and a face identity code d through transformer-based networks E((-). Then, the estimated face identity representation is incorporated into each face parameter code using Identity-Conditioned Attention Mechanism (ICAM) Fi(-). Finally, all conditioned face parameter codes and face identity codes are concatenated, and projected back to the original GAN space by reconstructing the output feature x with a fully-connected model S(-). In the inference step, face parameter codes are manipulated to control the GAN space.

[086] The present techniques therefore provide a novel method and system (encoderdecoder neural network) designed to enhance the control and robustness of representation decomposition with improved privacy. In particular, the present techniques propose a Content-Conditioned Attention Mechanism (CCAM) that implements the constraints required to achieve conditionally independent distributions of different concepts in the present method.

[087] Content-Conditioned Attention Mechanism (CCAM) is implemented as a module in the present system and the flow is implemented by / qt ■ kT\ A0; = softmax ( —-=— ) v, \ / where qt = 0^$, k = dWK and v = dWv. Here, 0; corresponds to one of the concepts. Moreover, W^s indicates trainable linear operators. 0; = 0; + A0;.

[088] Figure 3A shows robustness of the present techniques. The present method provides robustness towards adversarial attacks for faces by accumulating adversarial noise on a content feature d.

[089] Figure 3B shows improved privacy of the present techniques. This model is a pipeline for membership inference attacks. Since the embedding computed by the present method (i.e., backbone) is content independent, the present method protects privacy for membership inference attacks.

[090] Preliminaries

[091] GAN space X: GAN models generate a high-resolution image I using a random noise z which can be denoted by I = G(z). Among several GAN variants, StyleGAN2 projects the random noise into multi-resolution features obtained from various abstraction levels x = [xk]£=1. Hence, different abstraction levels contribute to the coarse and fine details of generated images. The limitation of the GAN space X is that manipulation methods cannot be directly employed on X for content editing, since the face representations are entangled in the GAN space.

[092] Face parameter space A face parameter space J’ contains parameterizations p = [Pi e IRD1]^0 of a set of 3D morphable face concepts. The present techniques study the concepts of facial expression, scene illumination and head pose. Therefore, the parameters p0 = 0 are used for controlling the facial expression, Pi = y for controlling the scene illumination, and p2 = P for controlling the head pose.

[093] Furthermore, each parameter p; can be independently controllable. Therefore, in the present framework, a face parameter space does not need to be decoupled for better user control. These parameters are estimated using pretrained encoder models based on the Basel face model.

[094] Face identity space 2): A face identity space 2) contains face identity representations learned using a model that is pretrained on a dataset of 2.6 million images of over 2600 people. Face representations are obtained at the output of the first fully-connected layer of the model.

[095] The Present Framework

[096] The objective is to optimize a model f that learns multiple latent spaces (i.e., the face parameter space J’ and face identity space 2)) during reconstruction of the GAN space X, and it can be expressed by f: X -> T* x 2) -> X where J’ x 2) is the product space of J’ and 2). Note that, the face parameter space J’ accommodates N concepts, and the present model is structured as an encoder-decoder architecture.

[097] Encoder: The encoder model aims to decompose and encode a GAN feature x to multiple latent codes. For this purpose, a model is adapted in which each GAN feature x is initially projected into an independent embedding for each latent space through a linear layer by e( = W(x, Vl Later, multiple transformer models £)(•) (see Figure 2) are used to capture the hierarchical dependencies between multi-resolution codes. Notably, the transformers employed in the present model estimate mean [Mi,k]k=i> Vi and variance [^)^=1, Vi similar to VAE models. Finally, latent codes p, are estimated for each face concept using a reparameterization technique.

[098] Decoder: In the decoder model, distributions of latent codes of face parameters [pj" 0 are independently conditioned by a facial identity code d. The goal of this model is to decouple the face identity from other face concepts provided by the encoder, and thereby to obtain more generic and independent face parameter spaces.

[099] Assuming that two probability distributions estimated on the GAN and face parameter spaces are aligned with the present model f by p(x) ~ p(p0,..., pw, d), it is possible to factorize the second distribution by P(Po— ,Pw,d) = p(d)n"0 p(pdd) (1) which imposes the conditional independence of face parameters. That is, face parameters Pi, j = 0,l,2, are distributed independent of each other conditioned on the facial identity parameter d. By incorporating this expression into the decoder model, more expressive (i.e., uncorrelated) parameter spaces can be estimated in the present encoder model. To implement this formulation in the present framework while learning representations and latent spaces, a novel condition-based attention model is proposed.

[100] Identity-conditioned Attention Mechanism (ICAM): Up to this point, all face and facial identity parameters are treated separately to impose the independence of parameters on the corresponding spaces in the encoder model.

[101] The proposed Identity-conditioned Attention Mechanism (ICAM) implements the constraints required to achieve conditionally independent distributions of parameters on spaces in models. More precisely, a code d of a face identity is employed as the key and the value in an attention mechanism, while the code corresponding to each face parameter p, is used as the query. The codes are projected using learnable weights WQ, and by a kT = softmax(^=A)v (2) where q, = ptWQ, k = dWK, v = dWv, <, >is the vector inner product, and n is the dimension of latent codes. Then, each parameter is updated by Pi = Pi + <5Pi- (3) This formulation implies that the parameter is queried on face identity and adapted to the identity space through this model. In the final layer, all updated face and identity parameters are concatenated and projected to the original GAN space.

[102] Loss Functions: The models in the present framework are trained by minimizing the following loss ■^total ^mse T ^KL-^KL’ (4) where AKL >0 is the regularization term. The present framework learns latent spaces by minimizing the reconstruction error for the learned GAN features x, face parameters p and facial identity parameters d by ^mse = I |X - X| |2 + | |p - Wpp| |2 + | |d - Wdd| |2, (5) where || • ||2 denotes the ^2 norm. Linear projections Wp and Wd are employed on the face parameter and facial identity representation before calculating the corresponding loss terms, since the dimensions of ground truth vectors and estimated vectors are different. Also, KL divergence loss is utilized by = (6) where I is the identity matrix, 0 is the zero vector. Furthermore, p. and a correspond to the mean and variance values estimated in the encoder model employing transformers, respectively.

[103] Experiments

[104] Implementation Details: An Adam optimizer with a learning rate of 0.001, a batch size of 128 and an iteration of SOK is used to train our framework. Transformers used in the encoder model are adapted, as suggested by Savas Ozkan et al (CHD).

[105] Training time and memory usage comparisons are given in Figure 4A. Figure 4A shows a table comparing training time and memory usage of the present techniques (“ours”) against state-of-the-art baselines. Since the present framework is implemented by a unified structure, the training time of the present model is approximately 30 minutes (using a single NVI DIA A10 GPU). On the other hand, the training time of the CHD model is approximately 1 hour (using 4 NVIDIA A10 GPUs). For GS, since specific principle components should be labeled by humans (no training), the training time is set as infinite («). For other baselines, reported results are used. Furthermore, since the CHD model learns a generative model (i.e, a shallow version of StyleGAN2) to map the GAN space to a disentangled intermediate latent space, it generates a larger memory footprint compared to the present framework

[106] Dataset: The present framework utilizes a pre-trained StyleGAN2 model trained with the FFHQ dataset. The training involves randomly sampling 200,000 features from the StyleGAN2 space to generate GAN features. Some latent features are hierarchically combined to increase the diversity of faces. For each face image, face parameters (i.e., pose, expression, and illumination) and face identity representation are computed. The synthetic dataset released by StyleFlow is utilized in the evaluation.

[107] Baseline and Metrics: For comparative analyses, five baseline works are selected: GANSpace (GS), StyleRig (SR), StyleFlow (SF) and Conceptual and Hierarchical Decomposition (CHD). Face identity and edit precision metrics are used to evaluate the models on the StyleFlow dataset for preparing qualitative analyses, as reported by Savas Ozkan et al. Also, the relative gain A = 100 • (Acc2 - Accr} / (Acc^ between accuracy of two methods is reported, providing Accx and Acc2 where Acc2 >Accr.

[108] Qualitative and Quantitative Analyses

[109] In the experiments, both qualitative and quantitative evaluations of the present framework are conducted on the StyleFlow dataset by comparing performance of the present framework with that of the baselines. The test setups involve the random transfer of pose, illumination and, expression face parameters from target faces to source faces. Later, edited faces are generated using the transferred parameters, and these edited faces are compared with source faces in terms of identity preservation and editing precision.

[110] To calculate identity preservation scores, first illumination (ilium), pose, and expression (expr) are transferred separately, and then all three face concepts (all) are transferred together. In each case, the transferred attribute is used in image generation. Then, for each transferred attribute or set of transferred attributes, the identity scores are computed by calculating cosine similarity between generated and source faces. For instance, if the generated and source faces are identical, then the identity preservation score is 1.0.

[111] The results given in Figure 4B. Figure 4B is a table showing identity preservation scores measured by cosine similarlity, which indicates that the present method achieves superior performance and outperforms the state-of-the-art baseline. This implies that the manipulation step does not significantly alter the facial identities in GAN spaces. Notably, the score obtained by editing all three face concepts together shows a substantial improvement compared to the state-of-the-art. The reason is that each face parameter space is estimated by disentangling facial identity representation. Hence, the contradiction in GAN spaces, particularly when all three concepts are simultaneously edited, is mitigated. Consequently, this ability helps to improve the performance.

[112] Furthermore, the previous test is reiterated by focusing on calculating the editing precision for different face concepts. The results are presented in Figure 4C. Figure 4C is a table showing editing precision scores with mean square error. Interestingly, the present method (“ours”) obtains results comparable to the state-of-the-art CHD model. The reason is that the state-of-the-art CHD model also leverages transformers in its encoder model, sharing a similar architecture of the present model. Although editing precision scores may not exhibit significant improvements, identity preservation scores are improved for face manipulation.

[113] The output of the present framework is shown in Figure 5. Figure 5 shows sample results provided by the present techniques when transferring pose, illumuniation and expression, and all three of these attributes simultaneously, from random target faces to the input (source) faces. The results show that other concepts are not altered when each face concept / attribute is modified individually. This ability is provided by the projection of face parameters into distinct latent spaces which is a property introduced by the present framework. Furthermore, when all face concepts / attributes are edited simultaneously, the present framework can generate consistent results and successfully handle multiple face parameter changes for face editing.

[114] Figure 6 shows a visual comparison of the present techniques (“ours”) with the state-of-the-art baseline CHD. Similarly, all facial parameters are transferred from target images to source images. From the results, the present method preserves facial details associated with source images in the edited images while improving editing performance. In particular, for the present method, hairstyle (second row) and eyeglasses (third row) are retained in the edited images. Moreover, illumination (first and third rows), expression (first row), and head pose (third row) are more similar to source images when the present techniques are used compared to when the baseline model is used.

[115] An Ablation Study for Analyzing the Impact of the AKL Coefficient on the Accuracy: The impact of the regularization coefficient AKL used in Eq. (4) on accuracy of the present model is assesed. For this purpose, models are trained using AKL e {2.0,1.0,0.4}. Similarly, identity preservation and editing precision scores are reported by randomly transferring face parameters from target faces to source faces. The results are given in Figure 4D, which is a table showing identity and editing scores under different configurations to demonstrate the impact of the regularisation coefficient. Notably, for large coefficient values, identities are better preserved for edited faces, but editing precision for all face concepts is reduced. The reason is that the model starts overfitting to identity codes when large coefficient values are used. Conversely, when smaller coefficient values are used, the contribution of identity codes is decreased, and face parameter codes become dominant in controlling GAN spaces.

[116] Impact of Identity-Conditioned Attention Mechanism (ICAM): The impact of ICAM is analysed in terms of identity preservation. For this purpose, a base setup is established, built upon the present framework by aggregating the latent features of identity d and face parameters P(S after the VAE instead of utilizing ICAM. The objective of ICAM is to decouple face identity from face parameters. Hence, identity preservation scores are reported in Figure 4E, similar to those reported in Figure 4B. Figure 4E shows results of an ablation study to analyse the impact of ICAM in terms of identity preservation. In this analysis, cosine similarity scores drop for the case where ICAM is not utilized (i.e., Ours w / o ICAM). To this end, these results demonstrate the importance of ICAM for GAN space face manipulation

[117] In summary, the present techniques address the disentanglement of feature space of GANs by introducing a novel framework. The objective is to enhance the control for modification of face concepts such as head pose, scene illumination, and facial expression in GAN space. The present framework is optimized end-to-end manner with a variational autoencoder formulation. Since GAN space contains multiple-resolution features, the formulation is adapted for hierarchical representations by incorporating transformer networks. Furthermore, a novel Identity-conditioned Attention Mechanism (ICAM) is introduced where a latent space of codes is conditioned over other latent spaces of codes through a neural network model. Specifically, in the present case, face identities are conditioned over face parameters in the decoder model, facilitating the decoupling of identity representation during parameter space learning. Hence, generic and exclusive parameter spaces independent from identities can be computed. Consequently, the present framework allows to control the content of generated images. Since the present framework does not alter the trainable components of pre-trained StyleGAN2 models, the photo-realism is maintained. Experimental results demonstrate that the present method outperforms baselines in terms of identity preservation and editing precision.

[118] Figure 7 is a flowchart of example steps for generating modified images from input images using a trained machine learning, ML, model. The method is computer-implemented and comprises: obtaining an image depicting a human face (step S100); using the trained ML model to: determine, in vector space, an identity representation d for the human face in the obtained image, the identity representation encoding at least one specific identifying characteristic of the human face (step S102); determine, in vector space, at least one attribute representation for the human face in the obtained image, each attribute representation encoding an attribute of the human face in the obtained image (step S104); modify, in vector space, one or more of the at least one attribute representation (step S106); condition, in vector space, each modified attribute representation and each un-modified attribute representation with the identity representation (step S108); and generate a modified image using each conditioned modified attribute representation and each conditioned un-modified attribute representation (step S110).

[119] The step (S102) of determining at least one identity representation may comprise using a first encoder of the trained ML model. In some cases, the using a first encoder of the trained ML model may comprise using a transformer-based neural network trained to identify facial identity concepts. In other words, the first encoder may be a transformer-based model.

[120] Consequently, the step (S104) of determining at least one attribute representation may comprise using a set of second encoders of the trained ML model, where each second encoder of the set generates a representation of a specific attribute of the human face. Thus, one or more encoders of the model may be trained to determine / identify a specific attribute of the human face in an image that is not an identifying feature of the human face. The number of second encoders in the ML model may depend on how many specific attributes are required to be identified for the purpose of modifying images. For example, if it is only desirable to modify pose, then a single second encoder may be sufficient, but if it is desirable to modify pose, illumination, expression, and so on, then multiple second encoders may be required. In some cases, using a set of second encoders may comprise using a set of second transformerbased neural networks, where each second transformer-based neural network is trained to identify a specific attribute of human faces in images. In other words, each second encoder may be a transformer-based model.

[121] For each modified attribute representation and each un-modified attribute representation, the conditioning step (S108) may comprise: using the identity representation as a key and a value in an attention mechanism; and using the modified attribute representation or un-modified attribute representation as a query in the attention mechanism.

[122] Figure 8 is a flowchart of example steps for training a machine learning, ML, model to generate modified images from input images. The method is computer-implemented and comprises: obtaining a first training dataset comprising a plurality of target images depicting human faces and obtaining a second training dataset comprising a plurality of source images depicting human faces (step S200); and training the ML model using each image of the first and second training datasets, by: determining, in vector space, a target identity representation for the human face in a target image from the first training dataset, the identity representation encoding at least one specific identifying characteristic of the human face (step S202); determining, in vector space, at least one attribute representation for the human face in the target image, each attribute representation encoding an attribute of the human face in the source image (step S204); determining, in vector space, a source identity representation for the human face in a source image from the second training dataset (step S206); determining, in vector space, at least one attribute representation for the human face in the source image, each attribute representation encoding an attribute of the human face in the source image (step S208); replacing, in vector space, the determined at least one attribute representation for the human face in the source image with the determined at least one attribute representation for the human face in the target image (step S210); conditioning, in vector space, each replaced attribute representation of the source image with the source identity representation (step S212); generating a modified version of the source image using each conditioned modified attribute representation (step S114); and training the ML model to: maximise an editing precision score, the editing precision score defining how well the at least one attribute is transferred, during the replacing, from the target image to the modified version of the source image, and maximise an identity preservation score, the identity preservation score defining how well the identity representation for the human face in the source image is preserved, during the replacing, in the modified version of the source image. The editing precision score and identity preservation score are discussed above with reference to the qualitative and quantitative analyses.

[123] The training method may further comprise: calculating the identity preservation score by calculating a cosine similarity between the human face in the modified version of the source image and the human face in the source image. That is, by comparing the modified source image and the original source image, it is possible to determine how similar the human face between the images. Ideally, as noted above, no identity features from the target image have been transferred to the modified source image, and no identity features of the source image have been altered in the modified source image. The more similar the cosine similarity, the more the identity of the human face in the source image has been preserved in the modified source image.

[124] The method may further comprise: calculating the editing precision score by calculating a cosine similarity between each replaced attribute representation in the modified version of the source image with a corresponding determined attribute representation in the target image. Similarly, the more similar each attribute representation is in the modified source image to each corresponding attribute representation in the target image, the better the attribute representation has been transferred to the modified source image. The term “corresponding attribute representation” is used to mean the same attribute representation type. That is, if pose is replaced in the modified source image, then the pose in the modified source image is compared with the pose in the target image.

[125] Figure 9 is a block diagram of an apparatus 100 for generating modified images from input images using a trained machine learning, ML, model 106, the apparatus 100 comprising: a display 108; and at least one processor 102 coupled to memory 104, for: obtaining an image depicting a human face; using the trained ML model to: determine, in vector space, an identity representation for the human face in the obtained image, the identity representation encoding at least one specific identifying characteristic of the human face; determine, in vector space, at least one attribute representation for the human face in the obtained image, each attribute representation encoding an attribute of the human face in the obtained image; modify, in vector space, one or more of the at least one attribute representation; condition, in vector space, each modified attribute representation and each un-modified attribute representation with the identity representation; and generate a modified image using each conditioned modified attribute representation and each conditioned un-modified attribute representation, and output on the display 108.

[126] As noted above, the apparatus 100 comprises at least one processor 102 and memory 104. The memory may store instructions that, when executed by the at least one processor individually or collectively, cause the at least one processor to perform the above-described steps.

[127] The apparatus 100 may be a smart device. The apparatus may be a smartphone. A smartphone is an example of a smart device. The apparatus may be a smart appliance. A smart appliance is another example of a smart device. An example of a smart appliance is a smart television (TV), a smart fridge, a smart oven, a smart vacuum cleaner, a smart robotic device, and so on. More generally, the apparatus may be a constrained-resource device, but which has the minimum hardware capabilities to use a trained ML model and / or to train an ML model. The apparatus may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuum cleaner, smart lawn mower, smart oven, etc). It will be understood that this is a non-exhaustive and non-limiting list of example apparatuses.

[128] The apparatus 100 may further comprise an image capture device or camera 110 for capturing images, and a storage 114 storing captured images. Thus, obtaining an image depicting a human face may comprise obtaining an image from the captured images in the storage.

[129] The at least one processor 102 may be further configured to: provide a graphical user interface, GUI, 112 on the display 108; and request, via the graphical user interface 112, information on at least one attribute representation to be modified, and how the at least one attribute representation is to be modified. For example, the GUI may comprise a list of attribute representations that are determined for the obtained image, and how they can be modified. Thus, prior to the modifying, the processor may receive user input, via the GUI, on how to modify at least one attribute representation.

[130] References: • StyleGAN2 (and FFHQ dataset) - Tero Karras, Samuli Laine, and Timo Aila, “A stylebased generator architecture for generative adversarial networks,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019, pp. 4401-4410. • Pretrained encoder models - Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart, “Learning an animatable detailed 3d face model from in-the-wild images,” ACM Transactions on Graphics (ToG), vol. 40, no. 4, pp. 1-13, 2021 • Basel face model - Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter, “A 3d face model for pose and illumination invariant face recognition,” in 2009 sixth IEEE international conference on advanced video and signal based surveillance, leee, 2009, pp. 296-301. • Face identity representation model - Omkar Parkhi, Andrea Vedaldi, and Andrew Zisserman, “Deep face recognition,” in BMVC 2015-Proceedings of the British Machine Vision Conference 2015. British Machine Vision Association, 2015. • Adam optimiser - Diederik P Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv: 1412.6980, 2014. • Savas Ozkan etal (CHD, i.e. Conceptual and Hierarchical Decomposition) - Savas Ozkan, Mete Ozay, and Tom Robinson, “Conceptual and hierarchical latent space decomposition for face editing,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp. 7248-7257. • GANSpace (GS) - Erik Harkonen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris, “Ganspace: Discovering interpretable gan controls,” Advances in Neural Information Processing Systems, vol. 33, pp. 9841-9850, 2020. • StyleRig (SR) - Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Perez, Michael' Zollhofer, and Christian Theobalt, “Stylerig: Rigging stylegan for 3d control over portrait images,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6142-6151 • StyleFlow (SF) - Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka, “Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows,” ACM Transactions on Graphics (ToG), vol. 40, no. 3, pp. 1-21, 2021.

[131] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Images