Methods and devices for generating avatar animation

Abstract

Adapting a first animation system to animate a target avatar of a second animation system, comprising: obtaining, from the first system, a set of source meshes S100, each source mesh represents a faci

Description

[002] Avatar facial animation is typically generated using an animation system (which may also be termed a tracking system and / or a capture system). Such systems track user expressions and uses parametrized linear models of facial expressions to generate avatar animations having the tracked user expressions. These parametrized linear models may be termed blend shapes or meshes and a desired facial expression may be obtained by combining a set of meshes using weights. The combined set of meshes may include a neutral expression mesh. The avatars are virtual characters. The blend shapes are thus typically designed for a specific avatar, more particularly to the neutral expression mesh for the avatar. A blend shape for one avatar cannot typically be replicated from one avatar onto another avatar. This may occur because the avatars often have exaggerated facial proportions and can have different facial proportions. Thus, transferring an expression from one avatar to another can lead to unnatural facial expressions and deformations.

[003] There are solutions which use a pairing between facial landmarks and avatar blend face shapes. However, this pairing needs to be manually generated, e.g. by an artist, for each new face mesh. This is necessary because a tracking system is normally designed to maps facial landmarks for a tracked user onto a set of blend shapes for a specific avatar. It may be possible to generate a machine learning solution which can generate the pairing. However, such an ML solution would require a large amount of training data.

[004] The present applicant has identified the need for an improved technique to adjust the output of a first animation system to work with an avatar from a second animation system. Summary

[005] In a first approach of the present techniques, there is provided a computer-implemented method for automated adaptation of a first animation system to animate a target avatar of a second animation system. The method comprises: obtaining, from the first animation system, a set of source meshes, wherein each source mesh represents a facial expression on a source avatar in the first animation system and each source mesh comprises a plurality of vertices; and obtaining, from the second animation system, a set of target meshes, wherein each target mesh represents a facial expression on the target avatar from the second animation system and each target mesh comprises a plurality of vertices. The method further comprises generating, from the set of source meshes, a set of sparse source meshes, wherein each sparse source mesh comprises a subset of the plurality of vertices within a corresponding source mesh; generating, from the set of target meshes, a set of sparse target meshes wherein each sparse target mesh comprises a subset of the plurality of vertices within a corresponding target mesh; and pairing each of the sparse source meshes with at least one of the sparse target meshes. The method further comprises defining a mapping to map a desired facial expression on the source avatar to a corresponding facial expression on the target avatar, wherein the desired expression on the source avatar is formed using a combination of source meshes from the set of source meshes; constructing, using the set of sparse target meshes and the set of sparse source meshes, a cost function which optimises the mapping to minimise error between the desired expression on the source avatar and the corresponding facial expression on the target avatar; determining, using the cost function, an optimised mappinFgieg; and using the optimised mapping to adapt the first animation system to animate the target avatar.

[006] Advantageously, the present techniques mean that an animated avatar with a desired expression is generated by the first animated system using a combination of source meshes and is then automatically mapped to the second animation system so that an animated avatar with the same desired expression, but a different appearance is generated. The output animated avatar has the appearance of being generated by a combination of target meshes from the second, different animation system. For example, the desired expression on the target avatar may be formed using a weighted combination of the set of target meshes. The mapping is automatically, rather than manually generated. The use of the sparse source and target meshes facilitates the automated adaptation on device by ensuring that there is at least a partial bijection when applying the mapping (as explained below).

[007] Constructing the cost function may comprise constructing a quadratic programming problem, QPP which comprises at least one quadratic term and at least one linear term and determining, using the cost function, an optimised mapping comprises solving the QPP. In general terms, a QPP may be defined as f(x) = ^xTQx + bTx + c where Q is a quadratic term and b is a linear term and c is a constant and the function is subject to equality and inequality constraints, for example: ajx = bt i E E ajx >bt i E I where a is a list (^, b^,... (an, bn), E is the subset of indices of those vectors at and constants bt used for Equality constraints and I is the complement of E, so used for inequality). By defining the cost function as a QPP, off-the-shelf quadratic programming solvers (for example using active sets defined in the textbook “Numerical Optimization”, by Nocedal et al published as second edition) may be used to solve the problem and hence optimise the mapping. For example, Q may be a symmetric square matrix which is positive definite and the constant term c may be ignored because it will not affect the minimum. This further enables, automatic adaptation, particularly on device.

[008] The set of source meshes may comprise a baseline source mesh which represents a neutral expression (i.e. a baseline expression from which all other expressions may be constructed or derived). The desired expression on the source avatar may be formed using a weighted sum of the distance between each of the plurality of vertices in the first source mesh and each of the corresponding vertices in each of the set of source meshes combined with locations of the plurality of vertices in the baseline source mesh. Similarly, the set of target meshes may comprise a baseline target mesh which represents a neutral expression (i.e. a baseline expression from which all other expressions may be constructed or derived).

[009] Each mesh in the set of source and target meshes (with or without a baseline mesh) may comprises a plurality of vertices. Vertices may be termed locations or points and may be defined using any suitable geometrical system, e.g. a Cartesian system having points defined by (x,y, z) co-ordinates. The vertices may be represented as a vertex list, e.g. a matrix in [R11 / 1x3 or may be flattened into a vector V e ]R3|l / |. Each mesh may also comprise a plurality of faces, e.g. triangles or other shapes, and the plurality of faces may be represented as a face list, e.g. a matrix in N|F|><3. When using a matrix, every entry of the vertex list may be represented as a row (x^y^) for 0 <i <|V| - 1 and when using a vector, every entry of the vertex list may be of the form: = [xo>yo>zo>xi>yi>zi>■■■ ■ Every entry of the face list may be a triple of indices for the rows of the vertex list (tQ i, tu, t2ii) for 0 <i <|F| — 1. When extracting vertices of a specific face using the vector representation, 3tj i rather than just t]:i (so we can extract the stride is used.

[010] The desired expression on the target avatar may be formed using a weighted sum of the distance between each of the plurality of vertices in the baseline target mesh and each of the corresponding vertices in each target mesh in the set of target meshes combined with locations of the plurality of vertices in the baseline target mesh. The distances may be calculated using the sparse source meshes and / or the sparse target meshes respectively.

[011] The desired facial expressions for the source and / or target avatar may be calculated using the distances as follows: V(M)(a) = V(M0) + - V(M0». where F(M0) represents the vertex list of the neutral expression which is to be transformed to the vertex list of the desired expression V(M)(a) (for the source avatar or the target avatar respectively), V(Mi) is the vertex list of each mesh (target mesh or source mesh, respectively) and at is the blending weight which blends the baseline expression with the ith expression. When using the representation of the plurality of vertices as a vector, the expression above may be alternatively written as: F(M)(a) = F(M0) + ..., Mw) • a Here is a matrix of size 3|F| xN and contains for every column the delta expression for the i-th expression. Each distance value or delta expression may be termed a displacement because the value represents the displacement of each vertex in a (sparse) target / source mesh from the corresponding vertex in the baseline (sparse) target / source mesh. Generally, defining the mapping may comprise defining the mapping as a set of mapping weights comprising a mapping weight for each pair of sparse source and target meshes. More specially, defining a mapping may comprising defining a mapping which map the displacements which are used to generate a corresponding facial expression on the target avatar to the displacements which are used to generate the desired facial expression on a source avatar, i.e. a mapping weight for each pair of displacements wherein the first displacement in the pair is between a sparse source mesh and corresponding baseline source mesh and the second displacement in the pair is between a sparse target mesh and corresponding baseline target mesh.

[012] The method may comprise pairing each of the sparse source meshes with at least one of the sparse target meshes. The method may comprise calculating, using the subsets of vertices of the sparse source and target meshes, a correlation value for each pair of sparse source and target meshes. S subsets of the sparse source meshes may be selected using the calculated correlation values. This may be achieved by defining, using the calculated correlation values for each pair of sparse source and target meshes, a set of constraints for the cost function. For example, the set of constraints may be defined by: ( 0 c / = J c / ’1 U {(j, 1)} l I 1 (c / -1 U {( / ,0)} j = -1 ^(Xj) — ^correlation correlation where Hcorreiation is the upper correlation threshold, Lcorrelation is the lower correlation threshold, 0 is a value of a weighting to be applied, C(i, j) is the correlation value between the ith sparse source mesh and the jth sparse target mesh. In other words, the constraints may be defined so that when the correlation value is greater than or equal to the upper correlation threshold, the sparse source mesh may be retained or selected when optimising the mapping. This may for example be achieved by defining the constraint so that a first fixed mapping weight (e.g. a weight of 1, when the weights are normalised) is applied when determining the optimised mapping. When the correlation value is below the lower correlation threshold, the sparse source mesh may be rejected or not selected when optimising the mapping. This may for example be achieved by defining the constraint so that a second fixed mapping weight (e.g. a weight of 0) is applied when determining the optimised mapping. When the correlation value is above the lower correlation threshold and below the upper correlation threshold, the sparse source mesh may be retained or selected during the optimising and a variable mapping weight may be applied because the mapping for this sparse source mesh needs to be determined. In other words, constructing the cost function may comprise defining a set of constraints for different correlation values. The set of constraints may be applied when optimising the mapping, more specifically when solving the quadratic programming problem to optimise the mapping.

[013] The constraints may be used to select a subset of the set of sparse source meshes. Selecting based on the calculated correlation values may comprise using an upper correlation threshold and a lower correlation threshold. Selecting the subset may comprise selecting sparse source meshes which are highly correlated with sparse target meshes. In other words, when the correlation value is greater than or equal to an upper correlation threshold, the sparse source mesh may be retained and in this way a first subset of the set of sparse source meshes may be selected. When a sparse source mesh is uncorrelated with a sparse target mesh, the uncorrelated sparse source mesh is preferably rejected. In other words, when the correlation value is below a lower correlation threshold, the sparse source mesh may be rejected. Selecting the subset may further comprise selecting a second subset of the set of sparse source meshes which have correlation values between the upper and lower correlation threshold.

[014] The mapping may be expressed as a matrix of mapping weight values Wsource^target. In other words, there may be a mapping which is expressed as a set of mapping weights for each pair of sparse source and sparse target meshes. There may be Ns source meshes (or blendshapes) and NT target meshes (blendshapes) and the mapping may be a linear transformation which transforms the blending weights of the source or first animation system (^source) into the blending weights of the target or second animation system (atarget). This may be expressed as: ^source^target^source ^target This linear transformation may be termed “pulling back” because it is a representation of the pullback of an expression on the source mesh onto the target mesh (provided the mapping between the two geometries is quasi isometric, which is generally the case when targeting humanoid faces).

[015] The linear mapping is optimised to minimise the difference between the error between the desired expression on the source avatar and the corresponding facial expression on the target avatar. In other words, Wsource^target may be selected by choosing the values of the weights so that the following expression is minimised: 1 - src - 2 f ^tgt)^source^target^source dB (^src)®source || 2 This minimization however makes Wsource^target implicitly a function of asource which is undesirable. Instead, the mapping may alternatively or equivalently be specifically expressed as AR_ ad src aD vv source^target where is the complete set of distances (displacements) between each of the plurality of vertices in the baseline target mesh and each of the corresponding vertices in each of the other target meshes which may be termed a target displacement matrix and ABsrc is the complete set of distances (displacements) between each of the plurality of vertices in the baseline target mesh and each of the corresponding vertices in each of the other target meshes which may be termed a source displacement matrix. In other words, the optimisation is now independent of asource.

[016] In other words, defining a mapping may comprise defining a source displacement matrix ABsrc which includes distances between each of the vertices in each source mesh in the set of source meshes and each of the corresponding plurality of vertices in the baseline source mesh; defining a target displacement matrix &Bt9t which includes distances between each of the vertices in each target mesh in the set of target meshes and each of the corresponding plurality of vertices in the baseline target mesh; and defining a mapping between the source and target displacement matrices. When optimising the mapping to map a desired expression on the source avatar to the target avatar, the mapping weight values are optimised preferably without changing to the blending weights which are applied to the distance between locations of vertices.

[017] The distances between each of the plurality of vertices in the baseline target mesh and each of the corresponding vertices in an ith target mesh may be expressed as a vector of values, e.g. Sb^. Similarly, the distances between each of the plurality of vertices in the baseline source mesh and each of the corresponding vertices in a jth source mesh may be expressed as a vector of values, e.g. 8b-rc. The method may comprise defining separate mappings for each source mesh by separating the source displacement matrix into source displacement vectors for each source mesh. For example, the method may comprise defining a separate mapping for each source mesh using: AR^^ / n — Ahsrc lad ^source^targetj uuj where 8b-rc is a jth source displacement vector, ^Btgt is as defined above, and ^source^target.jis a vector of weights from the full mapping matrix. The weight matrix may comprise a plurality of vectors of mapping weights wherein each vector is for one source mesh in the set of source meshes and each vector comprises a weight to be applied to each sparse target mesh in the set of sparse target meshes. In other words, defining the mapping may comprise defining a source displacement matrix which comprises a plurality of columns or rows each of which represents a source displacement vector for each source mesh in the set of source meshes, wherein each source displacement vector comprises displacement distances between each of the vertices in the baseline source mesh and each of the corresponding vertices in one source mesh; defining a target displacement matrix which comprises a plurality of columns or rows each of which comprises a target displacement vector for each target mesh in the set of target meshes, wherein each target displacement vector comprises displacement distances between each of the vertices in the baseline target mesh and each of the corresponding vertices in one target mesh; and defining a mapping between the source displacement matrix and the target displacement matrix.

[018] When using the sparse meshes, the ith sparse target mesh may be similarly expressed as a vector and the notation may be used. When using the sparse meshes, the jth sparse source mesh may be similarly expressed as a vector and the notation 8bfrc(Lsrc) may be used. The distances between each of the plurality of vertices in the baseline target mesh and each of the corresponding vertices in each of the other target meshes may be expressed as a displacement matrix which combines the vectors, for example: AB^L^) = [5 / ^(1?^),..., where ABtflt(Ltgt) is the complete set of distances between each of the plurality of vertices in the baseline target mesh and each of the corresponding vertices in each of the other sparse target meshes and may be termed the sparse target displacement matrix.

[019] Constructing the cost function may comprise constructing a separate or individual cost function for each sparse source mesh and where there is selection of sparse source meshes, a separate or individual cost function for each selected sparse source mesh. When the set of constraints is used to select a subset of the sparse source meshes, a separate or individual cost function may be constructed for each selected sparse source mesh. The quadratic term which is the same for each separate cost function may be calculated from the sparse target displacement matrix for example using Q = [AB^CL^fAB^CL^) where ABtflt(Ltgt) is the sparse target displacement matrix and [ABtgt(Ltgt)]r is the transpose of the sparse target displacement matrix. The linear term for each cost function may be defined using the sparse target displacement matrix and a sparse source displacement vector, for example using bi = [ABtgt(Ltgt)]T6b?rc(Lsrc) where [ABtgt(Ltgt)]r is the transpose of the sparse target displacement matrix and 8b?rc(Lsrc) is the source displacement vector of the ith sparse source mesh. In other words, defining the quadratic term may comprise using a sparse target displacement matrix which comprises (e.g. as a column or row) a target displacement vector for each sparse target mesh in the set of sparse target meshes, wherein each sparse target displacement vector comprises displacement distances between each of the vertices in the baseline target mesh and each of the corresponding vertices in a sparse target mesh. Similarly, defining a different linear term for each source mesh may comprise using the sparse target displacement matrix and a sparse source displacement vector, wherein the sparse source displacement vector comprises displacement distances between each of the vertices in the baseline source mesh and each of the corresponding vertices in one sparse source mesh.

[020] As set out above, the desired expressions for the source and / or target avatar may be calculated using the following expression: V(M)(a) = V(M0) + - V(M0». In the first animation system, a set of N animations (i.e. a set of N desired facial expressions) may be represented as Ns blending weights where Ns is the number of source meshes. The blending weights for the first animation system may be termed source blending weights. The set of blending weights may be defined as the weights to generate each of the desired facial expressions on the source avatar by blending the set of source meshes with the baseline mesh. It will be appreciated that there are also blending weights for the second animation system which may be represented as NT blending weights {a'i(t)}" r, where NT is the number of target meshes. This animation data may may be considered as manifestations of a vector valued stochastic process, namely a family of random variables. We are seeking a linear transformation in the form of a set of mapping weights W e [R®*® which minimise the cost function e(t) (also termed an error function). Since the blending weights are a stochastic process, the error function may be defined as a stochastic process, for example: e(t) = ABTWa(t) - ABsa(t) = (ABTW - ABs)a(t) where {«,( / :) )^ are the blending weights, ABT is the sparse target displacement matrix (for ease of notation the (L) which is used above has been omitted), ABS is the sparse source displacement matrix, and W is a matrix of remapping weights. Constructing the cost function may further comprise calculating the first order statistics, e.g. the mean (i.e. average) and the covariance matrix for the error and hence for the blending weights. The goal is to have an average error and variance which is as small as possible but still preserves semantic details.

[021] In this example, the quadratic problem may be characterised by: r T 1 r = t (IE [a] (t) IE [a] (t)T + pX[a](t)^dt <o Q = Afi[a]®AB^ABr k b = —vec^Bg)7 (AM[a]®ABT) where IE[a] is the mean of the blending weights, S[a] is the covariance of the blending weights, is the sparse target displacement matrix (for ease of notation the (L) which is used above has been omitted), ABS is the sparse source displacement matrix, and W is a matrix of remapping weights and where p. >0 and is a constant.

[022] The quadratic term may be calculated from the sparse target displacement matrix and a matrix operator which is calculated from the mean and the covariance of the blending weights for example using: Q = AM[a]®AB^ABT where T 1 r = — I (E[a](t)E[a](t)T + JuS[a](t))dt o where E[a] is the mean of the blending weights, S[a] is the covariance of the blending weights, ABT is the sparse target displacement matrix (for ease of notation the (L) which is used above has been omitted), and where p. >0 and is a constant. The linear term may be calculated from the sparse target displacement matrix, the sparse source displacement matrix, and the matrix operator, for example using: b = — vec(ABs)T(AM[a]®ABT) where ABS is the sparse source displacement matrix and the other terms, including the matrix operator A^ [a]® are defined above.

[023] Generating a set of sparse source meshes may comprise obtaining, from a user or any other source, a set of source landmarks, wherein each source landmark has a location which is defined by a source vertex (or index - terms may be used interchangeably). Generating a sparse source mesh for each source mesh may comprise comparing the plurality of vertices within the source mesh with the locations of the source vertices of the set of source landmarks; and selecting the subset of the plurality of vertices which match the locations of the source vertices of the set of source landmarks, and defining the sparse source mesh using the selected subset of the plurality of vertices. The number of vertices in the sparse source mesh may be reduced further by using only a subset of the source landmarks, e.g. using only a subset of the indices indicating locations of the source landmarks on a source mesh. A similar process may be carried out to generate the set of sparse target meshes. It will be appreciated that by retaining only the landmarks (or even only a subset of the landmarks), the sparse source mesh has less information than the corresponding source mesh and similarly, the sparse target mesh has less information than the corresponding target mesh.

[024] Other ways of reducing the information in the sparse source or target mesh relative to the corresponding source or target mesh may be used. One aim may be to have the number of vertices in each sparse source mesh equal to the number of vertices in each sparse target mesh so that there is a bijection between the sparse source meshes and the sparse target meshes. In other words, an equal number of vertices for each sparse source mesh and each sparse target mesh may be retained. In other words, there may be a selection of the same number of vertices when using the landmarks as described above.

[025] There may be the same number of the source and target meshes or there may be a different number of source meshes (Ns) and target meshes (Nt).

[026] The generation of the set of source and target meshes may be considered to be a preprocessing step. There may optionally be further pre-processing, for example to remove seams and / or to make the mesh watertight by removing any holes, e.g. for the eyes or mouth.

[027] Calculating the correlation value may comprise calculating the Pearson correlation. For example, the correlation between the ith sparse source mesh and the jth sparse target mesh may be defined as the cosine distance: ^b^ ,8blrc} )= ii ii where 8btgtj is a vector of distance values for the jth sparse target mesh and 8bsrct is a vector of distance values for the ith sparse source mesh.

[028] In a second approach of the present techniques, there is provided a computer-implemented method of generating a target avatar of a second animation system using a first animation system which has been adapted as described above. The method of generating the target avatar may comprise receiving an image of a user having a desired facial expression; receiving a selection of the target avatar; generating, using the first animation system, a source avatar having the facial desired expression; and generating the selected target avatar by applying the mapping to the generated source avatar. The image may be a single image or may be a video. The selection may be received from a user.

[029] The features described above with respect to the first approach apply equally to the second approach and therefore, for the sake of conciseness, are not repeated.

[030] In a third approach of the present techniques, there is provided a user device for automated adaptation of a first animation system to animate a target avatar from a second animation system, the user device comprising: at least one processor coupled to memory, for individually or collectively carrying out the method of automatically adapting the first animation system to the second animation system as described above. The user device may also be used to generate the target avatar after having been adapted as described above.

[031] The user device may be a smart device. The user device may be a smartphone. A smartphone is an example of a smart device. The user device may be a smart appliance. A smart appliance is another example of a smart device. An example of a smart appliance is a smart television (TV), a smart fridge, a smart oven, a smart vacuum cleaner, a smart robotic device, a smart lawn mower, and so on. More generally, the user device may be a constrained-resource device, but which has the minimum hardware capabilities to adapt and run a tracking system. The user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuum cleaner, smart lawn mower, smart oven, etc). It will be understood that this is a non-exhaustive and non-limiting list of example devices.

[032] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by at least one processor, causes the at least one processor individually or collectively to carry out any of the methods described herein.

[033] In the cases where the present techniques are implemented or executed on a device comprising multiple processors, the present techniques may be implemented by one or more of the multiple processors. That is, the present techniques may be implemented by or executed by the processors individually or collectively.

[034] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[035] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[036] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[037] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[038] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[039] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[040] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method. Brief description of the drawings

[041] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[042] Figure 1 is a flowchart showing the steps for generating a mapping between animation systems according to the present techniques;

[043] Figure 2 is a set of source meshes for a first animation system;

[044] Figure 3 is a set of target meshes for a second animation system;

[045] Figures 4 and 5 show respectively a source mesh and a target mesh together with the location of landmarks;

[046] Figure 6 is a flowchart showing the steps which may be used in the method of Figure 1 for selecting a subset of source meshes using constraints;

[047] Figure 7 is a flowchart showing the steps which may be used in the method of Figure 1 for defining the cost function as a quadratic programming (QP) problem;

[048] Figure 8 is a flowchart showing the generation of an animation using the mapping generated by the present techniques and

[049] Figure 9a is an example user image;

[050] Figures 9b and 9c show the neutral expression and three example meshes for the first and second tracking systems respectively;

[051] Figures 9d and 9e show output avatars from the first (source) and second (target) tracking systems respectively;

[052] Figure 10 is a flowchart showing the steps which may be used in the method of Figure 1 for defining the cost function as an alternative quadratic programming (QP) problem; and

[053] Figure 11 is a schematic drawing of a system for implementing the techniques described above. Detailed description of the drawings

[054] Broadly speaking, the present techniques generally relate to a method for determining a set of weights to be applied to transfer a facial expression from a first avatar which is generated by a first animation system to a second, different avatar which is typically generated by a second, different animation system. The animation systems may be termed tracking or capture systems because they track locations on a user’s face and use these locations to map a facial expression to an avatar’s face. Each animation system uses blend shapes to generate the facial expression on an avatar. Thus, the set of weights may be used to map between the two different sets of blend shapes. Advantageously, the present techniques enable animation of a third-party avatar which is not part of the first animation system without needing to change a source file for the third-party avatar.

[055] In Figure 1, a first set of meshes which are source meshes (also termed source blend shapes and the terms may be used interchangeably) are obtained at step S100. These blend shapes are used by an animation system to generate an animated avatar which transitions over time between multiple expressions particularly facial expressions. An avatar is a virtual character and may have a normal human face, a stylised human face with exaggerated features or represent a different character, e.g. an animal. Examples of the animation systems which may be used include Meta™, AR Emoji™, FLAME™ and ARKit™. The animation system which uses the source meshes may be termed a source animation system or a first animation system.

[056] As shown in Figure 2, each source blend shape 20 is a parametrised linear model of a facial expression and may comprise a mesh having a plurality of vertices and which are grouped as a plurality of shapes, e.g. triangles. Optionally, each blend shape 20 may have one or more holes, e.g. for open eyes 22 or an open mouth 24.

[057] Returning to Figure 1, we also obtain a step S110, a second set of meshes or target meshes (also termed target blend shapes and the terms may be used interchangeably). These target meshes are the meshes which are used in a second animation system for a similar purpose as the meshes in the first animation system. The second animation system may also be a different system to the first animation system which is selected from known systems such as Meta, AR Emoji, FLAME and ARKit.

[058] As shown in Figure 3, each target blend shape 30 is also a parametrised linear model of a facial expression and may comprise a mesh having a plurality of vertices and which are grouped as a plurality of shapes, e.g. triangles. Optionally, each blend shape 30 may have one or more holes, e.g. for open eyes 32 or an open mouth 34.

[059] For each of the source and target, there are a plurality of blend shapes each having different positions of the facial landmarks such as eyes, mouth and so. One of the blend shapes may represent a neutral or baseline expression. To generate an expression such crying or fury, a weighted combination of the blend shapes is used to transform the neutral expression into the desired expression. In general terms, each blend shape or mesh may be defined as: M = (7, F) where = V is the vertex list and = F is the face list (triangles in the examples shown in Figures 2 and 3). Each vertex is typically represented by a three-dimensional coordinate (x, y, z). The face collection defines implicitly the mesh topology. As shown in Figures 2 and 3, there may be a set of meshes {MJ with same topology and vertex count.

[060] . The vertices for each source or target blend shape may be represented as a vertex list, e.g. a matrix in ]R|lz|><3. Each mesh may also comprise a plurality of faces, e.g. triangles or other shapes, and the plurality of faces may be represented as a face list, e.g. a matrix in Mlflx3. When using a matrix, every entry of the vertex list may be represented as a row z;) for 0 <j <|7| - 1. Every entry of the face list may be a triple of indices for the rows of the vertex list (toi, t1:i, t2 i~) for 0 <i <|F| - 1. So for example the vertices of the i - th triangle face are given by rXy . c0,l yt0,t Zt . c0,l xt • rl,l yt^t Zt . ri,i xt . . r2-1 yt2,t Zt . ^2,1 This allows the representation of triangular meshes. Two manifold meshes may also be used. In general, two manifold meshes can be represented in more complicated formats (half-edge, winged edge, etc...) but they can all be transformed into each other with known algorithms.

[061] In the proposed implementation, the vertex list may be flattened into a vector V e ]R3|lz| and when using a vector, every entry of the vertex list may be of the form: = [xo>yo>zo>xi>yi>zi>■■■ ■ So when extracting vertices of a specific face using the vector representation, 3tji rather than just tj i can be used. In other words, the stride is extracted. This flattened representation is useful because the argument of the ||-||j or || - Hi which are used in the optimisation below may be solved using dot products rather than tensor notations.

[062] These meshes can be used in an expression of the form: V(M)(a) = V(M0) + 2" - V(M0». where V(M0) represents the vertex list of the neutral expression which is to be transformed to the vertex list of the target expression V(M)(<5), V(Mt) is the vertex list of each blend shape and at is the blending weight to be applied to each blend shape. This is described for example, in “Practice and Theory of Blendshape Facial Models” by Lewis et al published in Euro Graphics - STARs in 2014. Both the target and source blend shapes have a different set of blending weights which are used to transform the neutral expression to the desired expression. These weights may also vary over time when the avatar animation is a video rather than a single image.

[063] When using the representation of the plurality of vertices as a vector, the expression above may be alternatively written as: V(M)(a) = V(M0) + ....MN) • a Here is a matrix of size 3|y| xN and contains for every column the delta expression for the i-th expression. Each distance value or delta expression may be termed a displacement because the value represents the displacement of each vertex in a (sparse) target / source mesh from the corresponding vertex in the baseline (sparse) target / source mesh.

[064] As explained in more detail below, the aim of the present techniques is to generate a mapping to allow automation of the remapping of the different sets of weights. In other words, the aim is to find a matrix having weight values (for example between 0 and 1) which minimize a displacement distance so that when the weight matrix is applied to the target blend shapes, the corresponding set of source blend shapes is output. The weights which are applied to the source blend shapes to create the desired expression can then be applied. The automation can be integrated onto the user device, even if this is a resource constrained device.

[065] Returning to Figure 1, pre-processing may optionally be applied to both the sets of source and target blend shapes. The pre-processing may include obtaining a set of landmarks for each of the source and target blend shapes as shown at step S102 and S112. Figures 4 and 5 illustrate example source landmarks 40 and example target landmarks 50. The landmarks may be a set of mesh vertex indices. Although the term landmark is used, the indices may also be feature points.

[066] At step S104, the set of source meshes is processed using the source landmarks to generate the set of sparse source matrices (also termed sparse source blend shapes). Similarly, at step S114, the set of target meshes is processed using the target landmarks to generate the set of sparse target matrices (also termed sparse target blend shapes). By sparse blend shape, it is meant that the sparse blend shape contains less information than the original blend shape. Typically, this means that topology information is discarded and / or only a subset of the landmarks are retained. This process of obtaining and using landmarks is described for example in “Facial retargeting with automatic range of motion alignment” by Ribera et al published in ACM Transactions on Graphics in 2017.

[067] This may be expressed mathematically as: ABsrc(Lsrc) = preprocess(ABsrc)(Lsrc) ABtflt(Ltflt) = preprocess(ABtflt)(Ltflt) where ABsrc is a matrix representation of the source blend shapes {BSrcj}1<j<L constructed as ABsrc = [56^% ...,5bsrcw], Lsrc are the source landmarks, is a matrix representation of the target blend shapes [Bt3tj]1<j<L constructed as &Bt3t = [Sb13^, ...,6bt3tN], Lsrc are the target landmarks which are a set of mesh vertex indices. Each of the sparse source blend shapes and the sparse target blend shapes comprises the same number of mesh vertices so that there can be a one-to-one mapping between the vertices. There may be the same number N of the source and target blend shapes or there may be different numbers of source and target blend shapes. There needs to be at least a partial correspondence so that there is a subset of vertices for both source and target blend shapes so that there is a mapping (bijection) between the two subsets. It is noted that the mesh vertices in each of the sparse source and target meshes will vary over time when an animated avatar is being generated from a video image of a user.

[068] There may optionally be further pre-processing, for example to remove seams and / or to make the mesh watertight by removing any holes, e.g. for the eyes or mouth.

[069] Once the pre-processing is completed, the next step S120 is to determine a correlation between the sparse source blend shapes and the sparse target blend shapes. For example, the Pearson Correlation C(i,j) may be used and so the correlation between the ith source blend shape and the jth target blend shape may be defined as the cosine distance: {88^,88^} where 8btgtj is a target blend shape and 8bsrct is a source blend shape. The total number of target blend shapes is NT and the total number of source blend shapes is Ns. Notice that the correlation is topology independent. In this case, there are partial correspondences between the vertices for each of the target and source blend shapes and the correlation can be calculated using the sparse blend shapes. An alternative way of expressing the correlation is defined in the table below where V is a vector, BS? is the ith sparse target mesh and BS? is the jth sparse source mesh: corr(v(BSj), V(BSf) ) corr(v(BS}), V(BS^) ) corr(v(BS}), V(BS^) ) corr(v(BSj), V(BSf) ) corr(7(BS2), V(BS2s) ) corr(v(BS2), V(BS^) ) corr(v(BSST), V(BS2s) ) corr(v(BS£T),V(Ms))

[070] The next step S122 is to select a subset of the sparse source blend shapes using the determined correlation. As explained in more detail in Figure 6 below this is done using a set of constraints. When a sparse source blend shape is highly correlated with a sparse target blend shape, it is desirable to keep the highly correlated sparse source blend shape. Similarly, when a sparse source blend shape is uncorrelated with a sparse target blend shape, it is desirable to reject the uncorrelated sparse source blend shape. The source blend shapes which are neither highly correlated nor uncorrelated are retained but a mapping weight is learnt to optimise the mapping from the source to the target tracking system.

[071] The next step S124 is to construct a cost function for the mapping weights as a quadratic programming problem. This is explained in more detail in Figure 7 below. The QP problem is solved for each sparse source blend shape which has been selected based on the correlation at step S126. This can be done using any state-of-the-art solver, such as those described in the textbook “Numerical Optimization”, by Nocedal et al published as second edition, “OSQP: an operator splitting solver for quadratic programs” by Stellate et al published in Springer Nature 2020 and “NASOQ: numerically accurate sparsity-oriented QP solver” by Cheshmi et al published in ACM Transactions on Graphics in 2020. Once the weights have been obtained, they can be output at step S128.

[072] The mapping weights may be expressed as a matrix Wsource^target : There are Ns source meshes (or blendshapes) and NT target meshes (blendshapes) and the mapping is a linear transformation which transforms the blending weights of the source or first animation system (asource) into the blending weights of the target or second animation system (atarget). This may be expressed as: ^source^target^source ^target The linear mapping is optimised to minimise the difference between the error between the desired expression on the source avatar and the corresponding facial expression on the target avatar. In other words, Wsource^target may be selected by choosing the values of the weights so that the following expression is minimised: 1 t t - src - 2 o ||AB ^tgt^^source-^target^source dB (^src)^source ||

[073] This minimization however makes Wsource^target implicitly a function of asource which is undesirable. Instead, by observing the following bounds, an independent problem can be defined. The bounds are: ||A5 ^igt^^source^target ^source ^source ||2 source->target source-^target (Lsrc) ) ’ ^source -ABs-(Lsre))||j|asouree||i So instead of solving the asource dependent problem the independent problem can be defined as || - ABsrcasrc^ ||^ Another way of expressing the mapping may be source^>target [^source^target.l ’ ■■■ >^source^>target,Nr\ where wsource^targeti is a vector of weights to reproduce the i-th source blendshape 8bfrc by appropriate weighted combination of all the target blend shapes ABtflt and Nt is the number of target blend shapes. In other words, the mapping weights map the target blend shapes to an individual source blend shape by: = fihsrc lad ujsource -^target,i uui where = [8bl9t, ...,6bn9t] is the matrix of target blend shapes. Once we have the weights for each i-th source blendshape 6bfrc, we can calculate the full mapping as AR^g^T / / _ Apsrc sources target In other words, we can calculate the full matrix mapping from the target blend shape matrix ^Btgt to the source blend shape matrix ABsrc = [5bfc,..., 5b^rc]. In each of the expressions above, the delta term is used to indicate displacement of a location of a vertex in the expression from a location of the corresponding vertex in the neutral expression.

[074] Figure 6 shows the detail of selecting a subset of the sparse source blend shapes using the determined correlation. The first step S120 corresponds to the step in Figure 1 of determining the correlation between each of the sparse source and target meshes. An upper bound (or upper correlation threshold) and a lower bound (or lower correlation threshold) is defined. The constraints calculation may then be defined by: ( 0 c / = J c / ’1 U {(j, 1)} I I L (c / -1 U {( / ,0)} J = -1 ^(Xj) — ^correlation correlation where Hcorreiation is the upper correlation threshold, Lcorrelation is the lower correlation threshold, 0 is the value of the weight to be optimised, C(i, j) is the correlation value between the ith sparse source blend shape and the jth sparse target blend shape. Merely as an example, the upper bound may be 0.9515625 and the lower bound may be 0.06, in other words, the range may be [0.6, 0.9515625], In this example, the upper bound is calculated as (1+a) / 2 and the lower bound is calculated as (1+b) / 2 which avoids any negative values in the range. However, this is merely an example and other ways of defining the bounds may be used.

[075] Thus, as shown in Figure 6, at step S602, there is a check to see if the correlation value is below the lower bound. If the correlation value is below the lower bound, the ith source blend shape and the jth target blend shape are not at all correlated and the ith source blend shape is disregarded by setting the weight to 0 as shown at step S604. If the correlation value is above the lower bound, there is then a check at step S606 to see if the correlation is above or equal to the upper bound. If the correlation value is below the upper bound, the variable weight 0 is set at step S608 because the weighting for this sparse blend shape needs to be determined. If the correlation is above the upper bound, there is a high correlation and thus the weight is set to 1 as shown at step S610.

[076] Once the constraints are determined, the constraint values can be used to solve the QP problem at step S626 (in line with step S126 of Figure 1). The weights for the mapping can then be output at step S628 (in line with step S128 of Figure 1).

[077] One example for setting up of the cost function calculation is detailed in Figure 7. As noted above, we are trying to construct a weight matrix which may also be termed a remapping table Wsrc^tgt as a constrained QP problem so that we can calculate the optimized weights. Thus, a first step S700 is to define the cost function, for example as: 1 2 min-\\AWsrc^tgt — ^src^tgt — 1 9 (_^src->tgt ) 0 where Wsrc^tgt is the weight matrix for example having individual values between 0 and 1, A is the matrix of sparse target blend shapes which have been selected using the constraints above, Y is the matrix of sparse source blend shapes which have been selected using the constraints above, and g is an appropriately selected function.

[078] The cost function above can be re-written as: f(Wsrc^tgt) = 2 Siti ^(^src^tgt.i)- where as described above o)Source^target,i is a vector of mapping weights to reproduce the i-th source blendshape using target blend shapes and there are N target blend shapes. An appropriate choice of g means that the cost function can be reformulated as sum of separable problems at step S702. 1 2 min~ ||Arnsrc_>tgt,i Yj||p 0 <<1 , i 1,..., Nsrc giCmsrc^tgtj) 0 where gj are constraint functions.

[079] Each of the separable problems above is a quadratic cost function which can be expressed as: 1 / (%) = ~xTQx + bTx + c where Q is a quadratic term and b is a linear term and c is a constant. The quadratic cost function is generated for each subproblem and thus at step S704 a linear term is constructed for each selected source mesh. The linear term may be expressed as: for 1 <i <Nsrc bj = [ABtgt(Ltgt)]T5b?rc(Lsrc) where Nsrc is the number of selected source meshes, bj is the linear term for each selected source mesh, [ABtgt(Ltgt)]r is the transpose of the delta target blend shape matrix constructed as ABtflt(Ltgt) = ...,56^(1 / ^)] and 8b?rc(Lsrc) is the ith selected sparse source blend shape. The quadratic term may be calculated at step S706, for example from Q = [^Bt3t(Lt3t)]T^Bt3t(Lt3t) where [ABtgt(Ltgt)]r and ABtflt(Ltgt) are defined above.

[080] Each subproblem can be solved at step S708 using off the shelf quadratic programming solver (for example using active sets defined in the textbook “Numerical Optimization”, by Nocedal et al published as second edition). The QP problem is defined by the linear term, the quadratic term and the constraint values. At step S728, once the QP problem has been solved, the complete set of weights Wsrc^tgt can be output.

[081] Figure 8 is a flowchart showing how the remapping table can be used to generate an output using a first tracking system which corresponds to an output from a second tracking system and Figures 9a to 9e give examples of what is generated at the various stages. At step S800, a user image is received, for example as shown in Figure 9a. The user image may be a single image or a video of a user in which the user’s expression may be changing over time. At the same time, or in a subsequent step, a user input selecting an avatar is received at step S810. The user selection is of an avatar which is normally generated by a second (target) tracking system, e.g. Meta™. At step S802, the input user image is processing by the first tracking system, e.g. the AR emoji system, to generate a first mesh which represents a user’s expression in the input user image. The first mesh is generated by the first tracking system in the normal way by combining the blend shapes (source meshes) used by the first tracking system. Merely as examples, Figure 9b shows the neutral expression 900 together with three of the Ns source meshes 902a, 902b, ..., 902Ns for the first tracking system. Figure 9c the neutral expression 910 together with three of the Nt target meshes 912a, 912b, ..., 912Ntfor the second tracking system.

[082] At step S804, the first mesh is converted to a second mesh using the remapping weights which have been calculated as explained above. The second mesh is an approximation to the mesh which would have been generated by the second tracking system in the normal way by combining the blend shapes (target meshes) used by the second tracking system. At step S806, the generated second mesh may then be used to generate an animated avatar having the expression(s) from the input user image. This animated avatar is generated by the first tracking system but has the appearance of being generated by the second, different tracking system. Figures 9d and 9e compare the outputs from the first and second tracking systems and show example output meshes which represents the user’s expression in the input user image for both systems.

[083] The remapping table means that there is no need for a manual mapping of blend shapes from the first tracking system to the second tracking system. In effect, the output of the first tracking system is adjusted on to a new output mesh (and the underlying target meshes which are combined to generate the output mesh). Thus, meshes from the second tracking system can be used by the first tracking system without any manual configuration or adjustment to the tracking system. If an expression cannot be found on the target mesh, no expression may be performed or an approximation may be generated. The approximation may be generated by the use of constructive blend shapes. The first tracking system is thus made reusable for different avatars. In other words, the method described above aims to bypass the manual practice of pairing the tracked facial landmarks to blend shapes by attempting to generate a pairing between the blend shapes of different face meshes to a specific face mesh, so that the output of the tracking system can be adjusted for a wider range of face meshes, automatically.

[084] In the example detailed in Figure 7, the QP is solved for each individual source mesh but an alternative expression for the QP may be derived by considering the problem as a stochastic problem. For example, suppose we’re given a set of A? animations represented as Ns dimensional blendshapes weights we can think of these animation data as manifestations of a vector valued stochastic process. The remapping weight matrix which we wish to find is a linear transformation W e which minimises the error between the output from the first and second animation systems. The first step S1000 is to define the error e(t) as a stochastic process, for example as: e(t) = &Bt W(t) - &Bsa(t) = (&BtW - &Bs)a(t) where (OliU are the weights applied to each blend shape, ABT is the matrix of sparse target blend shapes (for ease of notation the (L) which is used above has been omitted), ABS is the matrix of sparse source blend shapes, and W is a matrix of remapping weights.

[085] Since a(t) is a stochastic process linearly transformed into the error then e(t) is also a stochastic process. A stochastic process is a well-known term of art and may for example be defined as a family of random variables, e.g. a^t). In this case, there is one random variable per time stamp (t). The statistics for each random variable is fully characterised by either the density px or the distribution Px. For simplicity sake, the first order statistics only may be used, for example the mean and covariance and these may be defined, for a random variable {Xt}teT, by: Mean : E[X] = f^NXdPx Covariance : x[x] = IE[(X - IE[X])(X - E[XDr]

[086] As such for every t e Rwe can calculate at step S1002 the covariance matrix S[e](t) for the error as: S[e](t) = ^BtW - ABs)S[a](t)(ABrMZ - &BS}T. The goal is to have an average error and variance which is as small as possible but still preserves semantic details. Accordingly, some variance is required. This can be modelled using the following energy function S[e](t)) = where p >0 and is a constant, ||• ||2 is the squared norm, e(t) is the error to be minimised, t is time and S[e](t) is the covariance for the error. We observe that for given t we can make f smaller by making smaller the mean error and the variance.

[087] If we assume that the time set is the bounded interval [0, T] (which would be the case in practice) then f e L2([0, T]) therefore we can calculate the following integral dt, The cost function may thus be expressed as at step S1004 as: pe(t)ll2+#l|S[e](t)ll2\2 \ 2 / e(t) = ^BtW - ABs)a(t) S[e](t) = (&BtW - ^Bs)X[a](t)^BTW - ABS) where all the terms have been defined above.

[088] Such a cost function is hard to minimize with respect to W but can be approximated using the following cost function at step S1006 which uses the mean and covariance of the error and the weights W) = X----r---~dt < E[e](t) = (&BTW - ABs)E[a](t) = (ABrW - ABs)^[a](t) V I <W <u where E[e] is the mean of the error, E[a] is the mean of the blending weights, S[a] is the covariance of the blending weights and S [e] is the covariance of the weights. ABT is the sparse target displacement matrix (for ease of notation the (L) which is used above has been omitted), ABS is the sparse source displacement matrix, and W is a matrix of remapping weights, ||-||2 is the squared norm and ||-||2 is the Frobenius norm. Observe that this is not a quadratic program in the standard form, this is because U is not in the standard form: 1 —xtQx + bT x + c where x is the unknown vector, b is a known given vector and there is also an integral in the middle. Furthermore, W e ]Rwtxns (the unknown weights) is a matrix and not a vector. Using known properties including vectorisation of the product of two or three matrices and properties of the kronecker product, the function U can be rewritten as a quadratic function in vec(VF) [ / (VF) = |vec(V / )T<2vec(VF) + vec(W)Tb + c where vec(V / )T is the transpose of vec(VF) which is the vectorisation of the full remapping weight matrix, Q is a quadratic term and b is a linear term and c is a constant. The vectorization of a matrix is a linear transformation which converts a matrix into a vector. Typically, this is done by stacking the columns of the matrix on top of one another so that: T vec(W) = where vtj is the element in the i-th row and j-th column of matrix W.

[089] The quadratic term and linear terms can be found by expanding at step S1008 the average error norm ||IE[e](f) H2 and the covariance error norm and linear terms and the constant may be expressed as: 2 ||[e](t) || . So the quadratic 1 r Q = — I (E[a](t)E[a](t)T + JuS[a](t))dt ®AB£ABt 1 fT —vec^Bg) — E[a](t)E[a](t)T + pX[a](t)dt ®AB> * Jn T||ABsE[a](t)||2+M||ABsVz^ — dt The constant c does not change and thus does not contribute to the minimum of the function U and if a following matrix operator A^ [a] is defined, the QP can be characterised by: 1 r A^[a] = — I (E[a](t)E[a](t)T + pX[a](t)^dt Q = Afi[a]®AB^ABr b = —vec(ABs)T(AM[a]®ABT) where E[a] is the mean of the blending weights, S[a] is the covariance of the blending weights, &BT is the sparse target displacement matrix (for ease of notation the (L) which is used above has been omitted), &BS is the sparse source displacement matrix, and W is a matrix of remapping weights and where p. >0 and is a constant.

[090] The quadratic problem can be solved at step S1010 using off the shelf quadratic programming solver (for example using active sets defined in the textbook “Numerical Optimization”, by Nocedal et al published as second edition). The QP problem is defined by the linear term, the quadratic term and the constraint values. The constraint values may be the same as defined above. In this example, to calculate Q and b we discretize using rectangle rule, entry wise since is an integral of a matrix [ f(t)dt « V f(k8T)8T Jo At step S1028, once the QP problem has been solved, the complete set of weights Wsrc^tgt can be output.

[091] Figure 11 is an example system for implementing the methods described above. The system comprises a user device 1150. The user device comprises: at least one processor 1152 coupled to a memory 1154. The at least one processor 1152 may comprise one or more of: a microprocessor, a microcontroller, and an integrated circuit. The at least one processor 1152 may include one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs). The memory 1154 may comprise volatile memory, such as random access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example. The user device may also have an image capture device 1156, e.g. a camera, for capturing an image, e.g. a single static image or a video image of a user which is to be transformed into an avatar. There is also an input / output interface 1158 for receiving data, e.g. from a separate electronic device 400.

[092] The user device may be a user device which is capable of generating an animated avatar. For example, the user device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuum cleaner, smart oven, or smart lawn mower). It will be understood that this is a non-exhaustive and nonlimiting list of example devices.

[093] The user device 1150 comprises an animation system 1160, e.g. AR emoji or similar app. The user device 1150 also stores a remapping table 1162 to allow the animation system 1160 to generate a third-party avatar using the blend shapes from the animation system 1160. The remapping table may be generated on the user device as described above. The at least one processor may be configured to operate individually or collectively, e.g. to generate an animation using the animation system 1160 and the remapping table 1162 and / or to generate the remapping table 1162 as described above.

[094] The system may also comprise a separate electronic device 1100 which also comprises: at least one processor 1102 coupled to a memory 1104, as well as other standard components such as an I / O interface 1108. It will be appreciated that for the electronic device 1100 and the user device 1150, there may be other standard components which are omitted for clarity. The separate electronic device may be a server or may comprise multiple interconnected devices. The electronic device 1100 may be used to generate the remapping table which is used by the user device 1150. For example, the user device 1150 may connect to the electronic device 1100 after a user has selected a third-party avatar which is not part of the animation system 1160 on the user device 1150 and the remapping table 1162 may be calculated by the electronic device 1100 in real-time and sent to the user device 1150 for use.

[095] References: • “Practice and Theory of Blendshape Facial Models” by Lewis et al published in Euro Graphics - STARs in 2014. • “Facial retargeting with automatic range of motion alignment” by Ribera et al published in ACM Transactions on Graphics in 2017. • The textbook “Numerical Optimization”, by Nocedal et al published as second edition, • “OSQP: an operator splitting solver for quadratic programs” by Stellate et al published in Springer Nature 2020 and • “NASOQ: numerically accurate sparsity-oriented QP solver” by Cheshmi et al published in ACM Transactions on Graphics in 2020.

[096] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

1. A computer-implemented method for automated adaptation of a first animation system to animate a target avatar of a second animation system, the method comprising:obtaining, from the first animation system, a set of source meshes, wherein each source mesh represents a facial expression on a source avatar in the first animation system and each source mesh comprises a plurality of vertices;obtaining, from the second animation system, a set of target meshes, wherein each target mesh represents a facial expression on the target avatar from the second animation system and each target mesh comprises a plurality of vertices;generating, from the set of source meshes, a set of sparse source meshes, wherein each sparse source mesh comprises a subset of the plurality of vertices within a corresponding source mesh;generating, from the set of target meshes, a set of sparse target meshes wherein each sparse target mesh comprises a subset of the plurality of vertices within a corresponding target mesh;defining a mapping to map a desired facial expression on the source avatar to a corresponding facial expression on the target avatar, wherein the desired expression on the source avatar is formed using a combination of source meshes from the set of source meshes;constructing, using the set of sparse target meshes and the set of sparse source meshes, a cost function which optimises the mapping to minimise error between the desired expression on the source avatar and the corresponding facial expression on the target avatar;determining, using the cost function, an optimised mapping; andusing the optimised mapping to adapt the first animation system to animate the target avatar.

2. The method of claim 1, wherein constructing the cost function comprises constructing a quadratic programming problem, QPP, which comprises at least one quadratic term and at least one linear term and determining, using the cost function, an optimised mapping comprises solving the QPP.

3. The method of claim 2, further comprising:pairing each of the sparse source meshes with at least one of the sparse target meshes; anddefining the mapping as a set of mapping weights comprising a mapping weight for each pair of sparse source and target meshes.calculating, using the subsets of vertices of the sparse source and target meshes, a correlation value for each pair of sparse source and target meshes.

4. The method of claim 3, further comprising:calculating, using the subsets of vertices of the sparse source and target meshes, a correlation value for each pair of sparse source and target meshes;defining, using the calculated correlation values for each pair of sparse source and target meshes, a set of constraints for the cost function; andapplying the set of constraints when solving the quadratic programming problem.

5. The method of claim 4, comprising:comparing the calculated correlation values for each pair of sparse source and target meshes to an upper correlation threshold; andfor each pair of sparse source and target meshes having a calculated correlation value greater than or equal to the upper correlation threshold, defining the constraint for the pair as applying a mapping weight having a fixed value when determining the optimised mapping.

6. The method of claim 5, comprisingcomparing the calculated correlation values for each pair of sparse source and target meshes to a lower correlation threshold; andfor each pair of sparse source and target meshes having a calculated correlation value above a lower correlation threshold and below the upper correlation threshold defining the constraint for the pair as applying a variable mapping weight which has a value which is determined when determining the optimised mapping.

7. The method of claim 6, wherein for each pair of sparse source and target mesheshaving a correlation value equal to or below the lower correlation threshold, defining the constraint for the pair as applying a mapping weight of zero when determining the optimised mapping.

8. The method of any one of the preceding claims wherein:the set of source meshes comprises a baseline source mesh;the set of target meshes comprises a baseline target mesh; anddefining a mapping comprises:defining a source displacement matrix which comprises a source displacement vector for each source mesh in the set of source meshes, wherein each sourcedisplacement vector comprises displacement distances between each of the vertices in the baseline source mesh and each of the corresponding vertices in one source mesh;defining a target displacement matrix which comprises a target displacement vector for each target mesh in the set of target meshes, wherein each target displacement vector comprises displacement distances between each of the vertices in the baseline target mesh and each of the corresponding vertices in one target mesh; anddefining a mapping between the source displacement matrix and the target displacement matrix.

9. The method of claim 8, further comprising:defining, for each source mesh, a specific mapping which maps a source displacement vector to the target displacement matrix; andconstructing the cost function by constructing, for each sparse source mesh, an individual quadratic programming problem, QPP, to optimise the corresponding specific mapping, wherein each individual QPP comprises at least one quadratic term and at least one linear term.

10. The method of claim 9, further comprising defining a quadratic term using a sparse target displacement matrix which comprises a target displacement vector for each sparse target mesh in the set of sparse target meshes, wherein each sparse target displacement vector comprises displacement distances between each of the vertices in the baseline target mesh and each of the corresponding vertices in a sparse target mesh.

11. The method of claim 10, comprising defining the quadratic term for each individual quadratic programming problem as:Q= [AB^L^rAB^L^)where ABtflt(Ltgt) is the sparse target displacement matrix and [ABtgt(Ltgt)]r is the transpose of the sparse target displacement matrix.

12. The method of claims 10 or 11, further comprising defining a different linear term for each source mesh using the sparse target displacement matrix and a sparse source displacement vector, wherein the sparse source displacement vector comprises displacement distances between each of the vertices in the baseline source mesh and each of the corresponding vertices in one sparse source mesh.

13. The method of claim 12, comprising defining each different linear term as bi = [ABtgt(Ltgt)]T5b?rc(Lsrc)where [ABtgt(Ltgt)]r is the transpose of the sparse target displacement matrix and 8b?rc(Lsrc) is the source displacement vector of the ith sparse source mesh.

14. The method of claim 8, further comprising defining a set of desired facial expression on the source avatar which are generated using a set of blending weights, wherein the set of blending weights are weights applied to blend the baseline source mesh and each of the source meshes to generate each of the desired facial expressions on the source avatar using the set of source meshes.

15. The method of claim 14, wherein constructing the cost function comprises constructing, using the set of blending weights and a quadratic programming problem, QPP which comprises at least one quadratic term and at least one linear term.

16. The method of claim 15 or claim 16, wherein constructing the cost function comprises calculating a mean for the set of blending weights and a covariance matrix for the set of blending weights.

17. The method of claim 16, further comprising defining the quadratic term of the quadratic programming problem using the sparse target displacement matrix and a matrix operator which is calculated from the mean and the covariance matrix of the blending weights.

18. The method of claim 16 or claim 17, further comprising defining the linear term of the quadratic programming problem using the sparse target displacement matrix, the sparse source displacement matrix and the matrix operator.

19. The method of any one of the preceding claims, wherein generating a set of sparse source meshes comprises:obtaining a set of source landmarks, wherein each source landmark has a location which is defined by a source vertex; andgenerating a sparse source mesh for each source mesh by:comparing the plurality of vertices within the source mesh with the locations of the source vertices of the set of source landmarks; andselecting the subset of the plurality of vertices which match the locations of the source vertices of the set of source landmarks, anddefining the sparse source mesh using the selected subset of the plurality of vertices.

20. The method of claim 19, wherein generating a set of sparse target meshes comprises: obtaining a set of target landmarks, wherein each target landmark has a location which is defined by a target vertex; andgenerating a sparse target mesh for each target mesh by:comparing the plurality of vertices within the target mesh with the locations of the target vertices of the set of target landmarks; andselecting the subset of the plurality of vertices within the target mesh which match the locations of the target vertices of the set of target landmarks, anddefining the sparse target mesh using the selected subset of the plurality of vertices.

21. The method of any one of the preceding, comprising generating a sparse target mesh for each target mesh and a sparse source mesh for each source mesh so that there is an equal number of vertices in each sparse source mesh and each sparse target mesh.

22. A computer-implemented method of generating a target avatar of a second animation system using a first animation system which has been adapted as set out in any one of the preceding claims, the method comprising:receiving an image of a user having a desired facial expression;receiving a selection of the target avatar;generating, using the first animation system, a source avatar having the facial desired expression; andgenerating the selected target avatar by applying the mapping to the generated source avatar.

23. A user device for automated adaptation of a first animation system to animate a target avatar from a second animation system, the user device comprising:at least one processor coupled to memory, for individually or collectively carrying out the method of any one of claims 1 to 20.

24. A user device for generating a target avatar of a second animation system using a first animation system, the user device comprising:at least one processor coupled to memory, for individually or collectively carrying out the method of claim 22.

25. A computer-readable storage medium comprising instructions which, when executed by at least one processor, causes the at least one processor individually or collectively to carry out the method of any one of claims 1 to 22.5

Images