Computer-implemented method of animation system, data processing device, animation system, computer program, or computer-readable medium

Abstract

A computer graphics animation system is provided to help prevent the generation of undesirable shapes by providing realistic examples of objects that are incorporated into interpolation functions that can be used to animate new shape transformations of the objects.

Description

[Technical field]

[0001] The present invention relates to image data processing / generation / animation, and more particularly, but not by way of limitation, the present invention relates to morph target animation. [Background technology]

[0002] The movement of a computer graphic character is controlled by an animator using a set of control parameters. Given these parameters, the animation software calculates the shape of the character. Since computer graphic characters are represented as a set of vertices forming a mesh, this problem is also called the mesh deformation problem. The main challenge of this problem is to create a technique that can animate with the control parameters while keeping the generated shape desirable for use.

[0003] In physically based deformation approaches, a physical simulation is used to generate the character's shape, which usually includes the underlying muscle or bone kinematics, where the mesh deformation with control parameters follows the laws of mechanics that restrict the character's deformation to a realistic shape suitable for use. Physics based simulation requires knowledge of mechanics as well as advanced numerical methods that are computationally expensive. Furthermore, many computer graphics characters are imaginary characters (aliens, dragons, etc.), and it takes a lot of trial and error to achieve a specific desired deformation with physical simulation.

[0004] Example-based methods create various character poses that are used as reference states in animation. While these example shapes can specify the desired character deformations, physical simulation is an indirect process; that is, any character shape deformation can be calculated from kinematics. Example-based models of linear blendshapes are widely used in animation (e.g., Lewis, JP, Anjyo, K., Rhee, T., Zhang, M., Pighin, FH, & Deng, Z. (2014). Practice and Theory of Blendshape Facial Models. Eurographics (State of the Art Reports), 1 (8)). Traditional approaches include morph target shapes that are linearly blended, but linear blending often does not capture the nonlinear interactions of various tissue types and anatomical structures, as well as nonlinear soft tissue deformations. To avoid this, algorithmic constraints on the magnitude of the weights are sometimes introduced using constraint algorithms. Summary of the Invention [Problem to be solved by the invention]

[0005] It is an object of the present invention to improve morph target animation, or at least provide the public or industry with a useful option. [Means for solving the problem]

[0006] According to one embodiment, an animation system for generating a weighted interpolation between two or more morph target shapes comprises: an input means for receiving a set of input weights for the morph target shapes to generate the weighted interpolation; a multivariate continuous interpolation function configured to reproduce the morph target shapes and the constraint shapes when corresponding weights are provided to the multivariate continuous function as input weights for the morph target shapes, given a set of shapes including morph target shapes and constraint shapes representing a weighted combination of the morph target shapes, when corresponding weights are provided to the multivariate continuous function as input weights for the morph target shapes; and a means for generating the weighted interpolation by using the input weights as arguments to the multivariate continuous interpolation function.

[0007] According to another embodiment, a method for generating a weighted interpolation between two or more morph target shapes relative to a base shape, each morph target shape including a plurality of vertex coordinates that are topologically consistent with the vertex coordinates of the morph target shapes, the method including: receiving a plurality of input constraint shapes including a plurality of vertex coordinates that are topologically consistent with the vertex coordinates of the morph target shapes, each of the input constraint shapes being associated with a non-zero weight for one or more of the morph target shapes; generating additional constraint shapes using the input constraint shapes for a plurality of new weights for the morph target shapes and associating the additional constraint shapes with each of the new weights; receiving interpolation weights for each of the two or more morph target shapes and using the base shape, input constraint shapes, and additional constraint shapes to generate an interpolation function for interpolating between the two or more morph target shapes; and generating vertex coordinates corresponding to the weighted interpolation between the two or more morph target shapes using the interpolation weights as arguments to the interpolation function.

[0008] According to another embodiment, a plurality of n morph target shapes B are generated for a base shape B0. 1··· B n a method for generating a weighted interpolation between the morph target shape B 1··· B n Each of the morph target shapes B k For the morph target shape B k The weight w applied to k receiving a set of weights W, including a set of weights W, and a set of multiple m constraint shapes C 1··· C m each constraint shape is a morph target shape B 1··· B n the steps of: associating a non-zero weight (association weight) for one or more morph target shapes (association shapes) among the morph target shapes; generating a continuous multivariate interpolation function configured to reproduce each morph target shape and each constraining shape when the association weight of the constraining shape for each morph target shape or association shape is provided as an argument to the interpolation function; and generating the weighted interpolation using the weight W applied to the morph target shapes as an argument to the interpolation function.

[0009] According to another embodiment, a method for estimating underlying component shape weights for a complex shape includes receiving one or more proposed component shapes and obtaining component shape weights by solving a least-squares problem to which penalties and solution bounds are applied to ensure that the weights associated with the proposed component shapes are non-zero.

[0010] A method for generating a weighted interpolation between two or more control shapes, at least one of the control shapes being a complex shape consisting of a weighted combination of a plurality of component shapes, the method comprising: mapping the control shapes to their constituent component shapes and associated weights for each of the constituent component shapes to form a set of weighted target component shapes; providing the set of weighted target component shapes to an interpolator; and using the interpolator to interpolate between the set of weighted target component shapes to generate the weighted interpolation between the two or more control shapes. [Brief description of the drawings]

[0011] [Figure 1] 1 shows a schematic diagram of an example-based model of a linear morph target shape. [Diagram 2] 13 shows a schematic diagram of how an example-based model of a linear morph target shape produces an undesirable shape. [Diagram 3] 13 shows a schematic example of applying a convex hull constraint to a morph target shape. [Figure 4] 1 illustrates a flow diagram of a method for pre-calculating a nonlinear interpolation function according to an embodiment. [Diagram 5] 1 shows a flow diagram of a method of interpolation according to an embodiment; [Figure 6] Here is an example of a 2D lattice: [Figure 7] A three-dimensional hypercube model is shown. [Figure 8] 13 shows a screenshot of a user interface for controlling morph target shape weights. [Figure 9] Shows the base shape. [Figure 10] Showing multiple morph target shapes. [Figure 11] Showing left and right morph target shapes for symmetrical facial expressions. [Figure 12] 1 shows a constrained shape that is a combination shape. [Figure 13] Indicates viseme. [Figure 14] 1 shows a constrained shape that is an incremental shape. [Figure 15] Combination--Incremental shapes are constrained shapes. [Figure 16] 1 shows a flow diagram of a method for estimating component shape weights. [Figure 17] The animation flow is shown below. [Figure 18] Shows the animation system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Morph Target Shape Interpolation In morph target animation, morph target shapes, also known as blend shapes, base shapes, blend shape targets, or morph targets, represent "deformed" versions of a base shape (or reference shape), which may be stored as a set of vertex coordinates (or positions). The deformation of the base shape with the morph target shapes may be controlled by an animator using a set of control parameters. Given these control parameters, a morph target animation model calculates the shape of the deformation. A key challenge in computer graphics and animation technology is to be able to animate the resulting shape under the control parameters while keeping it desirable for use.

[0013] A morph target animation model may contain several morph target shapes, each with a different geometric shape. The morph target shapes may be represented, for example, by a topologically consistent set of vertex coordinates (geometric positions, or points), which allows an animator to deform a base shape with one or more morph target shapes to create a new shape. In facial animation, a conventional morph target animation model / morph target shape interpolator generates a facial pose as a linear combination of multiple expressions, or morph target shapes ("targets"). Figure 10 shows four example morph target shapes representing Facial Action Coding System (FACS) shapes: "cheek raiser" A (AU06), "lip pucker" B (AU18), "Jaw opener" C (AU26), and "Lip suck" D (AU28).

[0014] Figure 1 shows a schematic diagram of a linear example-based morph target animation model. The linear morph target animation model of the human face shown contains three provided shapes, namely B0, B1, and B2. Using only these three examples, the morph target animation model can generate various blended expressions by linearly combining the given facial shape examples. The base shape B0 represents the neutral or rest state of the face. The morph target shape B1 represents the jaw-opening deformation of the face, and B2 represents the face deformed to open the eyes. Interpolation (also called blending or morphing) refers to interpolating between multiple shapes in some way.

[0015] FIG. 1 shows an interpolation intermediate between B0 and B1 as 0.5B1, which can be depicted as a half-open jaw (obtained by multiplying B1 by a factor of 0.5), and an interpolation intermediate between B0 and B2 as 0.5B2, which can be described as a half-open eye (obtained by multiplying B2 by a factor of 0.5).

[0016] Weights can be applied to the morph target shapes to create incremental shapes (shapes intermediate between the base shape and the morph target shapes, sometimes called "intermediate" shapes). The weights applied to the individual morph target shapes may be set on the user interface using a slider, as shown in FIG. 8. In this specification, a shape having a weight w: 0 <w <1 for a single morph target shape is called an incremental shape.

[0017] The morph target animation model may be used to animate combinations of multiple morph target shapes having a unit weight: w = 1. For example, B1 + B2 represents a combination of B1 and B2. Any shape that is a combination of two or more morph target shapes having a unit weight is called a combination shape.

[0018] A combination-incremental shape is a shape that is a combination of two or more morph target shapes, whereby at least one of the morph target shapes forming the combination is weighted (i.e., has a non-unit weight). In FIG. 1, 0.5B1 + 0.5B2 shows a combination-incremental shape that is a combination of B1 having a 50% weight and B2 having a 50% weight. Thus, the morph target animation model shown in FIG. 1 provides a theoretically infinite set of possible facial expressions by various linear combinations of B1 and B2. The linear morph target animation model may include any number of morph target shapes B k and the morph target shapes B k are linearly blended using Equation 1 below. Here, B0 represents the vertices of the shape in the static geometric shape (base shape), and ΔB k is a vector representing the displacement of the k-th morph target shape relative to the base shape (ΔB k = B k - B0), and w kis Basic Shape B k where f is the composite shape. Here, the term "linear" means that the composite shape f is k This means that it depends linearly on

[0019]

number

[0020] Morph target animation provides the animator with an intuitive understanding of how shapes can be augmented and combined. However, there are several problems with using a simple linear model such as that described above with reference to Figure 1.

[0021] First, shape interpolation can produce unusable movements if the object being modeled does not deform linearly. This poses particular problems when modeling biology. For example, when modeling human faces, linear methods cannot reproduce the nonlinear soft tissue deformations that occur in the real world. However, when facial expressions are modeled with a linear morph target animation model, by assuming a linear relationship between blend weights and the deformed facial geometry, the expressions are linearized, resulting in unrealistic animation.

[0022] Second, the simple linear model in Figure 1 cannot represent the nonlinear interactions of various tissue types and anatomical structures. Figure 2 shows an example where an animator wants to obtain facial expression D as a more realistic representation of the combination of B1 and B2. For the purposes of this illustration, assume the biology of the human face such that when the eyes and jaw are opened simultaneously, the eyes cannot be opened as much as when the jaw is not also opened, and when the eyes are opened together with the jaw, the jaw opens in a slightly different manner (represented as a square rather than an oval). Using a simple linear scheme would produce B1+B2, but since the user wants to obtain facial expression D, the result of such modeling is not suitable for use. In other words, the combination of morph target shapes is not necessarily the sum of the appearance changes of the individual morph target shapes, and in fact, some combinations may result in clearly different changes in appearance. When creating a morph target animation model based on FACS units as morph target shapes, a modeler may be able to tweak individual action unit (AU) shapes to make the combination of morph target shapes look more acceptable, but this may sacrifice the expressiveness of the individual AUs.

[0023] Multiple shape transforms may share the same vertices, and blending these shapes will result in overlapping vertex displacements. This results in unpredictable blended / interpolated shapes, so combining many shapes may result in shapes that are unusable. To avoid this, constraints on the magnitude of the weights may be introduced. Several methods of applying constraints on the base weights may be applied, for example applying Equation 2 as a constraint ensures that the morph target model is within the convex hull of the morph target.

[0024]

number

[0025] There are problems with methods where the weights are constrained, since the weights cannot be modeled independently without violating the constraints. Also, the above constraint formula only ensures that the resultant shape is within the convex hull, it does not guarantee that the deformation is realistic. In effect, this constraint prevents destruction by reducing the magnitude of the blended vertex displacements. For example, as shown in FIG. 3, applying a convex hull constraint to the foundation may mean that the combination of morph target shapes 601 and 602 may look like shown in 603, while the desired result is that shown in 604.

[0026] Another way to define weights that result in a suitable morph target shape to use is to solve a least-squares problem such as Equation 3 below, where α k are the new weights to be solved for, and μ is a regularization parameter. The drawback of this approach is that it is difficult to choose the parameter μ to obtain a suitable morph target shape for use. Also, in this approach, w k This is computationally expensive since the system needs to solve the minimization problem every time is updated.

[0027]

number

[0028] Another approach involves modifying the specified morph target shapes according to some heuristic algorithm to constrain the model to a composite morph target shape suitable for use. The result is a nonlinear morph target model having the form: k =h(w k ) is a function of the basis weights. The key feature here is the devising of the function h, which often requires extensive experimentation and expertise.

[0029]

number

[0030] In the above approach, algorithm-based constraints are introduced. In other words, a constraint algorithm is introduced in such a way that the basis weights are modified by applying the algorithm to the entire model parameters. However, in practice, this means that the blend weights cannot be modeled independently without violating the constraints.

[0031] For example, in one particular case, a user may specify w1=1 and w2=1 and expect a blendshape f=B0+ΔB1+ΔB2, but the animation system after applying the constraints may recalculate w2 such that w2=α≠1, f=B0+ΔB1+αΔB2, leading to undesired results on the part of the user (in other words, the generated shape f is not what the user expected given the weights).

[0032] This leads to technical problems including how to computationally combine morph target shapes in a realistic and predictable manner that takes into account the nonlinearity of some morph target shape interpolation and / or morph target shape combinations. Calculating the interpolation between many morph target shapes is computationally expensive, and methods such as Radial Basis Function (RBF) interpolation become prohibitively slow for a large number of morph target shapes (dimensions).

[0033] Detailed Description and Embodiments of the Invention The morph target animation model allows the morph target shapes to be augmented and combined in a realistic, consistent, and easy to control manner. Constraint shapes are provided to create an interpolation function that is used to interpolate between a base shape and multiple morph target shapes. Constraint shapes are realistic examples of the combination and / or augmentation of morph target shapes that are mapped to weights for the individual morph target shapes that they represent. Incorporating the constraint shapes into a non-linear interpolation function helps prevent undesirable shapes from being generated. This can be thought of as devising a multivariate continuous function that "goes through" the constraint shapes. This interpolation function can be used to generate any desired pose that is a combination of morph target shapes under user control by applying weights to the morph target shapes.

[0034] Additional constraint shapes representing the new weights may be automatically computed in a non-linear fashion using the morph target shapes and the input constraint shapes. The interpolated composite pose will be a combination of all relevant morph target shapes, taking into account the effects of real-world non-linearity via the constraint shapes (both the input constraint shapes and the additional constraint shapes). Although non-linearity is introduced in the pre-computation stage, the actual interpolation of the vertices is linear. This allows for fast real-time interpolation.

[0035] All shapes (morph target shapes and constraint shapes) can be described in the morph target animation model via an interpolation function. The interpolation function is configured to reproduce the morph target shapes and constraint shapes when the corresponding weights are provided. Given a set of all shapes S (including both morph target shapes and constraint shapes), an interpolation function f(W) is generated such that f(W)=S, where W is the set of all the given interpolation weights.

[0036] Rather than expanding the dimensionality of the morph target shape space (the number of morph target shapes in the model), this shape space is modeled as a combination of nonlinear contributions to the morph target shape model. That is, the number of morph target shapes required for the model can be reduced, for example, by incorporating a blend of units instead of requiring new units. This can be achieved by using a high dimensional model representation (HDMR), which utilizes a mathematical formula for nonlinear combinations. In other words, the shape space can be modeled as a combination of nonlinear contributions to the morph target animation model.

[0037] The multiple morph target shapes n may be considered as vertices of an n-dimensional cube that forms the representation or transformation space. Each dimension is associated with a morph target shape, and a base shape is at the origin of the representation space. The edges of this hypercube represent the weights for the corresponding morph target shapes. Figure 7 shows a three-dimensional cube. The origin B0 represents the base shape, and the three morph target shapes B i , B j , and B. k Connected to B jk , B ik , and B. ij is a combination shape, for example, B jk is B j and B k It represents a combination of B ijk There are three morph target shapes B i , B j , and B. k Shape x is a combination shape of B with unit weight (1). k and B with a weight of 0.5 i and B with a weight of 0.8 j It is a combination of.

[0038] The interpolation function is constructed on an n-dimensional cube and passes through a constraint shape at points on the n-dimensional cube. In one embodiment, the interpolation function may have the following form:

[0039]

number

[0040] where B0 is the base shape (the geometric shape at rest) and ΔB k is the kth morph target shape deformation relative to the base shape B0, and w k are the corresponding morph target shape weights to be interpolated. Thus, the first part of this equation is simply a linear combination of the morph target shape weights. ΔC i is the i-th constraint shape transformation for any low-dimensional shape considered in the nonlinear interpolation function, including the base shape. In other words, ΔC i is the difference between the i-th constraint shape and the shape generated by the previous interpolation layer. i is ΔC i are calculated such that the interpolation function satisfies the criterion f(W)=S.

[0041] Thus, the term shown in Equation 6 below sums the displacement contributions of all morph target shapes, and the term shown in Equation 7 below sums the displacement contributions of all constraining shapes, including incremental shapes based on a single morph target shape, combination shapes, and constraining shapes that are combination-incremental shapes.

[0042]

number

[0043]

number

[0044] To generate the interpolation function, we use the morph target shape B that corresponds to the object being modeled. 1...n and constraint shape C 1...n Given a set of constraint shapes C 1...n Each of the constraint shapes is associated with metadata corresponding to its constituent morph target shapes and weights for those morph target shapes that it represents.

[0045] For example, in FIG. 2, shape D is a constraint shape that is a combination shape corresponding to the weight combination w1=1, w2=1. FIG. 14 is a constraint shape that is an incremental shape corresponding to a weight of 0.5 for the morph target shape representing lip sac (AU28) shown in FIG. 10D. FIG. 15 is a constraint shape that is a combination-incremental shape corresponding to a weighted combination of morph target shape AU06 (cheek raiser) at 100%, morph target shape AU12 (lip corner puller) at 80%, and morph target shape AU26 (jaw opener) at 60%. Thus, when the shape of FIG. 15 is put into a nonlinear interpolation function, inputting W={100% AU06, 80% AU12, 60% AU26} will output an interpolated shape identical to the shape of FIG. 15. If instead an interpolation of W={90% AU06, 80% AU12, 60% AU26} (not provided as a constraining shape) were sought, the interpolation function would take into account the information provided by the associated constraining shapes, including the constraining shape of Figure 15, and output a new interpolated shape with a subtle cheek bump compared to the constraining shape of Figure 15.

[0046] In summary, a nonlinear interpolator is a method for finding the weights w given a nonlinear interpolation function f(W)=S for all shapes S in the system. 1··· w n For deformation ΔB k and ΔC i and Modifier Beta iFor example, we can generate a constraint shape C u is introduced, the constraint shape C in the nonlinear interpolation formula corresponds to the morph target shapes B1 and B2 with w1=1 and w2=1. u Modifier β applied to u But the constraint criterion is: B0+ΔB1+ΔB2+β u ΔC u =C u In this way, the desired composite shape can always be reproduced by the introduction of further constraint shapes.

[0047] To use a nonlinear interpolation function, a set of interpolation weights W, i.e., W for each morph target shape to be interpolated, is received as input to the nonlinear interpolation function (w 1···n ), the interpolated shape is output.

[0048] Weight w 1··· w n (0 <w k The set of interpolation weights (<1) may be pre-specified in some manner, for example an animator specifying interpolation weights for each morph target shape via sliders as shown in Figure 8. In other embodiments, the interpolation weights may be provided automatically via an animation system that animates keyframes, may be randomly generated, or may be specified in other ways.

[0049] The morph target shapes and / or constraint shapes may be created in any suitable manner, including from scanned data from real objects, engineered modifications, or physically simulated data, or a combination of methods.

[0050] The interpolation method may be applied to morph target shapes / blend shapes or deformable geometric control based on shape primitives. Morph target shapes may represent any deformable object, such as machines, bodies, body components, etc. Examples include, but are not limited to, human or animal bodies, faces, hands, limbs, or non-human characters, such as fantasy creatures or characters. Implementations of the present invention are described with reference to morph target shapes representing faces, particularly human faces.

[0051] In other embodiments, the method may be used to blend facial personalities or anatomical variations. Instead of blending expressions of the same subject, the system may be used to blend subjects with different personalities or anatomical variations between subjects. For example, an engineer may create a cat face, a dog face, and a dog-cat hybrid expression that the engineer considers as a combination shape, and an interpolation system may be used to create a weighted blend between the dog and cat faces using this and the constraint shapes provided by the engineer.

[0052] The morph target shapes may represent a face with a particular expression or subtle expressions. The human face can form an almost infinite number of expressions as a result of contracting one or more facial muscles to various degrees. Complex expressions or movements can be "decomposed" into basic movements such as those defined in the Facial Action Coding System (FACS) taxonomy. FACS classifies human facial movements according to the appearance of the face. Facial expressions can be coded using FACS by decomposing them into a set of specific action units (AUs). Thus, FACS provides a reliable and accurate method for describing facial changes due to specific muscle movements. Complex shapes such as facial expressions and visemes expressing specific emotional expressions may be described in terms of a set of weighted AUs. In other embodiments, the morph target shapes may represent visemes, which are generic facial images corresponding to specific sounds, which can be considered the visual equivalent of phonemes or units of sound in spoken language.

[0053] A shape can be a mesh or other point or set of points in a multi-dimensional space. For example, a shape may be defined as a set of vertices (points) that represent the state of a geometric shape. The set of points for each shape are topologically consistent with each other. Shape input may be represented as a matrix of vertices along with metadata such as the name and category of the shape (whether it is a morph target shape, a base shape, a constraint shape, a combination shape, etc.). In some embodiments, a suitable geometric shape definition file format may be used to input the shape vertices, such as a 3D model format OBJ (.OBJ) file format. A naming convention for naming the shape file may be used to provide metadata about the shape. One example of a possible naming convention is: Name1_weight1_Name2_weight2..., where weight i =w i is 0 <w i A number that is ≦1. i = 1, weights may be omitted. All w i The shape with .times. ...

[0054] Calculating a nonlinear interpolation function: 4 shows a flow diagram for pre-calculating a non-linear interpolation function according to one embodiment. At step 401, a shape is classified as a morph target shape, an incremental shape, or a combination shape.

[0055] Deltas are calculated for the morph target shapes and the constraint shapes in step 402. Instead of storing the absolute coordinates of the morph target shapes and the constraint shapes, the deltas store the displacement vectors relative to the base shape and any other shapes already considered in the nonlinear interpolation function.

[0056] In step 403, a lattice of points is defined / computed. The morph target shape model may be considered as placing the morph target shapes (targets) at (at least) some vertices of an n-dimensional hypercube, with the origin as the base shape and the hypercube edges representing the weights for the corresponding morph target shapes. Combination shapes are the lattice nodes placed at the vertices of the hypercube. Incremental shapes are the lattice nodes along the edges between the base shape (origin) and the morph target shape targets. Combination-incremental shapes are the lattice nodes at positions in the lattice corresponding to the weights for each constituent morph target shape. The input shapes do not necessarily form a complete lattice. Additional constraint shapes may be generated to complete or at least extend the lattice. Additional constraint shapes may be incremental shapes, combination shapes, and / or combination-incremental shapes.

[0057] FIG. 6 shows an example of a 2D lattice, where the input shapes (including morph target shapes and constraint shapes) are shown as dots and the unknown points to be generated are marked with cross marks. In other words, the additional constraint shapes to be generated are provided with weights as indicated by the cross marks. Any suitable method may be used to generate the additional constraint shapes, such as radial basis function interpolation (RBF), or another meshless interpolation method. Meshless in this context means that no regular mesh is required for the interpolation.

[0058] Partitioning into lower dimensional subspaces Since high-dimensional interpolation can be computationally expensive, a division into lower dimensional spaces may be performed to facilitate the interpolation. The division into lower dimensional spaces may facilitate the generation of additional constraint shapes. Instead of applying the interpolation between d-dimensional hypercubes, the hypercubes are divided into an array of grids. For example, each group of constraint shapes may be divided into its incremental dimension, which means the number of morph target shapes that are combined with it. For example, the incremental dimensions are as follows: the term represented in Equation 8 below has an incremental dimension d of 1; the term represented in Equation 9 below has an incremental dimension d of 2; the term represented in Equation 10 below has an incremental dimension d of 3, etc. Following this, each group according to its incremental dimension may be further divided into subgroups based on the number of Euclidean spaces of dimension d that can be formed. A complete lattice of each subgroup is created, and the function values ​​missing in the obtained grid are calculated using a suitable meshless interpolation method such as RBF. As a result, an array of grids is associated with each incremental dimension.

[0059]

number

[0060]

number

[0061]

number

[0062] Detailed Description of Generation of Interpolation Functions According to an Embodiment In short, the interpolation can be done by considering each shape type and improving the accuracy of the interpolation. An example is described in detail below.

[0063] Interpolation using morph target shapes only Any number of morph target shapes B k are linearly blended using the following formula:

[0064]

number

[0065] Here, B0 represents the vertex of the shape in the stationary geometric shape (base shape), and ΔB k is a vector that represents the displacement of the kth base shape relative to the base shape (ΔB k =B k -B0).

[0066] Interpolation using combination shapes The interpolation function may use an approximate Taylor series expansion or other suitable multivariate interpolation system, such as multivariate Lagrangian interpolation, to interpolate between combination shapes. Abstractly, a morph target shape can be viewed as the tangent space of the base shapes of an n-dimensional shape "ensemble" embedded in an m-dimensional surrounding space. i =w j The shape associated with =1 is B ij and 0 <w i <1, 0 <w j The shapes associated with the morph target shapes B are denoted as terms in Equation 12 below. The set of all shapes S is divided into three non-overlapping subsets, i , Combination Shape B ij···k , and an incremental shape expressed by the following Equation 13. Taking the base shape B0 as the origin or reference shape, an interpolation function f(w) is created such that f(W)=S, where W represents the set of all given interpolation weights.

[0067]

number

[0068]

number

[0069] Based on the Taylor expansion, the blendshape function f(w) in the neighborhood of the point (w=0) can be expanded as follows:

[0070]

number

[0071] Retaining certain terms in the expansion (2) produces approximations with different levels of accuracy. As a result, an interpolation can be done by considering each shape type and improving the interpolation accuracy. Given only the morph target shape, we have:

[0072]

number

[0073] The unit step size (i.e., Δw j = 1) finite differences are used.

[0074]

number

[0075] Combination Shape B ij···k Given,the mixed partial derivatives can be held in (2) with a finite difference approximation.

[0076]

number

[0077] When the higher order derivatives, Eq. (18) below, are zero, a linear (with respect to individual coordinates) model of the form Eq. (19) below is generated.

[0078]

number

[0079]

number

[0080] or in the following compact form:

[0081]

number

[0082] An additional term is present in (8) only if the corresponding combination shape is given. The coefficient ΔB is calculated by taking the interpolation points at their given values ​​(combination shape values) and constructing an interpolation function such that ijk can be obtained.

[0083]

number

[0084] The result is as follows:

[0085]

number

[0086] Interpolation using incremental shapes The interpolation function may use any suitable regular or irregular grid interpolation system to interpolate between the incremental shapes, including but not limited to tensor spline interpolation, trigonometric polynomial interpolation network, or thin plate spline interpolation. The interpolation technique that can be used here may depend on the completeness of the hypercube. If the lattice is a complete lattice, tensor spline interpolation may be preferred. If the lattice is an incomplete / partial lattice, trigonometric polynomial interpolation network or thin plate spline interpolation may be used. In another embodiment, cubic Hermite splines may be used to embed the incremental shapes. The parameters of the Hermite polynomials may be automatically estimated from nearby nodes, along with constraints to minimize the interpolation overshoot, eliminating the need to set up a nonlinear incremental network.

[0087] In the next example of interpolation using incremental shapes, tensor spline interpolation is used. The model (8) is quasi-linear. Considering the incremental shape adds nonlinearity. The incremental shape, expressed in Equation 23 below, is incorporated using a non-zero modifier term if a corresponding incremental shape exists.

[0088]

number

[0089]

number

[0090] Here, the function expressed by the following Equation 25 is devised as the following Equation 26.

[0091]

number

[0092]

number

[0093] The function h is interpolated between the points of the incremental shape, on top of the interpolation performed using only the join points, and tensor spline interpolation is used to implement the function h. The interpolation expression (11) takes the given shape values ​​exactly at the interpolation points, providing a nonlinear approximation to the points being interpolated. The function h is defined using tensor product splines in M-dimensional space. n=2 M vertices {Vm}, m = 1, , n and corresponding function values ​​{F m}, m = 1, . . . , n. For all points V m has two distinct values ​​for each dimension (v i a , v i b ), so for any point x in this hypercube, x i ∈[v i a , v i b ]. Next, x i ∈[v i a , v i b ], the line segment:

[0094]

number

[0095] Two values ​​of y a andy b And these values ​​are x j It is also a function of . Here, the function y a (x j ) and y b (x j ) in the interval [v j a , v j b ]x j is defined for

[0096]

number

[0097] As a result, an iterative sequence is obtained to calculate the function h that performs the interpolation in the hypercube. A set of grid data of the incremental shape expressed by the following Equation 29 is obtained, and the interpolation points (x i , x j , x k ), given the point (x i , x j , x k ) is found, and then interpolation is performed using (15) and (16).

[0098]

number

[0099] According to the described embodiment, the interpolation algorithm may include two stages.

[0100] Precomputation Stage Divide all shapes into three subsets: basic shapes, combination shapes, and incremental shapes. Calculate ΔB and ΔC from (10) and (14). Divide ΔC into groups based on the increment dimensions expressed by the following Equations 30 to 32. Each group of ΔC of incremental dimension d is divided into subgroups based on the number of Euclidean spaces of dimension d that can be formed. A complete lattice of each subgroup is constructed and the missing function values ​​in the obtained grid are calculated from the RBT method. As a result, the grid g ijk The array is associated with a group represented by the following Equation 33.

[0101]

number

[0102]

number

[0103]

number

[0104]

number

[0105] Interpolation execution stage The point to be interpolated w = (w1, w2, . . . , w N ), calculate f(w) from (8). All grids ijk Iteratively apply to the point (w i , w j , w k , , ) belongs to, and the function h ijk Calculate. The final result is the sum of f(w) from (8) and h ijk All values ​​of .

[0106] advantage Combining constraint shapes nonlinearly to perform computer graphic animation provides realistic constraint shapes of the object, thus preventing the generation of undesirable shapes. The interpolator improves blending of morph target shapes. Shapes that are combinations and / or weights of morph target shapes are input to the model as constraint shapes, and therefore can be reproduced. The constraint shapes are taken into account in the interpolation model via a nonlinear interpolation function. Thus, the interpolator works in a predictable and intuitive manner.

[0107] Because nonlinear deformations are automatically accounted for in the interpolator, the digital modeler does not need to create many examples as in the case of traditional example-based methods, reducing the time, effort, and resources required to develop realistic models. The interpolator is easier to use because the dimensionality of what the animator needs to work with is reduced. Constraint shapes are accounted for as nonlinear terms in the base morph target shapes, not as additional dimensions. For example, in a slider-based animation system, it is actually easier for an animator to control 60 sliders than 210+ sliders. Combination shapes and / or incrementals are no longer additional shapes that need to be manually controlled or specifically accounted for. Control is smoother and easier to use.

[0108] Since real-time interpolation is a linear process using an interpolator, fast real-time interpolation can be achieved even on machines with low specifications. The number of morph target shapes and the number of constrained components can affect the performance of the interpolator, but since the cost of real-time interpolation is fixed, it mainly affects the pre-calculation time.

[0109] Textures may be blended in a similar manner: rather than blending vertex positions, color and transparency channels may be blended (e.g. R,G,B,A). Similarly, examples of incremental and combination textures are created, where the channels are blended according to the interpolation methods described here.

[0110] Morph Target Shape Mapper Morph target shapes may be component shapes, also called "foundation shapes", in other words, shapes that represent unique basic movements from rest and may be considered as basic building blocks for creating further shapes. In other words, they are basic variations of shapes. FACS action units are a good example of component shapes or "foundation shapes" because complex facial expressions can be interpreted as a blend or combination of AUs. The facial animation setup allows animators to build complex facial expressions by blending AUs to generate natural variations. Visems may or may not be considered as component shapes ("foundation shapes"). Figure 13 shows examples of Visems including Visem "AI" A, Visem "I" B, Visem "O" C, Visem "TS" D, and Visem "V" E.

[0111] The mapper addresses the problem of figuring out which weighted combination of component shapes reflects a particular shape when the weighted combination is not known a priori, such as when a user creates the desired example shape in terms of a set of geometric points without using morph target shape interpolation methods. In other words, the mapper uses unknown weights (unsolved shapes) to calculate a combination of component shapes that estimates a given shape. For example, an animator may want to use a scanned 3D model of a person exhibiting a laughing expression as a morph target shape, but it may be difficult to manually map this expression to the corresponding component shapes and / or component shape weights. Furthermore, if the geometry of one of the component shapes is modified, the mapping results may become out of date. Thus, animator-mapped sequences are prone to errors in a constantly evolving development environment.

[0112] The mapper can map example shape vertices to component shape vertices to estimate these weights. In one embodiment, the mapper estimates the underlying component shapes and their respective weights of the unresolved shape using a least-squares problem with a penalty term that depends on the number of morph target shapes known to be part of the unresolved shape (or the number of specific shapes that will be included in the result).

[0113]

number

[0114] Solution bounds, 0 ≤ w i ≦1, where T is the target shape, f(w) is the shape combination model, and λi are the penalties. The mapper takes as input the vertices of the named shape dataset and the estimated unsolved shape.

[0115] 16 shows a flow diagram of one method for estimating values ​​of user-specified component shape weights corresponding to unresolved shapes. In step 1601, dimensionality reduction is performed by constructing a dimensional space and projecting all component shapes into this subspace. In step 1602, the dimensionality of the unresolved shapes is reduced by projecting them into the sub-dimensional space. In step 1603, a least squares solver is generated. The least squares solver may have the following form:

[0116]

number

[0117] where ΔTk and ΔF are the dimensionality-reduced basis shape and example shape displacements, respectively, and the coefficients λi reflect user-specified component shape weight penalties. In step 1604, component shape weights are generated by solving a least-squares problem.

[0118] Component Shape Suggestions Users can provide suggestions for component shapes that the mapper should map. To enable this, the proposed component λ i The penalty factor of w is reduced, which increases the likelihood that these proposed component shapes will be used during the mapping process. Furthermore, the solution bounds are such that 0≦w for the proposed component shapes. i will be changed to ≦1.

[0119] In another embodiment, both the component shapes and their associated weights may be proposed and implemented, in which case the exact values ​​of the weights are defined for the proposed component shapes in the least-squares solver.

[0120] Dimensionality reduction Since the shape mesh may contain a large number of vertices, computing the least squares problem may be inefficient and time consuming. Therefore, the dimensionality of each shape may be reduced before solving the least squares problem. The dimensionality reduction may be done by singular value decomposition and projection of the data into a low dimensional space with a basis of the leading singular vectors, or by other suitable methods. Numerical simulation experiments show that by taking six leading singular vectors, reliable weight estimates of the face model can be obtained, although other suitable numbers of leading singular vectors may be used.

[0121] A naming convention for shapes may be used to facilitate animation, for example, a double underscore "_ _" may be used to indicate unknown shape names from interpolated shapes. EXsmile__AU06_AU12_AU26_25_ is a smile shape with predefined combination nodes and known weights for AU26 (AU06 and AU12 are also proposed, but their weights need to be estimated). PhBut__ is the shape of the phoneme “but” for which combination nodes need to be estimated. The variants of PhBut__ can be shouted phonemes (PHbutshout) or silent phonemes (PHbutsilent), and each variant is mapped separately into the FACS space via a mapper.

[0122] Animation System An animation framework 14 may be provided to enable the use of complex shapes, including two or more combination shapes, as animation targets or control shapes. Complex shapes, such as complex expressions made up of multiple underlying component shapes, can be difficult and time-consuming to animate because an animator must set weights for several component shapes individually. It is desirable to be able to animate complex shapes with a single control. However, components of a complex shape may not deform linearly, so simply interpolating between the complex shape and the base shapes may not look realistic. Combinations of component shapes may also interpolate in a manner that is not equal to the sum of their parts. Another issue is how to enable the animation of characteristic shapes whose underlying component shapes are unknown and may be difficult to manually ascertain. For example, in the context of facial animation, some actors have unique and complex facial expressions that even experienced animators may have difficulty mapping to the underlying component movements (e.g., FACS AU).

[0123] The animation framework 14 maps complex shapes (complex morph target shapes) to their constituent component shapes to form a set of target component shapes. When the constituent component shapes and / or weights are unknown, the mapper figures out the underlying component shapes and their respective weights. The interpolator interpolates between the weighted underlying target component shapes.

[0124] Detailed Description and Embodiments The morph target shape animation framework 14 is designed to incorporate a non-linear morph target shape interpolation system and a morph target shape mapper into one system while adding flexibility to the user. End users can view, manipulate and create shapes as animations without breaking them down into component shapes (base shapes) and example shapes. Animated shapes can be controlled via animation weights and the animation system can map all weights to component shape weights and perform morph target shape interpolation.

[0125] Figure 17 shows a schematic flow diagram of the animation method. A sorter 11 receives a number of control shapes and processes each control shape. For example, sorter 11 receives shapes {AU12, AU26, AU12_AU26, Smile__AU12_AU26}. If the control shape is a complex shape whose underlying component shapes are unknown, it is input to mapper 12, which outputs the underlying component shapes and their respective weights. The result is then passed to animation framework 14, which maps the user-provided control weights to component shape weights. Interpolator 13 interpolates between the control shapes by interpolating between the underlying component shapes of each control shape.

[0126] Weights are provided for the control shapes that are mapped to the component shape weights via a matrix, and then the interpolator uses the component shape weights as input to output the interpolated shape. The control shapes may be component shapes, complex shapes, or a combination of component and complex shapes. The control shapes may contain metadata describing any component shapes that are known to be components of the control shape.

[0127] Some examples are as follows: ·AU12: Component Shapes ·AU26: Component Shape · AU12_AU26: Complex shape consisting of two component shapes: The component shapes present are already known and are clear from the naming convention. Smile__AU12__AU26.05: A control shape whose component is known to be AU12__AU26.05, so there is no need to pass this control shape to the mapper, it can be passed directly to the animation framework, which will pass the underlying component shape and weights to the interpolator. Laugh: a control shape whose components are unknown. This control shape can be passed to the mapper, which will calculate the constituent component shapes and their respective weights. · Happy__Laugh_Smile: A control shape made up of two other control shapes (it is a complex shape).

[0128] FIG. 18 shows a schematic diagram of animation using the animation framework 14. Animation control weights 1801 for control shapes are provided to the animation framework 14. The animation framework 14 provides a mapping between the control shapes 1803 and the underlying component shapes 1802. The animator may have the option to animate complex shapes either by providing weights for the component shapes (using several component shapes as control shapes) or by using corresponding predefined complex shapes as control shapes. As shown in FIG. 18, the user may specify a weight of 1 for each of AU01L and AU01R, the control shapes, or a weight of 1 for AU01 (which is internally mapped to AU01L and AU01R via the animation framework 14) to create an equivalent composite shape. The incremental shape AU01_50 is internally mapped to AU12L_50 and AU01R_50, and the interpolator interpolates the combination of AU12L_50 and AU01R_50. Thus, AU01 provides convenience to the animator as they only need to toggle one slider, but does not add computational complexity to the interpolator as AU01 is not added as another component shape / morph target shape. The component shapes are passed to the Interpolator 13, which interpolates between the component shapes using the mapped weights, and the result of the interpolation 1806 is used for the animation.

[0129] Complex Shape A complex shape is composed of multiple component shapes. For example, a complex shape may represent an emotion that is composed of multiple AUs, as shown in the following table:

[0130] [Table 1]

[0131] Component shapes are a subcategory of morph target shapes whose weights form basis vectors in the space of input values ​​of the interpolation function. Component shapes represent unique fundamental movements from rest. FACS AUs are a good example of component shapes because complex facial expressions can be interpreted as a blend or combination of AUs. The facial animation setup allows animators to build complex facial expressions by blending AUs to generate natural deformations. Visems can be considered as basic component shapes for speech, but they are themselves composed of AUs. Thus, we can see that the classification as component shapes or "basic shapes" depends on the degree of abstraction.

[0132] A symmetric shape may not be considered a component shape, but rather a complex shape, which is a combination of component shapes. FIG. 11 shows left and right morph target shapes for a symmetrical facial expression, "left-lip-corner-puller" A and "right-lip-corner-puller" B, corresponding to a symmetrical FACS AU12. A complex shape (AU12_ _AU12L_AU12R) may be used as a control shape for a lip corner pulling facial expression (smile). The animation framework 14 maps the control shape to its component shapes AU12L and AU01R, and the interpolator 13 interpolates the component shapes AU12L and AU01R to create the smile.

[0133] advantage The animation framework allows complex shapes to be animated realistically with a single control, taking into account the complexities of underlying component shape interactions and interpolation. Complex shapes such as visemes, emotions, and other facial expressions can be defined as animation controls, without increasing the dimensionality of the interpolation system. Animators can animate complex shapes even when the shape's underlying component shapes are not easily visible.

[0134] interpretation The described methods and systems may be utilized in any suitable electronic computing system. According to the embodiments described below, an electronic computing system utilizes the techniques of the present invention using various modules and engines.

[0135] An electronic computing system comprises at least one processor, one or more memory devices or interfaces for connecting to one or more memory devices, input and output interfaces for connecting to external devices to enable the system to receive and act on instructions from one or more users or one or more external systems, a data bus for internal and external communication between the various components, and a suitable power supply. Additionally, the electronic computing system may comprise one or more communication devices (wired or wireless) for communicating with external and internal devices, and one or more input / output devices such as a display, a pointing device, a keyboard or a printing device.

[0136] The processor is configured to execute program steps stored as program instructions in a memory device. The program instructions enable various methods of implementing the invention described herein to be performed. The program instructions may be developed or implemented using any suitable software programming language and toolkit, such as, for example, a C-based language and compiler. Furthermore, the program instructions may be stored in any suitable manner such that they can be transferred to a memory device or read by the processor, such as, for example, stored on a computer-readable medium. The computer-readable medium may be any suitable medium for tangibly storing program instructions, such as, for example, a solid-state memory, a magnetic tape, a compact disk (CD-ROM or CD-R / W), a memory card, a flash memory, an optical disk, a magnetic disk, or other suitable computer-readable medium.

[0137] The electronic computing system is configured to communicate with a data storage system or device (eg, an external data storage system or device) to retrieve relevant data.

[0138] It will be understood that the systems described herein comprise one or more elements arranged to perform the various functions and methods described herein. The embodiments described herein are intended to provide the reader with examples of how the various modules and / or engines constituting the elements of the system are interconnected to enable the functionality to be implemented. Furthermore, the described embodiments explain in system-related detail how the steps of the methods described herein may be performed. Conceptual diagrams are provided to show the reader how various data elements are processed at various stages by various different modules and / or engines.

[0139] It will be understood that the arrangement and structure of the modules or engines may be adapted as appropriate depending on the requirements of the system and the user, such that various functions may be performed by different modules or engines than those described herein, and that certain modules or engines may be combined into a single module or engine.

[0140] The described modules and / or engines may be implemented or provided with instructions using any suitable form of technology. For example, a module or engine may be implemented or created using any suitable software code written in any suitable language, and the code is compiled to generate an executable program that may be executed on any suitable computing system. Alternatively, or in conjunction with an executable program, a module or engine may be implemented using any suitable combination of hardware, firmware, and software. For example, some of the modules may be implemented using an application specific integrated circuit (ASIC), a system on a chip (SoC), a field programmable gate array (FPGA), or any other suitable adaptable or programmable processing device.

[0141] The methods described herein may be performed using a general-purpose computing system specially programmed to perform the steps described. Alternatively, the methods described herein may be performed using specific electronic computer systems, such as data classification and visualization computers, database query computers, graphical analysis computers, data analysis computers, manufacturing data analysis computers, business intelligence computers, artificial intelligence computer systems, etc., that are specially adapted to perform the steps described on specific data captured from an environment related to a particular field. [Explanation of symbols]

[0142] 1 shape 2 Morph Target Shapes 3. Base Shape 4. Constraint Shape 5. Incremental Shape 6 Combination Shapes 7 Combination - Incremental Shapes 8 Modifiers 9 Component Shapes 10 Complex Shapes 11 Sorter 12 Mapper 13 Interpolator 14 Animation Framework 15 Control Shapes 16 Unsolved Shapes 17 Input 18 Addition 19 Lattice 20 Morph Target Animation Models 21 Interpolation Functions

Claims

1. 1. A computer-implemented method of an animation system for generating weighted interpolation between two or more morph target shapes for performing computer graphics animation, comprising: receiving a set of input weights for a morph target shape to generate the weighted interpolation; using a continuous multivariate interpolation function configured to, given a set of shapes including morph target shapes and constraint shapes representing weighted combinations of the morph target shapes, reproduce the morph target shapes and the constraint shapes when corresponding weights are provided to the continuous multivariate interpolation function as input weights for the morph target shapes; at least one constraint shape is an incremental shape that corresponds to a partial weight of one or more of the morph target shapes; the continuous multivariate interpolation function represents a non-linear relationship between weights and morph target shapes; generating the weighted interpolant by using the input weights as arguments to the continuous multivariate interpolation function; A computer-implemented method comprising:

2. 1. A computer-implemented method of an animation system for generating weighted interpolations between two or more morph target shapes relative to a base shape for performing computer graphics animation, each morph target shape including a plurality of topologically consistent vertex coordinates; receiving a plurality of input constraint shapes including a plurality of vertex coordinates that are topologically consistent with vertex coordinates of the morph target shapes, each of the input constraint shapes being associated with a non-zero weight for one or more of the morph target shapes, at least one constraint shape being an incremental shape that corresponds to a partial weight for one or more of the morph target shapes; using the input constraint shapes to generate additional constraint shapes for a number of new weights for the morph target shapes, associating the additional constraint shapes with each of the new weights, the additional constraint shapes being generated with the new weights to complete or extend an n-dimensional cube, the morph target shapes being the dimensions of an n-dimensional cube with the base shape as the origin and the edges of the n-dimensional cube as the weights for the morph target shapes; receiving interpolation weights for each of the two or more morph target shapes; generating an interpolation function for interpolating between the two or more morph target shapes using the base shape, the input constraint shape, and the additional constraint shape, the interpolation function representing a non-linear relationship between weights and the morph target shapes; and generating vertex coordinates corresponding to the weighted interpolation between the two or more morph target shapes using interpolation weights as arguments to the interpolation function.

3. The method of claim 2 , wherein at least one input constraint shape is a combination shape corresponding to a combination of the two or more morph target shapes having unit weights.

4. The method of claim 2 , wherein the additional constraint shapes are generated using meshless interpolation.

5. The method of claim 4 , wherein the additional constraint shapes are generated using radial basis interpolation.

6. A method according to any one of claims 2 to 5, comprising dividing the n-dimensional cube into lower dimensional spaces and generating additional constraint shapes for each lower dimensional space.

7. Base Shape B for performing computer graphics animation 0 Multiple morph target shapes B 1・・・ B n 1. A computer-implemented method for an animation system that generates a weighted interpolation between The above morph target shape B 1・・・ B n Each morph target shape B k For the morph target shape B k The weight w applied to ｋ receiving a set of weights W, including: Multiple m constraint shapes C 1・・・ C m each of the constraint shapes receiving the morph target shape B 1・・・ B n and at least one constraint shape C 1・・・ C m one or more morph target shapes B 1・・・ B n a step, which is an incremental shape corresponding to the partial weights of generating a continuous multivariate interpolation function configured to reproduce each morph target shape and each constraint shape when association weights of the constraint shapes for each morph target shape or association shape are provided as arguments to the continuous multivariate interpolation function; generating the weighted interpolation using the weights W applied to the morph target shapes as arguments of the continuous multivariate interpolation function; A computer-implemented method, wherein the continuous multivariate interpolation function represents a non-linear relationship between weights and morph target shapes.

8. The method of claim 1 or 7, wherein the continuous multivariate interpolation function is a high-dimensional model representation.

9. The method of claim 1 or 7, wherein the continuous multivariate interpolation function is linearized as a linearly weighted sum of modifiers applied to the morph target shapes and constraint shapes.

10. The method of claim 9 , wherein the linear weighted sum is based on an approximate Taylor series expansion.

11. The method of claim 10, wherein the accuracy of the interpolation is further improved by interpolating between the incremental shapes using mesh or meshless interpolation.

12. The continuous multivariate interpolation function has the formula: [0010] In the formula, ΔB k represents the corrective morph target shape; ΔC i represents the modified constraint shape; β i is each constraint shape C i represents the modifiers applied to The continuous multivariate interpolation function is 1・・・ C m The method of claim 7 , configured to hold for

13. A data processing device or animation system comprising means for carrying out the method according to any one of claims 1 to 12.

14. A computer program or a computer readable medium having a computer program stored thereon, comprising instructions which, when the computer program is run on a computer, cause the computer to carry out the method according to any of claims 1 to 12.

Images