System and method for high resolution texture interpolation based on facial expressions

Abstract

The invention discloses a method for determining a target expression texture. The method comprises the following steps: determining a target expression and a feature map of each training posture in P training postures; determining a mixed weight tv based on the following contents: using an approximation model trained by training gestures and a similarity measure # imgabs0 #, wherein the similarity measure # imgabs1 # represents the similarity between the target feature map at the vertex v and each training feature map; for each pixel of the plurality of 2D pixels: determining one or more vertices corresponding to the pixel based on the 2D coordinates of the pixel and the vertex-to-2D mapping; and determining a per-pixel mixed weight rn of the pixel based on the mixed weight tv of the corresponding vertex. The method further includes, for each high resolution pixel in the 2D rendering of the target facial expression, interpolating the 2D texture of the training facial pose based on the per-pixel mix weight rn, thereby providing a target texture.

Description

Technical Field

[0001] The present invention relates generally to computer animation and, more particularly, to systems and methods for rendering photo-realistic faces with animated skin textures in real time based on texture interpolation of known facial expressions. Background Art

[0002] Generally speaking, rendering means creating an image from a computer-based model. Animation typically involves a series of images (frames) displayed at a frame rate sufficient to make objects and characters appear to be moving. In facial animation, rendering typically involves creating two-dimensional (2D) images from three-dimensional (3D) models. Producing high-resolution facial renderings (e.g. for animated films) with current technology typically requires first scanning an actor's face using appropriate facial capture technology. In order to achieve photorealism and complexity in the look and feel of a character's skin in the corresponding facial rendering, it is typically necessary to scan the actor's face multiple times to obtain a variety of detailed skin information or textures, such as albedo, diffuse, and specular values ​​under various lighting conditions.

[0003] A shader is a computer program that calculates the appearance of a digital material based on incident light, viewing direction, and the material's properties (e.g., albedo, diffuse, specular, etc.), typically provided as texture maps or vertex attributes. Shaders can be used to apply various properties and features specific to certain vertices (e.g., in a 3D face model) and / or corresponding rendered pixels (in a rendered 2D image). Rendering of animation frames typically occurs "offline" (rendering many image frames before displaying the corresponding image) or "real time" (rendering image frames between displays of consecutive corresponding images). Similarly, shader programs are suitable for both offline and real-time applications. Offline shaders are typically executed by one or more general-purpose central processing units (CPUs), and real-time shaders are typically executed by one or more graphics-specific graphics processing units (GPUs).

[0004] Prior to rendering several different facial expressions, high-resolution textures may be captured. Sometimes on the order of 12 different texture types are captured, as well as 20 different facial expressions (also referred to herein as poses). The resulting data set is large and necessarily results in a correspondingly high computational cost to the processor executing the shader software, and a correspondingly large amount of computational time is required to render the final set of images of the animation. Furthermore, the pose (facial expression) associated with a rendered frame (rendered pose) is typically different from any captured pose. One reason for the high computational demands associated with shading is that the shader typically needs to blend or interpolate textures from multiple captured poses in the data set so that the final interpolated texture generates an accurate skin representation for the currently rendered pose. This huge computational overhead is a problem, particularly for real-time applications, where frame rates can be 6fps, 12fps, 30fps, 60fps, or higher.

[0005] Furthermore, existing technical solutions for rendering faces in animated films typically focus on controllability to achieve the desired art-directed effects. Further processing necessarily increases the file size and storage requirements of the resulting renders generated by the shaders.

[0006] Several methods have been proposed to provide real-time face rendering. One method is to calculate a scalar that represents the level of skin stretch / compression for each vertex of a parameterized face model (e.g., a 3D CG model). This single value can then be used in the shader to blend textures from multiple poses (described below) on the image pixel associated with that vertex. In its simplest form, the method requires three input poses, one in a relaxed or neutral state, one in a maximum stretched state, and one in a maximum compressed state. The shader is then able to use the compression ratio and blend the textures from these poses with a polynomial or spline equation at render time.

[0007] One drawback of this approach is that, with the exception of the neutral state texture, the other two textures (maximum stretch and maximum compression) cannot be captured directly from the actor's skin, since it is impossible to fully compress or stretch all areas of the face at the same time. Therefore, these textures must be carefully "painted" or otherwise created by a skilled artist. In addition, skin is subject to other types of stresses, such as distortion and asymmetric stretching and compression in orthogonal directions. Therefore, this state-of-the-art real-time shading solution can now achieve realistic skin reproduction in these scenes.

[0008] Another prior art approach to real-time rendering is to drive textures based on a set of predefined key poses (called blend shapes). Appropriately weighted sets of blend shapes can be used to approximate a wide range of 3D model poses. With this prior art, a face rendering is produced by selecting two or more of the most relevant poses (blend shapes) and inserting textures proportional to the activations (weights) of the corresponding blend shapes. In this solution, all possible blend shape poses must be captured and a large number of blend shape poses and textures are produced, typically around 100-200. Many textures become redundant because blend shapes are often local, such as lips moving forward while other features remain neutral. This method produces ideal results if the goal is to achieve an artistically guided result, but it is often too inefficient for real-time applications.

[0009] Some prior art approaches for producing face renderings (usually offline face renderings) follow the Facial Action Coding System (FACS), which encodes muscle-specific shapes. These muscle shapes are called action units or AUs, and can be used to encode almost any anatomically possible facial expression. Each FACS shape can be associated with a texture, and the same AU activation weights can be used to drive the blending of textures. For example, basic emotions can be defined based on the combination of multiple AUs, and the weights of these AUs can then be imported into a computer face rendering model to derive a set of animated textures.

[0010] There are other rendering approaches that use regression models to predict wrinkle formation on facial patches based on the overall, large-scale facial appearance. However, this approach only focuses on the texture associated with the specific part of the face where wrinkles occur. This regression model finds interpolation weights for multiple texture patches, so a separate step is required to blend all the patches. These methods tend to be tailored for each actor and require artists to create blending settings or split the face into patches.

[0011] There remains a need for face rendering techniques and systems that can represent facial texture variations due to deformation patterns of a limited set of scanned facial poses, thereby improving upon the prior art and / or ameliorating these or other shortcomings of the prior art. There is a particular need for real-time face rendering techniques.

[0012] The foregoing examples of the related art and their related limitations are intended to be illustrative rather than exclusive. Other limitations of the related art will be apparent to those skilled in the art by reading the specification and studying the drawings. Summary of the invention

[0013] The following embodiments and aspects thereof are described and illustrated in conjunction with systems, tools and methods, which are intended to be exemplary and illustrative, rather than limiting in scope. In various embodiments, one or more of the above problems have been reduced or eliminated, while other embodiments are directed to other improvements.

[0014] One aspect of the present invention provides a method for determining a high-resolution texture for rendering a target facial expression. The method includes: (a) obtaining a plurality of P training facial poses, each of the P training facial poses including: V high-resolution vertices and training pose positions of the V high-resolution vertices in a three-dimensional (3D) coordinate system; and at least one two-dimensional (2D) texture; (b) obtaining a target facial expression, the target facial expression including V high-resolution vertices and target expression positions of the V high-resolution vertices in a 3D coordinate system; (c) determining a target feature map based at least in part on the target facial expression and a neutral pose selected from the P training poses, the target feature map including a plurality of target feature map parameters of the target facial expression; (d) determining a training feature map including a plurality of training feature map parameters of the P training facial poses, each training feature map based at least in part on one of the P training facial poses and the neutral pose; (e) for each high-resolution vertex v in the V high-resolution vertices, determining a plurality of blending weights t based at least in part on the following factors: v : An approximate model trained using P training face poses; and a similarity matrix It represents the similarity between the target feature map at vertex v and each training feature map; (f) for each pixel n of multiple N pixels in 2D space: (f.1) at least partially based on the 2D coordinates of pixel n in 2D space and the mapping of multiple high-resolution vertices v to 2D space, determine one or more corresponding vertices corresponding to pixel n from multiple high-resolution vertices v, the mapping providing 2D coordinates of multiple high-resolution vertices v in 2D space; and (f.2) at least partially based on the mixing weights t of one or more corresponding vertices v Determine the per-pixel blending weight r of the group for pixel n n , the per-pixel mixing weight r n comprising a weight for each P training facial pose; and (g) for each high-resolution pixel in the 2D rendering of the target facial expression, a per-pixel blending weight r based at least in part on n The 2D textures of P training face poses are interpolated to provide the target texture for high-resolution pixels.

[0015] Determining a training feature map for a training pose P may include determining a plurality of W low-resolution handle vertices, wherein the W handle vertices are low-resolution subsets of the V high-resolution vertices, wherein W <V。

[0016] Determining training feature maps for P training poses may include, for each of the P training facial poses: determining a training feature map geometry corresponding to the training facial pose, the training feature map geometry including a plurality of F feature edges defined between training pose positions of corresponding pairs of low-resolution handle vertices of a plurality of low-resolution W handle vertices of the training pose; determining a plurality of training feature map parameters as a plurality of F training feature map parameters corresponding to the F feature edges, each of the plurality of F training feature map parameters being based at least in part on a corresponding feature edge of the training facial pose and a corresponding feature edge of a neutral pose; thereby obtaining P training feature maps, each of the P training feature maps including a corresponding plurality of F training feature map parameters.

[0017] Determining a plurality of feature graph parameters of the F feature graph parameters corresponding to the F feature edges may include, for each training feature graph parameter of the plurality of F training feature graph parameters, determining the training feature graph parameter using an equation of the form f i =(‖p i,1 -p i,2 ‖-l i ) (1) Where: f i is with the i th The i-th feature edge th Feature map parameters; p i,1 and p i,2 It is defined as i The training pose position of the handle vertex of the endpoint; l i is the i-th position in the neutral posture th The length of the side.

[0018] Determining training feature maps for P training poses may include, for each of the P training facial poses: determining a training feature map geometry corresponding to the training facial pose, the training feature map geometry comprising a plurality of F feature edges defined between training pose positions of corresponding pairs of low-resolution handle vertices of a plurality of low-resolution W handle vertices of the training pose; determining a plurality of training feature map parameters, each of the plurality of training feature map parameters being based at least in part on some or all of the F feature edges of the training facial pose and some or all of the feature edges of a neutral pose; thereby obtaining P training feature maps, each of the P training feature maps comprising a corresponding plurality of training feature map parameters.

[0019] Determining training feature maps for P training postures may include, for each of the P training facial postures: determining a plurality of training feature map parameters, each of the plurality of training map feature parameters being at least partially based on one or more original parameters, the one or more original parameters being determined at least partially based on the training facial posture and a neutral posture; thereby obtaining P training feature maps, each of the P training feature maps including a corresponding plurality of training feature map parameters.

[0020] Determining multiple training feature map parameters may include determining one or more of: deformation gradients based at least in part on training face poses and neutral poses; pyraid coordinates based at least in part on training face poses and neutral poses; triangle parameters based at least in part on training face poses and neutral poses; and 1-ring neighbor parameters based at least in part on training face poses and neutral poses.

[0021] Determining a target feature map may include: determining a target feature map geometry corresponding to a target facial expression, the target feature map geometry including a plurality of F feature edges defined between target expression positions of corresponding pairs of low-resolution handle vertices of a plurality of low-resolution W handle vertices of the target expression; determining a plurality of target feature map parameters as a plurality of target feature map parameters among F target feature map parameters corresponding to the F feature edges, each of the plurality of F target feature map parameters being based at least in part on a corresponding feature edge of the target facial expression and a corresponding feature edge of a neutral posture; thereby obtaining a target feature map including a plurality of F target feature map parameters.

[0022] Determining a plurality of F feature graph parameters corresponding to the F feature edges may include, for each feature graph parameter in the plurality of F feature graph parameters, determining a target feature graph parameter using an equation of the form, f i =(‖p i,1 -p i,2 ‖-l i ) / l i (1) Where: f i is with the i th The i-th feature edge corresponding to th Feature map parameters; p i,1 and p i,2 It is defined as i The endpoints of the handle vertices are the target expression positions, and l i is the corresponding i-th position in the neutral postureth The length of the feature edge.

[0023] Determining a target feature map may include: determining a target feature map geometry corresponding to a target facial expression, the target feature map geometry including a plurality of F feature edges defined between target expression positions of corresponding pairs of low-resolution handle vertices of a plurality of low-resolution W handle vertices of the target facial expression; determining a plurality of target feature map parameters, each of the plurality of target feature map parameters being based at least in part on some or all of the F feature edges of the target facial expression and some or all of the feature edges of a neutral posture; thereby obtaining a target feature map including a plurality of target feature map parameters.

[0024] Determining a target feature map may include: determining multiple target feature map parameters, each of the multiple target map feature parameters is at least partially based on one or more original parameters, and the one or more original parameters are determined based on the target facial expression and neutral posture; thereby obtaining a target feature map including multiple target feature map parameters.

[0025] Determining multiple target feature map parameters may include determining one or more of: deformation gradients based at least in part on the target facial expression and neutral posture; pyramid coordinates based at least in part on the target facial expression and neutral posture; triangle parameters based at least in part on the target facial expression and neutral posture; and 1-ring neighborhood parameters based at least in part on the target facial expression and neutral posture.

[0026] Determining one or more corresponding vertices from the plurality of high-resolution vertices v corresponding to pixel n may be based at least in part on proximity of 2D coordinates of the one or more corresponding vertices to the 2D coordinates of pixel n.

[0027] Determining one or more corresponding vertices from the plurality of high-resolution vertices v corresponding to the pixel n may include determining three vertices whose two-dimensional coordinates are closest to the two-dimensional coordinates of the pixel n, thereby defining a triangle around the pixel n in two-dimensional space.

[0028] Determining one or more corresponding vertices from the plurality of high-resolution vertices v corresponding to the pixel n may include determining the barycentric coordinates of a triangle relative to the 2D coordinates of the pixel n.

[0029] Determining one or more corresponding vertices from the plurality of high-resolution vertices v corresponding to the pixel n may include determining a vertex whose two-dimensional coordinates are closest to the two-dimensional coordinates of the pixel n.

[0030] The method may include selecting multiple high-resolution vertices v from V high-resolution vertices as a union of one or more high-resolution vertices corresponding to each pixel n in multiple N pixels; wherein, selecting multiple high-resolution vertices v from the V high-resolution vertices is performed before step (e), so that step (e) is performed only for the multiple high-resolution vertices v selected from the V high-resolution vertices.

[0031] The method may include selecting a plurality of high-resolution vertices v from V high-resolution vertices as all V high-resolution vertices.

[0032] Determine the set of per-pixel blending weights r for pixel n n The blending weights t may be based at least in part on three vertices defining a triangle around pixel n in 2D space. v and the barycentric coordinates of the triangle relative to the 2D coordinates of pixel n to determine the set of per-pixel blending weights r for pixel n n .

[0033] Determine the per-pixel blending weight r of the group for pixel n n It can include execution forms of r n =γ A t A +γ B t B +γ C t C operation, where t A ,t B ,t C represents the mixing weight t determined in step (e) v , used to define the three vertices of the triangle around pixel n in 2D space, γ A ,γ B ,γ C Represents the barycentric coordinates of the triangle relative to the 2D coordinates of pixel n.

[0034] Determine the per-pixel blending weight r of the group for pixel n n This may include: determining the set of per-pixel blending weights r for pixel n n is the blending weight t determined in step (e) for a vertex whose 2D coordinate is closest to the 2D coordinate of pixel n v .

[0035] The 2D space may be a 2D rendered 2D space.

[0036] The 2D space may be different from the space that the 2D rendering is in. The 2D space may be a UV space.

[0037] The plurality of N pixels in the 2D space may have a lower resolution than the pixels in the 2D rendering.

[0038] Interpolating the 2D texture for the P training face pose may be based at least in part on the locations of the high-resolution pixels mapped to the 2D space.

[0039] The method may include determining an approximate model based at least in part on a training feature map of P training facial poses.

[0040] Determining the approximate model may include, for each high-resolution vertex v in the V high-resolution vertices: solving the form w = dφ -1 , where: d is a P-dimensional identity matrix; and φ is a PxP-dimensional matrix of weighted radial basis functions (RBFs), where each element of v is based at least in part on training feature map parameters of the compared P training feature maps; thereby obtaining a PxP-dimensional matrix of RBF weights w corresponding to the high-resolution vertex v, thereby defining an approximate model including a set of V RBF weight matrices w.

[0041] For each high-resolution vertex v among the V high-resolution vertices, determining the weights of the weighted radial basis function (RBF) in the matrix φ can be based at least in part on a proximity mask that assigns a value of unity to feature edges around the vertex v and assigns a falloff value to feature edges farther away from the vertex v.

[0042] The proximity mask assigns exponentially decaying values ​​to feature edges that are farther away from vertex v.

[0043] The proximity mask can be determined according to an equation of the following form: Where: l i is the i-th neutral posture th The length of the feature edge; L v,i is the i-th path from vertex v to the neutral pose th is the sum of the neutral pose distances of the endpoint locations of the feature edges; and β is a configurable scalar parameter that controls the decay rate.

[0044] The method may include determining that the calculated i-th th If the proximity mask of the feature edge is less than the threshold, the i-th th The proximity mask of feature edges is set to zero.

[0045] The proximity mask may include assigning non-zero values ​​to a configurable number of feature edges that are relatively closer to vertex v, and assigning zero values ​​to other feature edges that are relatively farther away from vertex v.

[0046] Located at k th Column and l th The elements of the PxP matrix of the weighted RBF of the row (φ v,k,l ) can be given by an equation of the following form: Where: γ is the RBF kernel function; f k,i is the kth th Training posture i th The training feature map parameters of the feature edge; f l,i It is the first th Training posture i th The training feature map parameters of the feature edge; and α v,i Based on the i th The proximity of the feature edge to the high-resolution vertex v is assigned to the i-th th The weight of the feature edge.

[0047] The RBF kernel function γ may be a biharmonic RBF kernel function.

[0048] Determining the approximation model may include training the approximation model based at least in part on training feature map parameters of the P training feature maps to solve the sparse interpolation problem.

[0049] The method may include determining a similarity measure according to an equation of the form It represents the similarity between the target feature map at vertex v and each of the P training feature maps, which is a P-dimensional similarity vector The similarity vector K th element Where: γ is the RBF kernel function; f t,i is the i-th target feature map th The target feature map parameters of the feature edge; f k,i is used for the kth th The i-th training feature map of the training posture th The training feature map parameters of the feature edge; α v,i is assigned to the i-th vertex based on its proximity to the high-resolution vertex v. th The weight of the feature edge, and i(i∈1,2,…F) is an indicator describing the number of edges in each training feature map and the target feature map.

[0050] Determine multiple mixing weights t v Including execution form The operation is: where w is the RBF weight matrix of the high-resolution vertex v, It is a P-dimensional similarity vector, which represents the similarity between the target feature map at vertex v and each of the P training feature maps.

[0051] Interpolating the 2D textures of the P training face poses may include: performing texture query on the 2D textures of the P training face poses based on the high-resolution pixels in the 2D rendering to obtain each of the P training face poses (tex1, tex2, tex3 ... tex P ), each interpolated texture value is interpolated between texture values ​​at multiple texels of a 2D texture of a training face pose corresponding to one of the P training face poses; performing a weighted texture query on the 2D space based on the high-resolution pixels in the 2D rendering to obtain a set of interpolated per-pixel blending weights r * (r1,r2…r P ), which interpolates the per-pixel blending weight r of multiple pixels n in 2D space n between.

[0052] Interpolating the 2D textures of the P training face poses may include determining a target texture of high-resolution pixels (texture) according to an equation of the form, texture=tex1·r1+tex2·r2+…tex P ·r P Where: (tex1,tex2,tex3…tex P ) are the interpolated texture values ​​of P training face poses; and (r1,r2…r P ) is a set of per-pixel blending weights r for interpolation * .

[0053] Performing texture queries on the 2D textures of the P training face poses based on the high-resolution pixels in the 2D rendering may include mapping the high-resolution pixels in the 2D rendering to a UV space to determine 2D coordinates of the high-resolution pixels in the UV space.

[0054] Performing a weighted texture query on the 2D space based on the high-resolution pixels in the 2D rendering may include mapping the high-resolution pixels in the 2D rendering to the 2D space to determine a 2D coordinate of the high-resolution pixels in the 2D space.

[0055] Interpolating the 2D textures of the P training face poses may include: performing texture query on the 2D textures of the P training face poses based on the high-resolution pixels in the 2D rendering to obtain each of the P training face poses (tex1, tex2, tex3 ... tex P), each interpolated texture value interpolated between texture values ​​at multiple texels of the 2D texture of a corresponding training face pose in the P training face poses.

[0056] Interpolating the 2D textures of the P training face poses involves determining the target texture of the high-resolution pixels (texture) according to an equation of the following form: texture=tex1·r1+tex2·r2+…tex P ·r P Among them, (tex1,tex2,tex3…tex P ) is the interpolated texture value for each P training face pose; and (r1,r2…r P ) is a set of interpolated per-pixel blending weights r for high-resolution pixels in 2D space for 2D rendering n .

[0057] Performing a texture query on the 2D texture of the P training face pose based on the high-resolution pixels in the 2D rendering may include mapping the high-resolution pixels in the 2D rendering to a UV space to determine 2D coordinates of the high-resolution pixels in the UV space.

[0058] The method can be used to render an animation sequence including a plurality of animation frames corresponding to a plurality of target facial expressions at an animation frame rate. Steps (a), (d) and (f.1) can be performed for the animation sequence as a pre-calculation step before one or more of steps (b), (c), (e), (f.2) and (g); and steps (c), (e), (f.2) and (g) can be performed for the animation sequence in real time as part of step (b) when corresponding expressions among the plurality of target facial expressions are obtained.

[0059] A set of per-pixel blending weights r for each pixel n n may include a vector of P elements; and the method may include adding a per-pixel blending weight r n The corresponding elements of are provided to a graphics processor in color channels of one or more corresponding images having N pixels.

[0060] The method may be used to render an animation sequence including a plurality of animation frames corresponding to a plurality of target facial expressions at an animation frame rate. Steps (a), (d) and (f.1) may be performed for the animation sequence as a pre-calculation step prior to one or more of steps (b), (c), (e), (f.2) and (g). Steps (c), (e), (f.2) and (g) may be performed for the animation sequence in real time as part of step (b) when corresponding expressions in the plurality of target facial expressions are obtained. A set of per-pixel blending weights r for each pixel nn A vector may contain P elements. The method may include assigning a per-pixel blending weight r n The corresponding elements of the image are provided to the graphics processor in color channels of one or more corresponding low-resolution images, the resolution of which is lower than the resolution of the image being rendered.

[0061] The color channels of the one or more corresponding low-resolution images may include red, blue, and green (RGB) color channels.

[0062] The plurality of P training postures may include P training postures, where the number P is a multiple of three.

[0063] Another aspect of the present invention provides a computer-implemented method for training an approximate model of facial poses, which can be used to determine the similarity of a target facial expression to a plurality of training facial poses at each high-resolution vertex v in a set of V high-resolution vertices, which define a topological structure common to the target facial expression and the plurality of training facial poses. The method includes: obtaining a plurality of P training facial poses, each of the P training facial poses including a training pose position of V high-resolution vertices in a three-dimensional (3D) coordinate system; for each P training facial pose: determining a training feature map including a corresponding plurality of training feature map parameters, wherein each of the corresponding plurality of training feature map parameters is at least partially based on one or more original parameters, which are determined at least partially based on the training facial pose and a neutral pose selected from the P training poses; thereby obtaining P training feature maps, each of the P training feature maps including the corresponding plurality of training feature map parameters; and for each high-resolution vertex v in the V high-resolution vertices: solving the form of w=dφ -1 An equation is obtained in which: d is a P-dimensional identity matrix; and φ is a PxP-dimensional matrix of weighted radial basis functions (RBFs), wherein each element of φ is based at least in part on comparing training feature map parameters of P training feature maps; thereby obtaining a PxP-dimensional matrix of RBF weights w corresponding to high-resolution vertices v, thereby defining an approximate model comprising a set of V RBF weight matrices w.

[0064] The method may include determining a plurality of low-resolution W handle vertices, wherein the W handle vertices are a low-resolution subset of the V high-resolution vertices, wherein W <V。

[0065] Determining a training feature map including a corresponding plurality of training feature map parameters may include: determining a training feature map geometry corresponding to a training facial pose, the training feature map geometry including a plurality of F feature edges defined between training pose positions of corresponding pairs of low-resolution handle vertices of a plurality of low-resolution W handle vertices of the training pose; determining a plurality of training feature map parameters, each of the plurality of training feature map parameters being based at least in part on some or all of the F feature edges of the training facial pose and some or all of the feature edges of the neutral pose.

[0066] Determining a training feature map including a corresponding plurality of training feature map parameters may include: determining a training feature map geometry corresponding to a training facial pose, the training feature map geometry including a plurality of F feature edges, the plurality of F feature edges being defined between training pose positions of pairs of low-resolution handle vertices corresponding to a plurality of low-resolution W handle vertices of the training pose; determining the plurality of training feature map parameters as a plurality of F training feature map parameters corresponding to the F feature edges, each of the plurality of F training feature map parameters being based at least in part on a corresponding feature edge of the training facial pose and a corresponding feature edge of a neutral pose; thereby obtaining P training feature maps, each of the P training feature maps including a corresponding plurality of F training feature map parameters.

[0067] Determining a plurality of F training feature graph parameters corresponding to the F feature edges may include, for each training feature graph parameter in the plurality of F training feature graph parameters, determining the training feature graph parameter using an equation of the following form, f i =(‖p i,1 -p i,2 ‖-l i ) / l i (1) Where: f i is with the i th The i-th feature edge th Feature map parameters; p i,1 and p i,2 It is defined as i The training pose position of the handle vertex of the endpoint; l i is the corresponding i-th position in the neutral posture th The length of the feature edge.

[0068] Determining a training feature map comprising a corresponding plurality of training feature map parameters may include determining one or more of: deformation gradients based at least in part on training face poses and a neutral pose; pyramid coordinates based at least in part on training face poses and a neutral pose; triangle parameters based at least in part on training face poses and a neutral pose; and 1-ring neighborhood parameters based at least in part on training face poses and a neutral pose.

[0069] For each high-resolution vertex v among the V high-resolution vertices, determining the weights of the weighted radial basis function (RBF) in the matrix φ can be based at least in part on a proximity mask that assigns a uniform value to feature edges around the vertex v and assigns a decaying value to feature edges farther away from the vertex v.

[0070] The proximity mask can assign exponentially decaying values ​​to feature edges that are farther away from vertex v.

[0071] The proximity mask can be determined according to an equation of the following form: Where: l i is the i-th neutral posture th The length of the characteristic edge; L v,i is the i-th path from vertex v to the neutral pose th is the sum of the neutral pose distances of the endpoint positions of the feature edges; and β is a configurable scalar parameter used to control the decay rate.

[0072] The method may include determining that the calculated i-th th If the proximity mask of the feature edge is less than the threshold, the i-th th The proximity masks of feature edges are set to zero.

[0073] The proximity mask may include assigning non-zero values ​​to a configurable number of feature edges that are relatively closer to vertex v, and assigning zero values ​​to other feature edges that are relatively farther away from vertex v.

[0074] Located at k th Column and l th The elements of the PxP matrix φ of the weighted RBF of the row (φ v,k,l ) can be given by an equation of the following form: Where: γ is the RBF kernel function; f k,i is the kth th The i-th training posture th The training feature graph parameters of feature edges; f l,i is the i th The i-th training posture th The training feature map parameters of feature edges; and α v,i Based on the i th The proximity of the feature edge to the high-resolution vertex v is assigned to the i-th th The weight of a feature edge.

[0075] The RBF kernel function γ may be a biharmonic RBF kernel function.

[0076] Another aspect of the present invention provides a system comprising one or more processors configured to perform any of the methods described above or elsewhere herein.

[0077] In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the drawings and studying the following detailed description. It should be emphasized that the present invention relates to all combinations and subcombinations of the above-mentioned features and other features described herein, even if these combinations and subcombinations are described in different claims or claims with different dependencies. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Exemplary embodiments are illustrated with reference to the figures in the accompanying drawings.The embodiments and drawings disclosed herein are intended to be considered illustrative rather than restrictive.

[0079] Figure 1 is a flow chart describing a method for interpolating a high-resolution texture based on a set of face training data, according to certain embodiments. Figure 1A Depicted are exemplary systems for performing one or more methods described herein (eg, the methods in the Figures and / or any sub-processes thereof) according to certain embodiments.

[0080] Figure 2A is a flow chart illustrating a method of selecting a set of related postures from a plurality of training postures and determining their feature map geometry according to certain embodiments. Figure 2B is a flow chart illustrating a method of determining a training posture feature map according to certain embodiments.

[0081] Figure 3A An example of the marker layout used on an actor's face is shown. Figure 3B A low-resolution 3D mesh (feature map geometry) view (3D model) of the face is shown. Figure 3C A high-resolution view of facial expressions is shown.

[0082] Figure 4 is a flowchart illustrating a method of training an approximation model according to a specific embodiment of the present invention.

[0083] Figure 5 is a flow chart illustrating a method of performing rasterization calculations according to certain embodiments of the present invention.

[0084] Fig. 6A is an illustration of an exemplary UV mapping of an example face topology and an overlaid discretization of the UV space, showing the per-pixel coverage of the low-resolution weight texture map. Figure 6B yes Fig. 6A A detailed view of an example pixel (or conceptual unit) of a UV mapping, depicting the center of the pixel (or conceptual unit) and the surrounding vertices in UV space.

[0085] Figure 7 is a flow chart illustrating a method of performing real-time calculations for interpolating high-resolution textures according to certain embodiments. Fig. 7A is a flow chart illustrating a method for interpolating training pose weights to obtain blend weights, which method may be used to perform according to a specific embodiment Figure 7 A step. Figure 7B is a flow chart illustrating another method for interpolating a high-resolution texture according to another specific embodiment.

[0086] Figure 8 is an example of a set of weight textures represented as two RGB pixel maps. DETAILED DESCRIPTION

[0087] Specific details are set forth in the following description to provide a more thorough understanding for those skilled in the art. However, well-known elements are not shown or described in detail to avoid unnecessarily obscuring the disclosure. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

[0088] Figure 1 A high-resolution facial texture interpolation method 100 according to an example embodiment of the present invention is schematically illustrated. Generally speaking, the method 100 includes a training portion 133 and a rendering / interpolation portion 130. The training portion 133 of the method 100 receives a plurality of training poses 107 as input and generates an approximation model 123 and a rasterization matrix 127. The rendering / interpolation portion 130 of the method 100 interpolates between high-resolution textures associated with the training poses 107 to determine a texture suitable for rendering the target facial expression 503, thereby providing a rendered facial image 585. In some embodiments, the rendering / interpolation portion 130 of the method 100 is performed in real time (e.g., rendering animation frames at a frame rate of 24 frames per second or more in the intervals between consecutive frames), while the training portion 133 is performed "offline" (i.e., not in real time). In some embodiments, the method 100 is performed according to calculation parameters 113. Some of the calculation parameters 113 may be input into the method 100. Some of the calculation parameters 113 may be determined as part of the method 100.

[0089] Some aspects of the present invention provide a system 60 (an example implementation of which is Figure 1A ), for performing one or more of the methods described herein (e.g., Figure 1The system 60 may include a processor 62, a memory module 64, an input module 66, and an output module 68. The memory module 64 may store one or more data, models, and / or illustrations described herein. The processor 62 may receive the training poses 107 (via the input module 66) and may store these inputs in the memory module 64. The processor 62 may perform the training portion 133 of the method 100 to develop the approximate model 120 and the rasterization matrix 127 described herein, and store the model 120 and the matrix 127 in the memory module 64. The processor 62 may receive some of the calculation parameters 113 (via the input module 66) or may determine some of the calculation parameters 113 and may store these data in the memory module 64. In some embodiments (by way of non-limiting example, where real-time rendering is not involved), the processor 62 may also perform some or all of the interpolation / rendering portion 130, in which case the processor 62 may receive the target facial expression 503 and generate a rendered image 585 (which may be stored in the memory module 64). In other embodiments (by way of non-limiting example, where real-time rendering is involved, some or all of the interpolation / rendering portion 130 may be performed by a separate processor (e.g., a graphics processing unit (GPU) dedicated to rendering images, etc.)).

[0090] Method 100 begins at blocks 105 and 110. Block 105 includes obtaining training data including a plurality of various high-resolution 3D facial training poses 107. The 3D training poses 107 may be obtained at block 105 in any suitable manner. For example, the training poses 107 may be determined by capturing an actor's facial performance using a suitable facial capture studio including a plurality of cameras. For example, the facial capture studio may include a lighting stage setup (e.g., a lighting stage provided by University of Southern California ICT and Otoy Inc.) or a static multi-view capture setup. These methods of obtaining 3D training poses 107 typically include a plurality of different facial expressions controlled by an actor performing under a variety of different lighting conditions.

[0091] Each training pose 107A in the plurality of training poses 107 may include or be derived from a series of still images in the form of encoded digital data that together capture the facial geometry and texture information for that pose. The series of still images for each pose 107A may include images acquired by the facial capture hardware (e.g., a camera) from various angles and under various lighting conditions, for example, one still image per camera for each pose 107A. These individual images may be pre-processed and combined to form one or more 3D geometry / 3D models (typically 3D meshes) and one or more high-resolution texture maps.

[0092] In some embodiments, training poses 107 obtained through light-stage techniques involving actors are modified by artists to improve accuracy or to compensate for deficiencies in the light-stage process. Certain aspects of training poses 107, such as the geometry of the pose, may be obtained through a manual modeling process by artists using 3D modeling software, either in conjunction with or as an alternative to capture techniques. Artists may also manually make modifications to mimic certain facial features, such as those of a celebrity to be rendered.

[0093] The training poses 107 as a group may preferably include many different facial expressions that capture various skin deformation patterns in the form of a group or model of 3D facial geometries (e.g., a 3D geometry or 3D mesh configuration for each training pose 107A). For example, different facial expressions that may be depicted in the training poses 107 include a smiling expression, a frowning expression, and a raised eyebrow expression, among others. In some embodiments, a classification system for human facial movements is used to determine the various facial expressions that are part of the training poses 107. For example, the facial expressions in the poses 107 may include FACS expressions or other standardized expressions. In some embodiments, the block diagram 105 includes obtaining 10 to 30 different training poses 107, although fewer training poses 107 or a greater number of training poses 107 may be obtained. According to a specific non-limiting example embodiment, the block diagram 105 includes obtaining data from 20 training poses.

[0094] Each pose or expression 107A (from a plurality of training poses 107) includes a set of high-resolution 2D textures that reflect the appearance properties of the skin at that particular pose 107A. Each high-resolution 2D texture can be provided in the form of a 2D image of high-resolution pixels (commonly referred to as texels) that are located in a 2D space commonly referred to as UV space. In some embodiments, approximately 12 types of textures are captured at each pose 107A, although a different number of textures may be used. These different textures capture different properties of the skin in a manner that can be recombined by shader software and used for rendering. For example, a shader program may interpret many different textures that cumulatively represent the appearance of how a single point on the surface of the face reflects and / or absorbs incident light. For example, specular and roughness texture values ​​represent the reflection of the oil layer of the skin, diffuse reflectance values ​​reflect the color of the epidermis of the skin, and normal values ​​reflect the bumps and bumps of the skin. Other texture properties known in the art (including but not limited to occlusion, curved normals, environment, and albedo) may be included in a set of 2D textures included in or otherwise associated with each pose 107A of the training poses 107. Scans of various detailed texture information of the actor's face may be acquired under various lighting conditions to obtain the different textures required for each pose 107A.

[0095] At block 110, method 100 involves evaluating a plurality of computational parameters according to a computational budget. In some applications, the computational budget may be a fixed or given parameter of method 100. In such applications, the computational parameters of block 110 may be received (e.g., as an input), hard-coded, or otherwise determined based on the fixed or given computational budget. There are a number of key considerations in the evaluation performed at block 110. It is contemplated that the method 100 may be used in real-time rendering to interpolate realistic high-resolution textures (e.g., in applications such as video games or for rendering digital (e.g., CGI) avatars based on human faces. Therefore, in embodiments where the method 100 is used in real-time scenarios, one of the main block diagram 110 considerations is the desired frame rate in frames per second (FPS) for real-time rendering. In such embodiments, the block diagram 110 may include determining the desired frame rate for real-time texture rendering. Generally speaking, for higher desired frame rates for rendering, the overall computational budget will be correspondingly higher because the method 100 may be executed more times within the same time frame or equivalently, the period for rendering each successive frame is correspondingly lower. Therefore, in some embodiments where the overall computational budget is fixed or given, by selecting an appropriate frame rate, the computational budget for rendering individual frames can be adjusted as a result. As illustrative examples, the frame rate determined at the block diagram 110 may be 24 FPS, 30 FPS, 48 FPS, 60 FPS, 120 FPS, or 240 FPS.

[0096] Another consideration of block diagram 110 (relevant to certain applications of method 100) is the desired weight texture resolution. As will be described in further detail below, embodiments of the present invention relate to interpolating a high-resolution texture based on a low-resolution weight texture map 575 (also referred to herein for brevity as weight texture map 575). As explained in detail below, the low-resolution weight texture map is a two-dimensional array of pixels that spans the 3D face surface topology in UV space and provides a plurality of weight texture values ​​for each pixel n (where n∈1,2...N), which may be represented by a weight texture vector r n (567). In addition to determining the desired frame rate, block diagram 110 may also include determining a resolution for the low-resolution weight texture map 575 based on a computational budget. As illustrative examples, the weight texture map 575 may include resolutions of 32x32, 64x64, 128x128, 256x256, 512x512, and 1024x1024 pixels. Other solutions are possible. Where a lower weight texture map resolution is used, coarser areas of the rendered face 585 are determined by a corresponding combination of interpolation weights, and the computational cost may be relatively low. Conversely, finer areas of the rendered face 585 are determined by a corresponding combination of interpolation weights, where a higher resolution weight texture map resolution is used and the computational cost may be relatively high.

[0097] In addition, block 110 may include evaluating a number P of poses 107A in the set of training poses 107 for interpolating a high-resolution texture of a target facial expression (i.e., a target facial pose to be rendered). As will be described in further detail below, method 100 determines the texture of the target facial expression based on interpolating the high-resolution texture corresponding to the P training poses. Therefore, the greater the number P of training poses 107A used in the interpolation calculation, the higher the computational cost. For example, increasing the number P of training poses 107A quadratically increases the complexity of the blending weights (described below) used in the real-time texture interpolation calculation. However, increasing the number P of training poses used can more accurately make the texture of the target facial pose consistent with the texture of the most representative combination in the set of available training poses 107. The selection of the number P of poses 107A will also have an impact on the training process performed at block 120, as described below, where the greater the number P of poses 107A, the more time is generally required to train the model at block 120.

[0098] In some embodiments, determining the computation parameters at block 110 includes determining how many instructions the shader can execute for each rendered frame. This determination may include selecting the number P of high-resolution textures that can be interpolated and rendered during real-time computation given a computational budget. In some embodiments, this involves taking into account the specific details of the shader software being used.

[0099] Block diagram 110 generates an output of calculation parameters 113. Given the calculation budget of a specific running scene, calculation parameters 113 may include a low-resolution weight texture resolution and the number P of training poses 107A for interpolation calculation. Optionally, when method 100 is used as part of real-time rendering, calculation parameters 113 may include certain indications of the required frame rate and / or the number of instructions or operations that can be executed per frame. In some embodiments, the determination of the calculation parameters at block diagram 110 can be performed by a technician (possibly with the assistance of some computer-based systems). The technician may be a shader coder who determines the maximum number of texture operations (texture reading, interpolation, etc.) that can be input to the shader based on the required frame rate. In some embodiments, the determination at block diagram 110 is additionally or alternatively performed by a software algorithm, which is optionally capable of receiving a set of guidance instructions. Calculation parameters 113 may also include the resolution of the high-resolution texture uploaded to the GPU. For example, a reduced version of the high-resolution texture from the training pose 107 can be provided for use by less complex processing hardware.

[0100] The selection of the desired frame rate, the resolution of the low-resolution weight texture map 575, and the number P of poses 107A used for interpolation are interdependent, taking into account the computational budget. The higher any of these parameters is, the more computational budget is typically consumed. For example, if a higher frame rate is required, the resolution of the low-resolution weight texture map 575 and the number P of selected training poses 107A must typically be correspondingly lower if computational resources are limited. Similarly, if a higher resolution low-resolution weight texture map 575 is required or a greater number P of training poses 107A are used, the other parameters in the computational parameters 113 must typically be correspondingly lower if computational resources are limited.

[0101] After blocks 105 and 110 are completed, method 100 proceeds to block 115, which involves determining a neutral pose mesh and a plurality of feature maps 117. The number of feature maps determined in block 115 may correspond to the number P of selected training poses 107A. As used and explained in more detail herein, a "neutral pose mesh" is an example of a "feature map geometry" and is a specific feature map geometry corresponding to a "neutral pose" selected from the P selected training poses 107A. The neutral pose mesh provides a representation of facial geometry from which a "feature map" may be defined. The "feature map geometry" includes a low-resolution mesh representation of facial geometry for a training pose 107A (e.g., a specific one of the P selected training poses 107A) or other poses (e.g., a target facial expression 503 described in more detail below), including W vertices (which may be referred to herein as handle vertices) sparsely distributed around the face, with edges (having a defined edge length) connecting adjacent vertices. The edges of the feature map geometry may also be referred to herein as feature edges. Feature map geometry may also be referred to herein as low-resolution geometry.

[0102] A "feature map" corresponding to a training posture 107A (e.g., one of P selected training postures 107A) is a representation of the corresponding training posture 107A, which includes a set of primitive parameters on the feature map geometry of the corresponding training posture 107A, which primitive parameters can be based on features of the corresponding training posture 107A and the neutral posture mesh. In some embodiments, the primitive parameters include "edge parameters" or "edge features", where each edge parameter can be determined based on the edge length of the feature map geometry of the corresponding training posture 107A and the edge length of the corresponding edge of the neutral posture mesh. In some such embodiments, each edge parameter of the feature map of the training posture includes an edge strain, which includes the ratio of the edge length of the feature map geometry of the corresponding training posture 107A to the edge length of the corresponding edge of the neutral posture mesh. Such an edge parameter can indicate whether the edge of the feature map geometry corresponding to a particular training posture 107A is stretched or compressed compared to the corresponding edge of the neutral posture mesh.

[0103] In some embodiments, each primitive parameter of the feature map of the training pose includes a deformation gradient or other derivable parameter, which can be based on other additional or alternative features of the corresponding training pose 107A and the neutral pose mesh. By way of non-limiting example, these other additional or alternative features can include pyramid coordinates (e.g., as described in Sheffer, Alla & Kraevoy, V. (2004). Pyramid coordinates for morphing and deformation. 68–75. 10.1109 / TDPVT. 2004.1335149, which is incorporated herein by reference), linear rotation invariant coordinates (e.g., as described in Lipman, Yaron & Sorkine, Olga & Levin, David & Cohen-Or, Daniel. (2005). Linear Rotation-Invariant Coordinates for Meshes. ACM Trans. Graph. 24. 479-487. 10.1145 / 1073204.1073217, which is incorporated herein by reference), etc. The remainder of this specification may describe the primitive parameters of the feature graph as "edge parameters" or "edge properties", but without loss of generality, these edge parameters may additionally or alternatively include other primitive parameters, which may be based on, for example, triangle parameters, 1-ring neighborhood parameters, etc. Unless the context clearly dictates otherwise, references to edge parameters herein should be understood to include the possibility that such edge parameters may include additional or alternative primitive parameters.

[0104] Figure 2A Schematically depicted is a method 200 that can be used to determine a feature map of P selected training poses 107A. The method 200 receives as input the training poses 107 obtained from block 105 of the method 100 and the number P of training poses 107 to be used. The number P of training poses 107 to be used in the method 200 can be provided as part of the calculation parameters 113.

[0105] At block 210, the method 200 selects P poses from the input set of high-resolution facial training poses 107. In the currently preferred embodiment, the selected P poses 107A represent extreme facial expressions, i.e., the facial expressions of the selected P training poses 107A are significantly different from each other. This can be achieved in a variety of ways, including but not limited to: · The selection of which training postures 107A should be combined with other training postures based on the subjective opinions of trained technicians. Practice posture 107A with the greatest difference or a sufficiently large difference; · Through computer software-assisted selection, the cumulative difference values ​​between various combinations of input training postures 107 are analyzed and selected. Select P poses with the highest or sufficiently high cumulative difference; and • Selection is made by comparing the consistency of different input training poses 107 with various standard FACS poses. In some embodiments, the number of training poses 107A obtained at block 105 of method 100 is consistent with the number of P poses 107A determined from a previous execution of block 110. In other words, the training data 107 can be obtained according to the specific needs and limitations of the computing environment. As an illustrative example, where the required number of poses determined by block 110 is P=6, the number of training poses 107A obtained at block 105 can be 6 plus an optional number of buffer poses 107A to account for errors and / or perform calibration. This approach may be advantageous when the process of obtaining training poses 107 (e.g., in block 105) is expensive and / or time-consuming.

[0106] The selection of P poses at block 210 results in P sets of facial textures and geometries 213. The P sets of facial textures and geometries 213 are directly derived from the training poses 107 and include various sets of high-resolution textures and sets of high-resolution facial geometries for each of the P selected poses. In some embodiments, for each of the P geometry graphs, the P sets of facial textures and geometries 213 may include a high-resolution mesh whose vertex positions define a facial surface geometry graph and a corresponding plurality of 2D high-resolution facial textures. The method 200 proceeds to block 215 where a neutral pose is selected from the P poses selected at block 210. The neutral pose may be selected from the P poses as the pose that is closest to a relaxed expressionless face. In some embodiments, the neutral pose is defined according to the FACS standard for a 'neutral face'. As will be explained below, the feature map geometry (or low-resolution geometry) corresponding to the neutral pose (or neutral pose mesh) selected at block 215 provides the basis for defining the feature maps (and their edge parameters) for the P training poses. Additionally, at block 215 , each of the P training postures is indexed from 1≤k≤P, where k is the index of the particular posture, resulting in a posture index 217 .

[0107] Method 200 continues to block 220 , which includes extracting a neutral pose mesh (ie, a low-resolution feature map geometry corresponding to the neutral pose selected at block 215 ) and feature maps for the P training poses, thereby producing P feature maps and a neutral pose mesh 117 .

[0108] Figure 2BA method 250 is schematically illustrated that can be used to extract P feature maps and the neutral pose mesh 117 in block 220 of method 200 according to an example embodiment of the present invention. Method 250 involves using edge strains as edge parameters of the P feature maps (e.g., for associating edges of the feature map geometry of each of the P training poses with edges of the neutral pose mesh). In some embodiments, as described above, the feature map determined at block 220 is determined based on other primitive parameters that can supplement or replace edge strains. Examples of such other primitive parameters include, but are not limited to, calculated deformation gradients and pyramid coordinates. In some embodiments, in addition to or as an alternative to feature edges, a triangle mesh and / or 1-ring neighborhood of vertices can be used to compare the feature map geometry of the P selected training poses with the feature map geometry of the neutral pose.

[0109] Method 250 receives as input the P high-resolution texture and geometry groups 213 and the pose index 217 determined at blocks 210 and 215 of method 200. At block 255, method 250 defines a neutral pose mesh 257. As discussed elsewhere herein, the neutral pose mesh 257 is a feature graphic geometry (low-resolution mesh) corresponding to the neutral pose selected in block 215. Determining the neutral pose mesh 257 at block 255 first includes defining a plurality of W sparsely distributed handle vertices Hw, where w∈[1,2,…W] corresponds to a subset of the V vertices of the high-resolution mesh topology 107 of the training pose, i.e., W handle vertices are selected from the V high-resolution vertices of the mesh topology 107 of the training pose, and each handle vertex Hw corresponds to one of the V high-resolution vertices. These sparsely located vertices Hw may be assigned a numerical index in the range of 1≤w≤W. The adjacent pairs of edges connecting these low-resolution handle vertices Hw roughly correspond to the shape of the face and can represent various facial features and / or expressions accordingly. The adjacent pairs of edges connecting these handle vertices Hw (which can be called feature edges) can be assigned corresponding edge lengths according to the geometric shapes of the handle vertices Hw.

[0110] There are a number of possible ways to determine the neutral pose mesh 257 at block 255. For example, the neutral pose (selected from the P training poses 107A selected at block 215) may be segmented into a rectangular map of handle vertices Hw evenly distributed in the UV space of the original high-resolution neutral pose geometry selected at block 215. This approach has the advantage of being automatically applied and requiring little computation or manipulation since the rectangular map is applied indiscriminately to the face. In other embodiments, the vertices Hw used to determine the neutral pose mesh 257 are defined at block 255 with reference to reference points (vertices) located on the face to best capture how the edges of the face stretch and compress. These reference points (vertices) are typically located in areas around the nose, eyes, mouth, etc.

[0111] In some embodiments, the determination of the neutral posture grid 257 can be performed using an automatic face detection module and / or a manual selection procedure. The advantage of using reference points (vertices) for determination is that the vertices and edges can be adjusted based on the anatomical understanding of different facial elements. For example, it is known that the nose does not deform significantly under different facial expressions. Therefore, a smaller number of reference points (vertices) can be assigned to the nose area. In contrast, the forehead undergoes a large change in different facial expressions, and in some embodiments, a larger number of reference points (vertices) and edges connecting these points (vertices) can be assigned.

[0112] Figure 3A An example of a marker layout for capturing a digital representation of an actor's face is shown. As shown, multiple markers are distributed at various locations on the actor's face, where the distribution can be based on any of the methods described herein. Figure 3A The marker positions shown in can be used for the locations of the reference points (vertices) of the low-resolution (ie, feature map) geometry and the corresponding handle vertices Hw. Figure 3A Also shown are multiple edges connecting adjacent vertices. Figure 3A The number of edges shown is relatively low to simplify Figure 3A In practice, additional edges may be provided between other adjacent vertices (e.g., based on some suitable geometric definition of adjacency). Figure 3B shows an example representation of a low-resolution (i.e., feature map) geometric graph containing facial expressions that can be captured using edge connections (e.g. Figure 3A The complete edges are obtained by using the sparse labels / vertices Hw of the facial expressions in . Figure 3C Shown with Figure 3A and 3B An example representation of a corresponding high resolution facial expression, which may form part of the P face texture and geometry set 213 .

[0113] In some embodiments, vertices Hw of neutral pose mesh 257 are defined as part of the data acquisition process at block 105 of method 100. For example, vertices Hw of neutral pose mesh 257 may be defined by and correspond to motion capture markers on the face of an actor during execution of block 105 (where P training poses 107A are captured). In some embodiments, definition of neutral pose mesh 257 at block 255 includes the use of motion capture markers, and optionally supplementing virtually defined vertices located therebetween. In some embodiments, the resolution of vertices Hw, and therefore the accuracy of the facial features being tracked, may be set as part of determining the calculation parameters at block 110.

[0114] The resolution of the vertices Hw of the low-resolution mesh described herein is preferably selected to capture sufficient information to describe the current pose of the face in any given area. In some embodiments, it may be preferred to have a relatively high concentration of vertices Hw in locations where the texture and / or geometry is expected to have a relatively high degree of local variation (e.g., at the lips). This enables high-resolution weighted texture interpolation to distinguish finer skin deformations between the P training poses 107A. Conversely, in areas where the texture is expected to be relatively consistent (e.g., at the cheeks), relatively few vertices Hw may be provided to represent these areas. As a non-limiting illustrative example embodiment, the number of vertices Hw defining the geometry of the low-resolution mesh may be in the range of approximately 50-300. According to a more specific example, 100-200 vertices Hw define the geometry of the low-resolution mesh.

[0115] return Figure 2B In the method 250 of FIG. 250 , at block 260 , feature map geometries (also referred to as low-resolution meshes) for the P training poses are defined. The vertices Hw used in defining the feature map geometries for the training poses in block 260 are the same as the vertices Hw used in defining the neutral pose mesh 257 in block 255 . This allows a correspondence between the low-resolution mesh representations of each P training pose 107A relative to the neutral pose mesh 257 to be established (i.e., the representations have the same topology), so that the characteristics (e.g., edge lengths and / or other edge characteristics and / or other primitive parameters) of the low-resolution mesh (feature map geometry) corresponding to each P training pose 107A can be easily compared with the corresponding characteristics of the neutral pose mesh 257 (i.e., the feature map geometry for the neutral pose selected in block 215 ). Therefore, the vertex indices w∈[1,2,…W] of the low-resolution mesh (feature map geometry) for the P training poses correspond to the vertex indices in the neutral pose mesh 257.

[0116] After the low-resolution meshes (feature map geometries) of the P training poses 107A are defined at blocks 255 and 260, the method 250 proceeds to block 265, where a feature map 267 is calculated for each P training pose 107A based on features of the low-resolution mesh of block 260 for each P training pose 107A and corresponding features of the neutral pose mesh 257. As described above, each block 260 low-resolution mesh (feature map geometry) generally follows the contour of the corresponding training pose 107A, and the feature map 267 can be determined based on corresponding edge features (e.g., lengths of corresponding edges) in relation to the neutral pose mesh 257. The edge lengths in the low-resolution mesh representations (feature map geometries) in the block 260 of the P training poses can be used to associate the skin strain of the training pose with the neutral pose, and the feature map 267 is determined by comparing with corresponding edges of the neutral pose mesh 257 (e.g., taking the ratio of the edge lengths).

[0117] The skin strain of each of the F feature edges in the low-resolution grid (feature map geometry) of the block diagram 260 of the P training postures 107A can be represented as an F-dimensional feature vector f = [f1 ... f F ] T , where f i is the i th The relative stretch (also referred to herein as strain) of the feature edges, and the vector f may be referred to herein as a feature graph. In some embodiments, f i The definition is as follows: f i =(||p i,1 -p i,2 ||-l i ) / l i (3) where p i,1 and p i,2 corresponds to f i The position vectors of the endpoints of (e.g., a pair of corresponding vertices Hw in the corresponding feature map geometry), and l i is the corresponding i-th th The static length of the edge (e.g., the length between the same two vertices Hw in the neutral pose mesh 257). According to the sign in equation (1), the eigenvector value f with a negative value i represents the compression relative to the neutral pose mesh 257, and the eigenvector value f with a positive value i represents a stretch relative to the neutral posture mesh 257. Execution of block 265 generates a training posture feature map 267 for each of the P training postures 107A.

[0118] In some embodiments, other additional or alternative primitive parameters of the low-resolution mesh of block diagram 260 (feature map geometry) of P training poses 107A and corresponding parameters of neutral pose mesh 257 may be used to determine feature map 267. Non-limiting examples of such additional or alternative primitive parameters include deformation gradients and pyramid coordinates.

[0119] At its conclusion, method 250 returns a neutral pose mesh 257 and P feature maps 267 corresponding to the P selected poses 107A. Figure 2B The neutral pose grid 257 and P feature maps 267 shown in FIG. 2 together represent Figure 1 and 2A The P feature maps and neutral pose grid 117 shown in .

[0120] return Figure 1The method 100 of the embodiment of the present invention proceeds to blocks 120 and 125, where an approximation model 123 is trained and a rasterization calculation is performed to determine a rasterization matrix 127, respectively. According to a currently preferred embodiment, training the approximation model 123 at block 120 includes defining a set of radial basis functions (RBFs) (e.g., one per vertex) by which the training data can be interpolated and fitted to the model. Execution of block 120 results in an approximation model 123, such as the WPSD model 340 described below, which allows the determination of a correspondence between a target facial expression (target pose) and a plurality of P training poses 107A.

[0121] Bickel et al., in their paper “Pose-Space Animation and Facial Detail Transfer” presented at the ACM SIGGRAPH Symposium on Computer Animation in January 2008 (incorporated herein by reference), disclose a technique called weighted pose-space deformation (WPSD) for modeling the relationship between local skin strain and corresponding vertex displacements. When training the WPSD model, Bickel et al. employ a fine-detail correction d of a warped representation of a neutral pose relative to each training pose. The correction d contains a vector of size 3V, where each Cartesian coordinate (x, y, z) of each high-resolution vertex v has a displacement relative to the warped representation of the neutral pose, and V and v (in the context of training the block diagram 120 of the approximate model 123) represent the total number V of high-resolution vertices and the index v∈[1,2,…V] of each vertex for the selected P training poses 107A, respectively. The correction displacement d disclosed by Bickel is then represented in the set of RBFs trained on the P training poses 107A.

[0122] Some embodiments of the present invention utilize techniques similar to those disclosed by Bickel et al., except that, instead of applying WPSD in the context of calculating vertex displacements, embodiments of the present invention use the WPSD technique used in block diagram 120 to determine each high-resolution vertex texture interpolation weight. In other words, the present technique discloses determining each high-resolution vertex texture interpolation weight that represents the similarity of the target facial expression (target pose) 503 to the P training poses 107A rather than the absolute displacement. Such interpolation weights can be used to interpolate the high-resolution texture in a subsequent step, as described later in this document.

[0123] Figure 4 Schematically illustrates a block diagram 120 ( Figure 1) to develop and train the approximate model 123. In other embodiments, the training technique for generating the approximate model 123 may be implemented in the execution of block diagram 120 using other suitable techniques for solving the problem known as "sparse data interpolation". Non-limiting examples of such methods include least squares polynomial regression, k-nearest neighbor interpolation, weighted interpolation methods, and / or the like. Figure 4 In the illustrated embodiment, method 300 depicts a number of steps to obtain an approximate model 340. Approximate model 340 represents a Figure 1 An example of an approximate model 123 trained in block diagram 120 of .

[0124] Method 300 receives the data transmitted by executing method 200 ( Figure 2A ) and possible methods 250( Figure 2B ) obtained P texture and geometry groups 213 (see Figure 2A ) and P feature maps 267 and neutral pose grid 257 (P feature maps and neutral pose grid 117, see together Figure 1 and Figure 2B ) as input. The method 300 includes a loop (implemented by blocks 305 and 335) involving performing a plurality of steps for each high-resolution vertex v (indexed from 1≤v≤V). At block 305, an initial vertex v1 is selected from a complete set of high-resolution vertices V (which may be defined in each input geometry of the P input geometry groups 213) for training the approximation model 340. Similar to the handle vertices Hw used to define the P feature maps 267, each high-resolution vertex v corresponds to the same high-resolution vertex on different P high-resolution training poses 107A (the high-resolution training poses 107A have the same topology) and may be indexed from 1≤v≤V. Figure 4 In the illustrated embodiment, the interpolation problem can be solved independently at each high-resolution vertex v, and the approximation model 340 represents a set of trained RBFs, one for each high-resolution vertex v, and VRBFs in total. That is, for example, the solution for vertices located on the cheek is obtained independently of the solution for vertices located on the nose. In other embodiments, the loop implemented by blocks 305 and 335 is performed only for certain high-resolution vertices v (i.e., a subset of the total number of high-resolution vertices V, which may correspond to the union of the vertices determined in block 415), as further described herein.

[0125] At block 307, a vertex proximity mask 309 is calculated using the high-resolution neutral pose (which is one of the P geometry groups 213) and the neutral pose mesh 257. The vertex proximity mask 309 is used to assign relative weights based on the proximity of the high-resolution vertex v to different feature edges. According to an example embodiment, the vertex proximity mask 309 is calculated at block 307. th The i-th vertex th The weight of feature edges is α v,i , and can take the following forms: Where: has the same meaning as discussed above with respect to equation (1) (i.e., the i-th position of the neutral posture grid 257 th The static length of the low-resolution feature edge), and L v,i is the i-th path from the high-resolution vertex v to the neutral pose mesh 257 th The sum of the neutral pose distances of the endpoint positions of the low-resolution feature edges. For example, these distances can be Euclidean distances or geodesic distances. In some embodiments, for the edges around the current vertex, α v,i is 1, and decays elsewhere. The parameter β in equation (2) is a configurable scalar parameter that can be used to adjust the decay rate.

[0126] In some embodiments, when determining the weight α v,i , then, block 307 may further include calculating the edge strain value f of each characteristic edge of the P training postures 107A i (See formula (1)) multiplied by the neighbor weight α v,i To obtain the vertex proximity mask 309 of the current vertex v. In some embodiments, for a specific high-resolution vertex v, the low-resolution feature edge has a weight α v,i , which is less than some suitable, optionally user-configurable threshold (e.g., 0.0025), will be omitted in subsequent steps of method 300. According to certain example embodiments, in execution of method 300, vertex v will be affected by at most some suitable, optionally user-configurable threshold number (e.g., 16) of low-resolution feature edges. By limiting the number of feature edges considered, the computational complexity in execution of method 300 can be advantageously reduced by requiring each high-resolution vertex v to consider fewer low-resolution feature edges and to consider the strains that have the greatest impact on that particular vertex v. In other embodiments, all low-resolution feature edges are considered in the computation of each high-resolution vertex v, regardless of their proximity to the high-resolution vertex v. Execution of block diagram 307 on the set of high-resolution vertices V produces a vertex proximity mask 309 that contains each feature edge f for each high-resolution vertex v. i The weight α v,i ,.

[0127] After completing block 307, method 300 proceeds to subprocess 310, where a radial basis function representing the interpolation problem of the current vertex v is trained. Execution of subprocess 310 may involve using equations of the following general form: d=wφ (3) Where: the matrix d has dimensions P x P and has columns defining the target mixture weights to be learned by the RBF during training of the block diagram 310, the matrix w has dimensions P x P and represents the unknown variables of the RBF weights that the training subprocess 310 aims to solve for, and φ is a P x P dimensional RBF kernel that represents the distance of a particular vertex v in the training pose 107A to all other selected training poses 107A based on the relative strain of the characteristic edges around the particular vertex v or any other suitable similarity measure. For example, φ may represent the similarity of the characteristic map around a particular vertex v in each of the P training poses 107A to all the P training poses 107A measured by the L2 norm, which describes how similar the strain of the surrounding characteristic edges is to the P training poses.

[0128] Block 313 involves creating a P x P dimensional identity matrix d, where in addition to d 1,1 ,d 2,2 …d P,P All values ​​except (value 1) are zero. The identity matrix d represents the interpolation (mixing) weights to be learned during training in block diagram 310, where a set of RBF weights w is required so that the calculation on the right side of equation (3) achieves a similarity ratio of 1 for each of the P training poses with respect to itself and a similarity ratio of 0 with respect to all other poses.

[0129] In some embodiments, the RBF training technique of block 310 includes determining the distance in a weighted manner for each vertex, where the distance matrix (i.e., φ in equation (3) for the current vertex v) may be referred to as φ v ) adopts the form of distance-weighted RBF, where the kth th Column and l th The value of the row (φ v,k,l ) can be expressed as: Where: γ is the RBF kernel function (eg, a biharmonic RBF kernel in some embodiments); f k,i is the strain (kth th Training posture i th The feature map value determined by equation (1) is f l,i It is the first th Training posture i th edge according to the feature map value determined by equation (1), αv,i is assigned to i based on its proximity to the associated high-resolution vertex v th The weights of , which are obtained in block 307 and included in the proximity mask 309 by equation (2). The use of the proximity mask 309 exploits the fact that the feature edges f1…f F The relative stretching of has the characteristics of different proximity to the associated high-resolution vertex v. Therefore, using a distance-weighted RBF can effectively capture the effect of the attenuated influence of feature edges far away from vertex v, while allowing the feature edges closest to the high-resolution vertex v to have a higher degree of influence. As discussed above with respect to block diagram 307, when considering how to use weight α in method 300 v,i and feature edges, there are several options.

[0130] Equation (3) represents the basic linear equations of a typical RBF, which can be solved by matrix inversion in the form of: w=dφ -1 (6) At block 330, equation (6) is calculated to obtain a P x P weight matrix w, which represents the trained RBF weights 333 for the current vertex v, so that inference can be performed using the matrix w (e.g., via a dot product, as explained in more detail below). The matrix w (trained RBF weights 333) allows a target pose (target facial expression 503, described in more detail below) to be represented as a function of its similarity to the P training poses at a particular vertex v.

[0131] After completing block diagram 330 and subprocess 310, method 300 proceeds to decision block diagram 335, which evaluates whether there are any remaining high-resolution vertices v for which RBF weights 333 (matrix w) have not yet been calculated. If the query at block diagram 335 is affirmative, method 300 performs subprocess 310 for subsequent vertices v according to the steps described above. If the query at block diagram 335 is negative, method 300 ends. Execution of method 300 produces an approximation model 123, 340, which includes a set of RBF weights 333 (matrix w) for each high-resolution vertex v. The approximation model 123, 340 may include a tensor with dimensions [V, P, P], i.e., a Px P weight matrix w (RBF weights 333) for each vertex v∈{1,2,3…V}. For each vertex v∈{1,2,3…V}, the approximation model 123, 340 allows the target pose to be represented as a function of its similarity to the P training poses at the vertex v.

[0132] Although in the execution block diagram 120 ( Figure 1 ) is discussed in the context of determining an approximate model 123 ( Figure 4), but it will be appreciated that other additional or alternative suitable optimization or interpolation methods may be applied in block diagram 120 or method 300 to determine similarity weights from a sparse sample set. For example, when there are a large number of high-resolution vertices and / or a large number of training poses, further approximation or optimization techniques may be performed, such as k-means clustering, Lloyd algorithm, fitting multiple training poses in a least squares manner, etc. In some embodiments, such optimization (or portions of such optimization) may optionally be performed as a preprocessing step prior to execution of method 300.

[0133] The use of radial basis functions (as in method 300) in conjunction with a "sparse interpolation problem" is best suited for sparse interpolation problems where the number of training samples to be interpolated is relatively small. Based on current computing power, the sparse interpolation of method 300 (using radial basis functions) may be suitable for interpolating up to about P = 20 training poses. In some embodiments, when the number of training poses P exceeds 20 or computing resources are limited, different forms of machine learning (e.g., using neural networks) may be suitable for solving the sparse interpolation problem.

[0134] Return method 100( Figure 1 ), rasterization calculations are performed at block 125. As briefly discussed above, method 100 includes interpolating a high-resolution texture based on a low-resolution weight texture map 575. Execution of block 125 results in a rasterization matrix 127 that contains a mapping provided for each low-resolution pixel n (n∈1,2…N) in the weight texture map 575, where the high-resolution vertex v and the corresponding barycentric coordinates are used to convert each high-resolution vertex approximation model 123 into each low-resolution pixel weight texture r n (567) As described above, the resolution of the low resolution weight texture map 575 may be provided by the calculation parameters 113 determined at block 110.

[0135] Figure 5 Schematically illustrates a method 400 for rasterization calculation at block 125 of method 100 according to a particular embodiment. The rasterization calculation of method 400 depicts a plurality of steps for obtaining a rasterization matrix 430, which is consistent with method 100 ( Figure 1). The method 400 is provided with a plurality of inputs 407. In the illustrated embodiment, the inputs 407 include a UV representation 403 of the topology of the high-resolution training pose 107 and the computation parameters 113. The UV representation 403 includes an "unfolded" 2D description of the topology of the high-resolution training pose 107. At block 410, the method 400 includes mapping pixels of the low-resolution texture map 575 into UV space to obtain a UV map 450. Conceptually, the block 410 can be considered to involve creating a notional grid 455 of cells in UV space. The dimensions of the notional UV space cells 460 in the grid 455 of the block 410 correspond to the resolution of the low-resolution weight texture map 575 (which can be part of the computation parameters 113 determined in the block 110), and the UV coordinates of the pixels in the low-resolution weight texture map 575 are mapped to the centers of the cells 460 of the grid 455 of the block 410. Block diagram 410 may be understood as calculating normalized UV coordinates for pixels in the low-resolution weight texture map 575 .

[0136] The UV coordinate system (or UV space) includes normalized coordinates ranging from 0 to 1 in a pair of orthogonal directions U and V (i.e., [0,0] to [1,1]). In general, mapping a 3D object, such as a face (usually represented by a 3D mesh), to a 2D UV space is a well-known technique in the field of computer graphics for texture mapping. Each set of 2D coordinates (UV coordinates) in the UV domain uniquely identifies a location in the 3D surface (face) of the object. In addition, the input textures (e.g., P texture groups 213) associated with the training pose 107 have a unique mapping from texture pixel coordinates (texels) to corresponding UV coordinates. As discussed herein, the textures can represent different colors or different surface properties required to render a realistic face.

[0137] Each cell in the nominal cell grid 455 of block diagram 410 may have a constant step size. According to an exemplary embodiment, a particular cell (in the nominal grid 455 of block diagram 410) located at a particular row and column index has a step size in the U and V dimensions, respectively. and As described above, the resolution of the nominal grid 455 matches the resolution of the weight texture map 575 so that pixels of the weight texture map 575 are mapped to the centers 470 of corresponding cells 460 of the nominal grid 455. Representation of the 3D face mesh in UV space is desirable because graphics engines are typically configured to process texture data received in the form of texture data mapped to UV space. In other embodiments, projection mapping techniques may be employed in addition or alternatively at block diagram 410. In practicing various embodiments of the present invention, other texture mapping systems besides the UV mapping described above are possible. Another suitable example technique is Ptex developed by Walt Disney Animation Studios. TM system.

[0138] Fig. 6A An illustrative example of a UV coordinate system ranging from (0,0) to (1,1) and a nominal low-resolution grid 455 superimposed on the UV coordinate system is shown. The nominal low-resolution grid 455 includes the same Figure 1 ) so that the UV mapping 450 of the pixels of the low-resolution weight texture map 575 is mapped to the center 470 of the cell 460 of the nominal grid 455. Fig. 6A The 3D topology of the mesh associated with the input gesture 107 is also schematically depicted. Fig. 6A The illustrative example of shows a nominal low resolution grid 455 of 16x16 cells in UV space (corresponding to block diagram 110 ( Figure 1 ), but it will be appreciated that this resolution is merely illustrative and that block diagram 410 UV mapping 450 (imaginary grid 455) having other low-resolution weight texture resolutions is possible. Fig. 6A Also explicitly shown are the UV coordinates of a number of exemplary high-resolution vertices 465 (corresponding to vertices in the topology of the input training pose 107), with it being understood that there are many more such vertices.

[0139] return Figure 5 , the method 400 proceeds to block 415. Block 415 involves identifying the center 470 of each cell 460 of the nominal mesh 455 of block 410 (which corresponds to the UV space coordinate of a pixel of the weight texture map 575), and then for each UV cell 460 (each pixel of the weight texture map 575), determining a high-resolution vertex 465 in the set of high-resolution vertices V that forms the closest triangle (or other polygon) 475 around the center 470 of the cell 460. Fig. 6AAn illustrative example of such a block 415 determination is shown. In the example shown, since the UV coordinate system includes the range [0,0] to [1,1], the location of the center 470 of the UV cell 460 (which corresponds to the UV space coordinate of the pixel of the weight texture map 575) can be represented as a floating point number, and the selected high-resolution vertices 465 that form the nearest triangle can be identified by their vertex indices and corresponding floating point UV coordinates. In some embodiments, multiple vertices 465 can be selected in block 415 for each UV cell 460 (each pixel of the weight texture map 575), which form polygonal shapes other than triangles.

[0140] exist Fig. 6A In the example shown, example UV cell 460A is the 12th cell in the U direction and the 15th cell in the V direction of nominal grid 455, depicting an area around the forehead of a face. UV cell 460A has a center 470A. Figure 6B A schematic detailed view of a cell 460A is shown, including three closest vertices 465A, 465B, and 465C, forming an example of a triangle 475A around a center 470A. The indices of the three vertices 465A, 465B, and 465C are identified and recorded at block 415 as the vertices 465 (or corresponding pixels of the weight texture map 575) that form the closest triangle 475A around the center 470A of the UV cell 460A. Figure 6B In the example shown, vertex 465D, while located in UV cell 460A, does not form one of the closest vertices of triangle 475A around center 470A and therefore is not recorded in block diagram 415 as corresponding to UV cell 460A in the rasterization calculations of method 400 .

[0141] The process of block 415 is performed for each UV cell 460 in the nominal grid 455 of block 410. Fig. 6A In the example of , this process is performed for 256 different UV cells 460 in the nominal grid 455 of 16x16 UV cells. In embodiments where the number of high-resolution vertices V defining the geometry of the training pose 107A is high, there are many vertices that are not considered when executing the method 400 (e.g., Figure 6B465D in the example. In contrast, in embodiments where the number of vertices V defining the geometry of training pose 107A is low, a single high-resolution vertex 465 may be part of the closest triangle around the center of two or more individual UV cells 460 (which may correspond to two or more pixels of weight texture map 575). As discussed in more detail below, such a vertex 465 corresponding to multiple UV cells 460 (multiple pixels of weight texture map 575) may have different barycentric coordinates for each UV cell 460 (and different barycentric coordinates for each corresponding pixel of weight texture map 575).

[0142] After identifying the vertex 465 corresponding to the center 470 of the UV cell 460 (i.e., corresponding to the pixel of the weight texture map 575) at block 415, the method 400 ( Figure 5 ) proceeds to block 425, which involves determining, for each UV cell 460, the barycentric coordinates 427 of the corresponding UV cell center 470 (the pixel of the corresponding weight texture map 575) relative to the nearest triangle vertex 465 (as identified at block 415). The barycentric coordinates 427 determined at block 425 include a parameterization of the UV cell 460 (more specifically, the center 470 of the UV cell 460 or the pixel of the weight texture map 575) relative to the high-resolution pixel identified in block 415. For each UV cell 460 (each pixel of the weight texture map 575), block 425 may involve determining three barycentric coordinates λ1, λ2, and λ3, which sum to one, describing the location of the center 470 of the UV cell 460 or the pixel of the weight texture map 575 relative to the location of the high-resolution vertex 465 identified (in block 415) as corresponding to the UV cell 465 or the corresponding pixel of the weight texture map 575. Figure 6B An example is shown where UV cell 460A may have barycentric coordinates 427 (0.2, 0.2, 0.6) that describe the relationship between center 470A of UV cell 460A relative to vertices 465A, 465B, and 465C that form triangle 475A.

[0143] Rasterization Matrix 430 (Forming Method 400 ( Figure 5The output of the rasterization matrix 430 may include the barycentric coordinates 427 (determined in block 425) of all UV cells 460 of the nominal mesh 455 (or equivalently, all pixels of the weight texture map 575) and the corresponding vertex indices of the corresponding triangles and the indices of the UV cells 460 (or the indices of the pixels of the weight texture map 575). The barycentric coordinate portion of the rasterization matrix 430 may include a tensor having dimensions [W, H, 3], where W and H are the number of cells 460 in the U and V dimensions of the nominal mesh 455 of block 410 (e.g., which may correspond to the resolution of the low-resolution weight texture map 575 determined in block 110 and included in the calculation parameters 113), and 3 is the number of barycentric coordinates.

[0144] It should be understood that various modifications, additions and / or substitutions may be made when performing rasterization calculations in block diagram 125 and method 400. Exemplary additional or alternative embodiments include: Defining high-resolution vertices that correspond precisely to the UV coordinates of pixels of the weight texture map 575 (i.e., corresponding to the centers 470 of the cells 460 of the nominal mesh 455, rather than identifying triangle (or other polygon) vertices and calculating barycentric coordinates); Rather than computing the barycentric coordinates of the high-resolution vertices of the triangle (or other polygon) that forms the UV coordinates of the pixels of the weight texture map 575 (i.e., surrounding the center 470 of the cell 460 of the nominal mesh 455), the high-resolution vertex that is closest to the UV coordinates of the pixels of the weight texture map 575 (i.e., the vertex closest to the center 470 of the cell 460) is simply selected, and the WPSD weight (w) for the training pose 107 at that single high-resolution vertex is used for other aspects of the method 100; Use some technique other than barycentric coordinates to obtain the relationship between the high-resolution vertex 465 and the center 470 of the UV cell 460 (i.e., between the high-resolution vertex 465 and the UV coordinates of the pixels of the weight texture map 575). For example, parameters similar to barycentric coordinates can be determined for other polygons.

[0145] Return to method 100 ( Figure 1), after obtaining the approximate model 123 at block 120 and the rasterized matrix 127 at block 125, the method 100 proceeds to block 130, which involves rendering the target facial expression 503 to obtain a rendered facial image 585, wherein such block 130 rendering includes applying (e.g., interpolating) the high-resolution textures from the P training poses 107A to the target facial expression (target pose) 503. The target pose 503 can represent a frame of an animation. Prior to block 130 (e.g., in the training portion of the method 100 identified by dashed line 133), the method 100 includes steps that can be performed as part of a series of pre-processing or offline steps. Some embodiments of the present invention include multiple processing steps in block diagram 130, which can be performed in a real-time computing environment, that is, block diagram 130 can be optionally performed while digitally rendering the target facial expression (target pose) 503 as a continuous rendering frame 585 in an animation (e.g., within the frame rate of the corresponding animation) to provide a rendering 585 with highly detailed and animated textures. In a non-limiting example embodiment, block diagram 130 is performed to render a real-time digital 3D computer graphics (CG) avatar (also known as a CG character). For example, such a CG avatar can be applied to virtual reality, video games, etc. In other embodiments, block diagram 130 can be performed in an offline computing environment. A non-limiting example of an offline performance that can use block diagram 130 is to render a CG character in a feature film project.

[0146] Figure 7 Schematically, a method 500 is shown, which can be used for animation of high-resolution textures based on facial expressions, applying the high-resolution texture to a target facial expression (target pose) 503 by dynamic interpolation, and rendering the target facial expression 503 as a rendered facial image 585 in the block diagram 130 of the method 100. The method 500 is provided with a plurality of inputs 501. In the illustrated embodiment, the inputs 501 include: P sets of textures 213 corresponding to P training poses; P feature maps 117 ( Figure 1 )、267( Figure 2B ); Neutral Posture Grid 117 ( Figure 1 )、257( Figure 2B ); Vertex proximity mask 309 ( Figure 4 ); Approximate model 123( Figure 1 )、340( Figure 4 ); Rasterization matrix 127 ( Figure 1 )、430( Figure 5 ); and a target facial expression (target pose) 503. Method 500 may be performed once for each target pose 503 (eg, once per frame in an animation).

[0147] In general, the target facial expression (target pose) 503 includes a representation (e.g., a 3D mesh model or 3D mesh geometry) of a high-resolution facial expression (pose) having the same topology as the training pose 107A, thereby allowing the extraction (or otherwise allowing the determination) of low-resolution handle vertices corresponding to the low-resolution handle vertices used to create the feature map 117. The target facial expression 503 can be obtained in a variety of possible ways and formats, depending on the application of the method 500 (block 130).

[0148] In some embodiments, a high-resolution mesh captured from an actor is used as the target facial expression 503. Such a target facial expression 503 may be used in the block diagram 130 (method 500), for example, in an offline post-processing scene when rendering a CG character in a movie project. In some embodiments, when the target facial expression 503 is from a captured image of an actor, the handle vertices used in the previous steps of the method 100 may not be directly aligned with the captured vertices of the facial expression 503, in which case a direct mapping between the high-resolution target facial expression 503 and the low-resolution handle vertices may be obtained using triangle vertex indices and barycentric coordinates. This practice of mapping vertices obtained from a captured image of an actor to desired vertices is well known in the art of computer animation and rendering. In this case, it is more efficient to derive a low-resolution geometry graph (from which the feature map 117 is obtained) after applying all operations to obtain multiple target facial expressions 503 involved in rendering a particular scene.

[0149] In some embodiments, a target facial expression 503 may be provided to method 500 using a set of parameters that may be used to generate a target high resolution 3D geometry / expression, such as a set of appropriate blend shape weights, blend shape bases, and blend shape neutrals. The use of weighted combinations of blend shapes is often used in real-time facial rendering applications as parameterization of corresponding 3D geometry. In some cases, when the target facial expression 503 is provided in the form of a set of blend shape weights, the low resolution blend shape weights may be obtained by downsampling a high resolution blend shape base (multiple blend shapes), such that the input facial expression blend shape weights 503 (together with the downsampled blend shape base) may be used to obtain the corresponding low resolution mesh geometry (e.g., handle vertices).

[0150] It should be understood that the above examples are merely examples of a variety of suitable techniques that may be used to provide the target facial expression 503, which may be suitable for real-time and offline applications of block diagram 130 (method 500). Any suitable technique may be used to provide the target facial expression 503 as part of method 100 ( Figure 1 ) input and method 500( Figure 7 ) input.

[0151] Figure 7 The method 500 of an embodiment includes an approximation calculation subprocess 505 and a weighted texture calculation subprocess 550 .

[0152] The approximation calculation subprocess 505 begins at block 510. Block 510 includes determining a feature map geometry and corresponding edge properties (e.g., edge strains) and / or other suitable primitive parameters for the target facial expression 503. That is, block 510 involves determining a feature map 513 (referred to herein as the target feature map 513) for the target facial expression 503. Because the target facial expression 503 has the same topology as the training pose 107, there is a correspondence (or mapping) between the handle vertices of the target facial expression 503 and the handle vertices of the neutral pose mesh 257 (feature map geometry). Therefore, in some embodiments, block 510 involves extracting or otherwise determining these low-resolution handle vertices within the target facial expression 503. Extracting or otherwise determining low-resolution handle vertices from the target facial expression 503 in block 510 can be performed according to a variety of different techniques that are appropriate for the particular form of the target facial expression 503 used in method 500. Many examples of extracting low-resolution geometry graphics from high-resolution actor capture and high-resolution blend shape weights are described above. As described above, once the handle vertices are determined, the computation of the target feature map 513 at block 510 may be similar to that described above for block 265 ( Figure 2B ) is calculated as described above, and formula (1) is similarly used to determine the target feature map vector f = [f1…f F ] T .

[0153] The method 500 and the approximation calculation subprocess 505 proceed to block 515, which represents a loop (in conjunction with block 535) with a number of steps performed for each high-resolution vertex v of the target facial expression 503. Block 525 involves interpolating the training pose weights from the approximation model 123, 340 to infer or otherwise determine a blending weight 527 (also referred to herein as an interpolation weight 527) for the current vertex v of the target facial expression 503. The blending weight 527 may be defined based on the individual contributions of the P training poses 107A to the current vertex v of the target facial expression using the approximation model 123, 340, and in the case of the illustrated embodiment, may be defined based on the RBF weight w for the current vertex v contained therein.

[0154] Fig. 7A 525 process of inferring the blending weight 527 of the current vertex v according to a particular embodiment. The method 700 receives input, including: in the block 510 ( Figure 7) in the target feature map 513; in block diagram 307 ( Figure 4 ) in the vertex proximity mask 309 determined in block 265 ( Figure 2B ) in the training posture feature map 267; and in the block diagram 120 / method 300 ( Figure 1 and 4 ) is determined in the approximate model 123, 340. The method 700 begins with block 710, which involves determining a similarity vector φ v,t (715), which is a vector of length P, representing the vth th The similarity between the vertices and each of the P training poses 107A. Recall from above that the variable k is used to index the P training poses 107A (k∈1,2…P), and the similarity vector φ v,t (715) No. k th element This can be determined based on: in: Represents the kth similarity vector th elements, where superscripts v and t represent similarity vectors (715) corresponds to the vth th The high-resolution vertex; γ is the RBF kernel function (eg, a biharmonic RBF kernel in some embodiments); f t,i is the vector f corresponding to the target feature map 513 t The i th element (i.e., the i-th element of the target feature map 513 determined by formula (1) th The value of the characteristic edge); f k,i is the kth th The i-th training posture characteristic diagram 267 of the training posture th element (i.e., the kth element determined by formula (1) th The i-th feature map corresponding to the training posture th edge value); α v,i is assigned to the i-th vertex v based on its proximity to the current high-resolution vertex v (contained in proximity mask 309, e.g., via equation (2)) th The weight of the feature edge; and the index i (i∈1,2,…F) is the index describing the number of edges in each feature graph. It can be seen that Equation (7) is similar to Equation (4), except that the index l in Equation (4) is replaced by the index t in Equation (7), and the index in Equation (7) is fixed and refers to the target pose.

[0155] Once the similarity vector φ is determined v,t(715), method 700 proceeds to block 720, which uses the similarity vector φ v,t (715) and the approximate models 123, 340 to determine the vector t of the blending weight 527 of the current high-resolution vertex v of the target facial expression 503 with a length of P v In particular, the trained RBF weight matrix w corresponding to the current vertex v can be used (333, see Figure 4 ) to determine the vector t of the blending weight 527 of the current vertex v v Block 720 determines the vector t of the blending weight 527 for the current vertex v. v It can be determined according to the following equation: It should be appreciated that although the computation of equation (8) is actually a P scalar multiplication operation, it can be computed relatively quickly (ie, in real time). The output of method 700 (block 525) is a vector t of blend weights 527 for the current high-resolution vertex v. v , which is a set of weights corresponding to the similarity between the target facial expression 503 and each of the P training poses 107A at the current high-resolution vertex v.

[0156] Back to Figure 7 , block 530 may be optionally executed to adjust the mixing weights 527 (t v ) are normalized to ensure that the P final per-vertex interpolation (blending) weights 527 (t v ) is 1. Without loss of generality, assume that the mixing weight or interpolation weight 527 (t v ) is normalized.

[0157] The method 500 then proceeds to a decision block 535, which evaluates whether there are remaining vertices v of the target facial expression 503 for which the approximation calculations of the subprocess 505 will be performed. If the query at block 535 is affirmative, the method 500 increases the index of v and performs blocks 525, 530 for subsequent vertices v of the target facial expression 503 according to the above steps. If the query at block 535 is negative, the subprocess 505 ends, and the output of the approximation subprocess 505 includes a set of per-high-resolution vertex target facial expression (pose) blending weights 540 (i.e., P weights 527 (t for each vertex v of the target facial expression 503) v)). For each high-resolution vertex v of the target facial expression 503, the target facial expression blending weight 540 of each vertex represents the relative proximity of the similarity of the vertex to each P training posture 107A in the vicinity of the vertex v. As described above, the target facial expression weight 540 includes a vector t of length P for each high-resolution vertex v (v=1, 2, 3...V) of the target facial expression 503. v .

[0158] According to some example embodiments, one or more optimizations may be applied when executing subprocess 505 to allow interpolation of a smaller subset of high-resolution vertices during real-time computation. In one particular example embodiment, approximation subprocess 505 only computes blending weights 527 (t v ), it is known that these vertices surround (e.g., are part of a surrounding triangle or other polygon) the center 470 of the low-resolution cell 460 in the nominal mesh 455 (i.e., the vertices surrounding the UV coordinates of the pixels of the weight texture map 575 in the UV map 450; these vertices include the union of the vertices at the N low-resolution pixels of the weight texture map 575 determined in block 415) (see Figure 5 , 6A , 6B). This optimization is possible because once Figure 1 Block diagram 125 / Figure 5 Since the rasterization matrices 127, 430 are determined in the method 400 (preferably as part of a pre-processing step), it is known which high-resolution vertices (vertices surrounding the center 470 of the low-resolution cell 460 in the nominal mesh 455 or vertices surrounding the UV coordinates of the pixels of the weight texture map 575 in the UV map 450 or the vertices of the union of the N low-resolution pixels of the weight texture map 575 determined in the block 415) are important for texture interpolation. Therefore, the loop at the blocks 515-535 can advantageously omit vertices that do not correspond to (e.g., surrounding or otherwise corresponding to) the center 470 of the low-resolution cell 460 or the pixels of the weight texture map 575 or the union of the vertices of the N low-resolution pixels of the weight texture map 575 determined in the block 415 to increase the speed of real-time calculations.

[0159] After the approximation subprocess 505 is completed, the method 500 proceeds to the texture calculation subprocess 550 for determining the weight texture r of the low-resolution weight texture map 575. n As discussed in the context of determining the rasterization matrix 127, 430, the pixels of the low-resolution weighted texture map 575 used in the texture calculation subprocess 550 are mapped to the UV space (in the UV map 450) at the center 470 of the cell 460 of the above-mentioned nominal grid 455 ( Figure 5 ,6A ). Subprocess 550 may be performed as part of a real-time computation or as part of an offline computation. Subprocess 550 begins at block 555, which represents a loop (in conjunction with block 570) for performing a plurality of steps at each pixel n of the low-resolution weight texture map 575 (or each cell 460 of the nominal grid 455), which may be indexed from 1≤n≤N. At block 560, the indices of the high-resolution vertices corresponding to the current low-resolution pixel n are selected. The indices of these high-resolution vertices may be obtained from the input rasterization matrices 127, 430, which in turn may be obtained according to example method 400 ( Figure 5 ) is obtained. Block 560 may further include querying vertex-specific interpolation (blending) weights 527 (t v ) to select the P-element interpolation weight vector (t v As described above, in some embodiments, other methods of selecting a high-resolution vertex corresponding to a given low-resolution pixel n may be used. For example, in some embodiments, block diagram 560 includes selecting a P-element interpolation weight vector (t ) from the facial expression blending weights 540 for a vertex that is closest to the center 470 of the UV cell 460 corresponding to the current pixel n (i.e., a high-resolution vertex that is closest (in UV space) to the UV coordinate of the current pixel n of the low-resolution weight texture map 575). v As another example, block 560 may include selecting a plurality of P-element interpolation weight vectors (t v ) for multiple high-resolution vertices that are closest to the center 470 of the UV unit 460 corresponding to the current pixel n (i.e., multiple (out of 3) high-resolution vertices that are closest (in UV space) to the UV coordinate of the current pixel n of the low-resolution weight texture map 575).

[0160] Then, method 500 proceeds to block 565, where the weight texture r of the current pixel n is calculated. n (567). The weight texture 567 of the block diagram 565 for the current pixel n may be a vector r having P elements (corresponding to the P training poses 107) n In embodiments where a plurality of high-resolution vertices correspond to pixel n (ie, block 560 involves selecting a plurality of high-resolution vertices and a plurality of corresponding target facial expression weight vectors t from target facial expression weight vectors 540 v ), the target facial expression weight vector t corresponding to each vertex of the block diagram 560 vcan be multiplied by the barycentric weight attributed to that vertex, and these products can be added to obtain a weight texture r with P elements for the current pixel n. n (567). For example, for a particular pixel n, three high-resolution vertices (A, B, C) can be identified in block diagram 560, and each vertex has: barycentric coordinates (γ A ,γ B ,γ C ), which describes the relationship of the vertex (in UV space) to the current pixel n (or, equivalently, the center 470 of the current cell 460 corresponding to the current pixel n), and the associated P-element target facial expression blending weight 527 (t A ,t B ,t C ). Block 565 may include n =γ A t A +γ B t B +γ C t C Calculate the weight texture r of the current pixel n n (567). At the end of block 565, the P channel weight texture weight r corresponding to the current low resolution pixel n is determined. n (567). As described above, in some embodiments, block 560 may involve determining only a single vertex for the current pixel n. When a single vertex is determined for a particular pixel n at block 560, block 565 weights the texture r n (567) may correspond to the P element target facial expression blending weight vector t of the vertex v (Selected from target facial expression blending weight vector 540).

[0161] The method 500 proceeds to a decision block 570 which considers whether there are remaining pixels n to perform texture calculations of the subprocess 550. If the inquiry at block 570 is affirmative, the method 500 increases the index of n and performs blocks 560 and 565 for subsequent pixels n of the low-resolution weight texture map 575 according to the above steps. If the inquiry at block 570 is negative, the subprocess 550 ends by outputting a weight texture map 575 which includes N weight texture r n (567), each weight texture r n has P elements, and each weight texture r n corresponds to a low-resolution pixel n of the weighted texture map 575 (e.g., a cell 460 of the nominal grid 455 as described above in connection with the method 400). Figure 5 )).

[0162] Figure 8 An example weight texture map 575 (eg, a weight texture vector r for each low-resolution pixel n) that may be uploaded to a graphics engine is shown. n (567)). Figure 8 The weight texture map 575 shown shows one exemplary representation of a weight texture map 575 that may be output by execution of the method 500 . Figure 8 The weight texture map 575 shown includes six separate channel weight textures 605A, 605B, 605C, 605D, 605E, 605F (collectively and / or individually referred to as channel weight textures 605), each channel weight texture 605 corresponding to one of the RGB channels commonly used in graphics engines. Figure 8 In the example shown, the weight texture map 575 and the channel weight texture 605 include 128x128 low resolution pixel UV maps. That is, Figure 8 The weight texture map 575 shown in the example embodiment comprises a resolution of 128x128 pixels (or, equivalently, Figure 5 and 6A 128x128 cells 460 in the nominal grid 455 as explained above. Figure 8 In the example shown, P=6, so the weight texture map 575 includes P=6 elements (vectors r of length P=6) for each low-resolution pixel n in the 128x128 grid. n ), and these P = 6 elements (vector r of length P = 6 n ) represents the similarity between the target facial expression 503 and each of the P=6 training postures. Since the graphics engine usually uses images containing three RGB channels, the channel weight textures 605A, 605B, 605C, 605D, 605E, 605F can be provided as two RGB weight textures 607A, 607B (collectively and / or individually referred to as RGB weight textures 607), wherein there is one channel weight texture 605 in each RGB channel of the RGB weight texture 607. Figure 8 Each pixel n shown stores a corresponding weight texture r associated with the target facial expression 503 n The information of each of the six RGB color channels has a weight texture vector r n 575. In this way, each channel weight texture 605 stores a weight texture element r for each pixel n of the weight texture map 575. n , and the two RGB weight textures 607 store P=6 elements.

[0163] Figure 8The example shown may be applicable to performing method 100 with P = 6 (or other multiples of 3) training poses and weight texture 567 (vector r n ) For the case where each low-resolution pixel n contains P = 6 (or other multiples of 3) elements. Figure 8 In the example of FIG. 6 , there are two RGB weight textures 607, so the RGB weight texture 607 contains a total of six training pose weight texture elements (i.e., there is one weight texture element for each RGB channel of the two RGB weight textures 607). n Representing weight texture elements in the form of RGB color channels (channel weight texture 605) can be advantageous in the case of using a graphics engine (e.g., a graphics processing unit (GPU)) to interpolate the final high-resolution texture of the target facial expression 503, because the GPU can easily accommodate RGB data input. Such an example scenario is discussed below. For example, based on P texture groups 213 ( Figure 2A ) contains high-resolution skin textures and weight texture maps 575 ( Figure 7 , 8 ), the graphics engine can provide six diffuse reflection colors, six specular reflection colors, etc. for each related texture, each color corresponding to a different posture among the P training postures 107A and having a corresponding weight.

[0164] Figure 8 The principles shown in the weight textures 605 and 607 can be applied to embodiments of the method 100 with different calculation parameters. For example, when the approximation model is trained at block 120 of the method 100, when the number P of training poses P is P=3 or P=9, a single or three RGB weight textures 607 (three channel weight textures 605) can be used to represent the weight texture 567 (vector r ) of each low-resolution pixel n. n ) of the appropriate number of elements. Each color channel also does not need to store the same elements as the weight texture 567 (vector r n ) related data. For example, when the number of training postures P in the application of method 100 is 5, the weight texture map 575 can be represented by two RGB weight textures 607A and 607B, similar to Figure 8. However, since only P=5 training posture texture elements need to be represented, the RGB weight texture 607B may store weights associated with the training postures only in the red and green channels (a total of P=5 channel weight textures 605), while the blue channel is empty or used for other purposes. Different data types and data structures may also be used to represent and store the weight texture map 575. For example, the weight texture map 575 can be represented in a CMYK color space containing 4 channels, in contrast to the 3 channels of the RGB color space. For example, this may be advantageous in the case of interpolating textures for P=4 or P=8 (or other multiples of 4) training postures 107A. In other embodiments, the weight texture map 575 is not represented by color channels, but rather includes a single multi-channel attribute texture that accommodates all P elements of each low-resolution pixel n.

[0165] Return method 500( Figure 7 ), after completing subprocess 550, method 500 proceeds to block 580, where the weight texture 567 (vector r n ) to render the high resolution target facial expression 503 to provide a rendered facial image 585. As described in more detail below, the block diagram 580 rendering process may involve weighting the texture 567 (vector r) based on the weight texture 567. n ) interpolates between the P input textures of each type and may also involve a weight texture 567 (vector r ) based on the UV coordinates of each pixel (image pixel) being rendered and the texture value at the corresponding high-resolution texel and the low-resolution pixel n. n ) to perform interpolation. There are a number of possible ways to implement block diagram 580. In some embodiments, the low resolution weight texture 567 (vector r n , one for each low resolution pixel n) is input to the graphics engine (GPU) at block 580. With P training poses 107A ( Figure 2A ) The P groups of high-resolution textures 213 corresponding to the target facial expression 503 may be preloaded into the graphics engine before or when executing block diagram 580. By way of non-limiting example, the P groups of high-resolution textures 213 may include textures for diffuse reflection, ambient occlusion, normal, bent normal, specular reflection, etc. In some embodiments, the graphics engine includes shader software for extracting interpolation weights for the target facial expression 503 from the weight texture map 575 and interpolating the P groups of preloaded high-resolution textures 213.

[0166] As an example of such block diagram 580 interpolation, consider a particular high-resolution pixel of an image to be rendered (image pixel) and a particular texture type to be interpolated (e.g., diffuse). If we assume that the image pixel maps to the UV space of the corresponding high-resolution texel center of the P high-resolution textures 213 that need to be interpolated, then these P high-resolution textures 213 can return corresponding values ​​tex1, tex2, tex3...tex P , where these values ​​are the exact texture values ​​of the corresponding texels. If we further assume that the image pixels are mapped to the UV space centered at the low-resolution pixel n in the weight texture 567, then this corresponds exactly to the weight texture vector r corresponding to that low-resolution pixel n In this example scene, the render texture value of a specific texture type for a specific high-resolution image pixel can be interpolated according to the example expression: texture=tex1·r1+tex2·r2+…tex P ·r P (9) Where: texture is the texture value to be rendered for a specific high-resolution image pixel, r1, r2…r P is the weight texture vector r corresponding to the low-resolution weight texture pixel n n P elements of .

[0167] In some embodiments, it may be desirable to perform a more complex interpolation technique in block 580 that takes into account consecutive UV coordinates between adjacent high-resolution texels of the P corresponding texture groups and / or the known weight texture 567 (vector r n ) between consecutive UV coordinates of adjacent low-resolution pixels n. For example, an image pixel being rendered may not map directly to the center of a high-resolution texel of P textures 213 in UV space, but may instead map to somewhere in UV space between multiple high-resolution texels. In this case, a suitable texture filtering technique (also known as texture query or texture sampling) may be used to interpolate between texture values ​​of adjacent high-resolution texels. By way of non-limiting example, such texture filtering techniques may include bilinear interpolation, cubic interpolation, some other form of interpolation (any of which may be user-defined), and / or similar interpolation between texture values ​​of adjacent high-resolution texels. In this case, tex1, tex2, tex3…tex P The values ​​in equation (9) can be considered as texture query values ​​from the P training textures (e.g., interpolated between high-resolution texels).

[0168] Similarly, the rendered image pixel will not be mapped directly to the center of the low-resolution pixel n in the UV space, but will be mapped to somewhere in the UV space between multiple low-resolution pixels. In this case, a suitable weighted texture filtering technique (also referred to as weighted texture query in this article) can be used to filter the weighted texture 567 (vector r) of the adjacent low-resolution pixels. n ) to obtain an interpolated weight texture vector r*. By way of non-limiting example, such weight texture filtering techniques may include bilinear interpolation, cubic interpolation, some other form of interpolation (any of which may be user-defined), and / or weight textures 567 (vector r) of neighboring low-resolution pixels. n ) to obtain the interpolated (weight texture filtering or weight texture query) weight texture vector r*. In this case, r1, r2…r in equation (9) P The value of can be considered as an element of a weight texture query weight texture vector r*, which is a weight texture vector r* in the weight texture 567 (vector r) of the adjacent low-resolution pixel. n ). This interpolation technique can advantageously alleviate the fact that the weight texture 567 is provided at a low resolution (the vector r n , one for each low-resolution pixel n), and allows for smooth transitions in interpolated weights between regions of the face, avoiding or mitigating artifacts (visible seams or edges) in blended high-resolution textures rendered on the face.

[0169] Without wishing to be bound by theory, the inventors believe that the low-resolution weight texture 567 (vector r n , one for each low-resolution pixel n) to achieve a higher resolution with ideal visual effects. In the embodiment of the present invention, the approximate model 123, 340 is used to achieve the local consistency of the solved interpolation weights (for example, the per-vertex facial expression blending weight 540 (including the blending weight 527 (t v ), because nearby output vertices are influenced by a similar set of feature edges and thus produce similar influence weights from various training poses.

[0170] The graphics engine described herein generally refers to computer hardware and related software that receives a set of software instructions and inputs to process and render animated 3D graphics. These graphics engines typically contain shader software for generating detailed textures and colors in 3D scenes. The software instructions executed by the graphics engine are typically compiled and executed on one or more GPUs, but can also be executed by one or more central processing units (CPUs). Examples of graphics engines suitable for the present invention include, but are not limited to, Unreal Engine TM 、Unity TM、Frostbite TM and CryEngine TM In some embodiments, the weight texture 567 (vector r) is set at block 130 of method 100. n ) is applied to the target facial expression 503 and the target facial expression 503 is rendered as an image 585 entirely by a suitable graphics engine. In other embodiments, the rendering of the high-resolution texture at block 580 of method 500 is performed only by the graphics engine, while the remaining portions of method 500 are implemented using one or more other suitable computer processors.

[0171] At block 580, the weight texture 567 (vector r n ) After modifying the rendering of the target facial expression 503 to provide the rendered facial expression 585, the method 500 ends. Returning to the method 100 ( Figure 1 ), method 100 may optionally proceed to decision block 135, which evaluates whether the performance of applying texture to target facial expression 503 and rendering target facial expression 503 (i.e., rendered facial expression 585) at block 130 is satisfactory. In some embodiments, the determination at block 135 is based on whether the target frame rate and / or desired resolution defined in computation parameters 113 has been achieved by real-time computation at block 130.

[0172] If the inquiry at decision block 135 is negative, method 100 returns to block 110 where new computation parameters 113 may be defined to better meet the desired metrics. For example, if a desired frame rate is not achieved or GPU usage exceeds defined limits, performance may be considered unsatisfactory. Defining new computation parameters 113 may include one or more of the following: Reduce the number of training positions; Reducing the granularity of the low-resolution weight texture map 575 (eg, the resolution of the UV map 450); and Reduce the resolution of high-resolution textures.

[0173] In some embodiments, the determination by block 135 whether the performance of the real-time calculation is satisfactory is based on an evaluation of the quality of the rendered facial expression 585 (see Figure 1 and Figure 7 ). For example, the quality of rendering facial expression 585 may be defined according to one or more of the following: The number of high-resolution texture types used (e.g., specular, diffuse, etc.) accurately depicts the current the range of details that appear or disappear in facial expressions (e.g., wrinkles); Whether the granularity of the low-resolution weight texture map 575 subdivides the face into enough regions (e.g., the weight texture map 575 or the UV map 450 ( Figure 5 , the number of low-resolution pixels N in FIG6) is sufficient to distinguish the facial expressions from the resulting textures. the impact of Whether high-resolution textures contain enough detail for rendering. In some embodiments, when it is determined that the quality of the rendered facial expression 585 is not high enough, any of the above inputs to method 100 may be appropriately changed. For example, this may include capturing more high-resolution textures, increasing the granularity of the low-resolution weight textures, and increasing the number of high-resolution vertices. In some embodiments, such changes may be accompanied by corresponding changes in computation parameters 113 (e.g., reducing the target frame rate when applying more high-resolution textures). In some embodiments, the evaluation of block 135 may be performed in whole or in part by a human artist or other user.

[0174] If the evaluation at block 135 is positive, then method 100 ends. By performing method 100 and the methods described herein, photo-realistic texture details that vary with real-time facial expressions may be achieved in a computationally efficient manner.

[0175] In some embodiments, the rasterization calculation ( Figure 1 and Figure 5 ) and the corresponding rasterization matrix 127 ( Figure 1 4 (shown in dashed lines) are not required and are an optional part of method 100. In such an embodiment, low-resolution 2D map 450 and corresponding nominal grid 455 (determined in block 410 of method 400) may be replaced by high-resolution pixels in the 2D space of the image being rendered in block 130, and block 130 may be modified to perform interpolation on the high-resolution pixels in the 2D space of the image being rendered (rather than on the low-resolution pixels mapped to the 2D UV space rasterized in block 125).

[0176] In some such embodiments, Figure 1 The block diagram 130 is not through Figure 7 The method 500 can be performed by implementing Figure 7B To perform the method 750 shown in FIG. Figure 7B The method 750 may have multiple Figure 7The steps of method 750 are the same as or similar to the steps of method 700, and similar reference numerals are used to refer to these steps. The input of method 750 is different from the input of method 700 in that rasterization matrix 127, 430 is not an input of method 750. The approximation calculation 505 of method 750 can be substantially the same as the approximation calculation 505 of method 700 described above, and as described above, the P weight 527 (t of each vertex v) of the per-vertex blending weight 540 (i.e., the target facial expression 503) for each vertex v can be generated. v )). For each high-resolution vertex v of the target facial expression 503, the target facial expression blending weight 540 of each vertex represents the relative approximation of the similarity between the vertex and each of the P training postures 107A in the vicinity of the vertex v. As described above, the target facial expression weight 540 includes a vector t of length P for each high-resolution vertex v (v=1, 2, 3...V) of the target facial expression 503. v .

[0177] At the end of the approximation calculation 505, the method 750 may proceed to the rendering process of block diagram 752. The rendering process of block diagram 752 of the method 750 is similar to the combination of the texture calculation (block diagram 550) and the rendering (block diagram 580) of the method 700, except that the steps of the rendering process of block diagram 752 are performed in a 2D space corresponding to the image (and corresponding pixels) of the target facial expression 503 being rendered. In practice, the program of the rendering process of block diagram 752 may be performed by a graphics processing unit based on the per-vertex blending weights 540 for the target facial expression 503 (i.e., the P weights 527 (t v )) can be output from the approximation calculation 505 and passed to the graphics processing unit as a per-vertex attribute.

[0178] Block 752 rendering includes a loop that is executed once for each high-resolution pixel n of a plurality of N high-resolution pixels in the 2D image being rendered that corresponds to the target facial expression 503. The output of the block 752 rendering process is a rendered facial image 585 that corresponds to the target facial expression 503. It should be understood that the variables n and N still correspond to a single pixel n and a total number of pixels N, except that in the context of method 750, these are pixels in the 2D space of the rendered image (rather than in 2D UV space as in method 700). For each pixel n (n∈{1,2,…N}) in the two-dimensional space of the rendered image, block 752 rendering includes block 756, which involves selecting a corresponding vertex corresponding to the current pixel n and a corresponding target expression per-vertex weight (t v). The vertex selection in block 756 may be similar to block 560 (method 700) or blocks 415 and 425 (method 400, Figure 5 ), except that, for block diagram 756, the 2D space is the space of the image being rendered. As discussed above with respect to blocks 560, 415, and 425, selecting the vertex corresponding to the current pixel n in block diagram 756 may involve selecting the vertex triangle that is closest to the current pixel n when mapped to the 2D space and determining the corresponding barycentric coordinates. In some embodiments, the selection of the vertex corresponding to the current pixel n in block diagram 756 may involve selecting a single vertex (or multiple vertices other than three vertices) that is closest to the current pixel n when mapped to the 2D space. As with the case of block diagram 560 discussed above, once the vertices are determined, their corresponding target expression per-vertex weights (t v ) is also known.

[0179] The rendering process at block 752 then proceeds to block 758, which involves calculating the per-pixel blending weight r for the current pixel n. n (760). The process of block 758 is similar to the process discussed above for calculating the weight texture r for each low resolution pixel. n The process of block 565 of (567) is different in that the mixing weight r of each pixel is calculated in block 758 n The 2D space of (760) is the 2D space of the image being rendered, and pixel n is a pixel of the image being rendered. It can be understood that the weight texture r discussed above n (567) can also be thought of as a “per-pixel blending weight” r n (567), the difference is that the per-pixel mixing weight r n (567) and the per-pixel blending weight r n The 2D space of (760) is different from the corresponding pixel.

[0180] The per-pixel blending weight 760 of the block diagram 758 for the current pixel n may be a vector r having P elements (corresponding to the P training poses 107) n In embodiments where multiple high-resolution vertices correspond to pixel n (ie, block 756 involves selecting multiple high-resolution vertices and multiple corresponding target facial expression weight vectors t v ), the target facial expression weight vector t corresponding to each vertex of the block diagram 756 is v Multiply by the barycentric weight attributed to that vertex and add these products to obtain a per-pixel blending weight r with P elements for the current pixel n n(760). For example, for a particular pixel n, there are three high-resolution vertices (A, B, C) identified in block 756 (e.g., a triangle defined by vertices (A, B, C), and the particular pixel n has a barycentric coordinate (γ A ,γ B ,γ C ), which describes the relationship of pixel n (in the 2D space of the image being rendered) to the vertex (A, B, C); and the associated P-element target facial expression blending weight 527 determined in blocks 525, 530. Block 758 may include determining the target facial expression blending weight 527 according to r n =γ A t A +γ B t B +γ C t C Calculate the per-pixel mixing weight r of the current pixel n n (760) According to the result of block diagram 758, the P channel mixing weight r corresponding to the current pixel n is determined. n (760). As described above, in some embodiments, block 756 may involve determining only a single vertex for the current pixel n. When a single vertex is determined for a particular pixel n in block 756, block 758 blends the weights r n (760) may correspond to the P element target facial expression blending weight vector t of the vertex v .

[0181] The rendering process of block 752 then proceeds to block 762, which involves using a per-pixel blending weight r n (760) interpolates the texture to render the current pixel n of the rendered facial image 585 corresponding to the target facial expression 503. Block 762 of method 750 is similar to block 580 of method 700, except that the process of block 762 is performed for the current high-resolution pixel n (image pixel) of the image 585 being rendered. Specifically, block 762 involves rendering the high-resolution target facial expression 503, where the texture is represented by the per-pixel blending weight 760 (vector r n ) is modified to provide a texture value for the current pixel n of the rendered face image 585. Similar to block 580, the rendering process of block 762 involves a per-pixel blending weight 760 (vector r n ) interpolates between the P input textures of each type, and may also involve interpolating based on the UV coordinates of the image pixels and the texture values ​​at the corresponding texels. As described above, with the P training poses 107A ( Figure 2A) The corresponding P set of high-resolution textures 213 may include textures for diffuse reflection, ambient occlusion, normal, bent normal, specular reflection, etc. As an example of block diagram 762 interpolation, consider a specific texture type (e.g., diffuse reflection) corresponding to a specific high-resolution image pixel n in the 2D space of the image 585 being rendered, where tex1, tex2, tex3 ... tex P are texture lookup values ​​for the P high-resolution textures 213 (corresponding to the P training poses). See above (in the description of block diagram 580) for discussion of texture lookup or texture filtering. As described above, by way of non-limiting example, such texture filtering techniques may include bilinear interpolation, cubic interpolation, some other form of interpolation (any of which may be user-defined), and / or similar interpolation between texture values ​​of adjacent texels. In this case, the rendered texture value for a particular texture type may be interpolated according to the example expression: texture=tex1·r1+tex2·r2+…tex P ·r P (10) Where: texture is the texture value that needs to be rendered for the current high-resolution image pixel n, r1, r2…r P is the weight texture vector r corresponding to the current high-resolution image pixel n n P elements of .

[0182] The block 752 rendering process then proceeds to block 764, which involves asking if there are more pixels n to render. If so, the block 752 rendering process loops back to perform blocks 756, 758, and 762 for the next pixel n. When all N pixels in the 2D space of the image 585 have been rendered, the block 752 rendering process for the current target facial expression 583 is complete.

[0183] In practice, the process of rendering block diagram 752 may be performed by a graphics processing unit when rendering the target facial expression 503, wherein the per-vertex blending weight 527 (t v ) can be passed to the graphics processing unit as user-defined per-vertex attributes. Explanation of Terms

[0184] Unless the context clearly requires otherwise, throughout the specification and claims: · The words “including”, “comprising”, etc. should be interpreted in an inclusive sense rather than an exclusive or exhaustive sense; that is, they should be interpreted in the sense of “including but not limited to”; "Connect", "couple" or any variation thereof, means any direct or indirect connection or coupling between two or more elements; the connection or coupling between elements may be physical, logical, or a combination of both; · "This", "above", "below" and words of similar meaning when used to describe this specification shall refer to this specification as a whole and not to any specific part of this specification; · "or", referring to a list of two or more items, includes all of the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list; The singular forms "a", "an" and "the" also include the meaning of any appropriate plural forms.

[0185] For the purpose of illustration, specific examples of systems, methods, and devices have been described herein. These are examples only. The techniques provided herein may be applied to other systems in addition to the example systems described above. In the practice of the present invention, many changes, modifications, additions, omissions, and substitutions may be made. The present invention includes variations of the described embodiments that will be apparent to those skilled in the art, including variations obtained by: replacing features, elements, and / or actions with equivalent features, elements, and / or actions; mixing and matching features, elements, and / or actions of different embodiments; combining features, elements, and / or actions of the embodiments described herein with features, elements, and / or actions of other technologies; and / or omitting combined features, elements, and / or actions in the described embodiments.

[0186] In some embodiments, the present invention may be implemented in software. For greater clarity, "software" includes any instructions executed on a processor, and may include (but not be limited to) firmware, resident software, microcode, etc. Both processing hardware and software may be centralized or distributed (or a combination of the two), in whole or in part, as known to those skilled in the art. For example, software and other modules may be accessed via local memory, a network, a browser or other application in a distributed computing environment, or other means suitable for the above purposes.

[0187] Processing may be centralized or distributed. When processing is distributed, information, including software and / or data, may be stored in a centralized or decentralized manner. Such information may be exchanged between different functional units via a communications network, such as a local area network (LAN), wide area network (WAN), or the Internet, wired or wireless data links, electromagnetic signals or other data communications channels.

[0188] The software and other modules may reside on a server, workstation, personal computer, tablet computer, image data encoder, image data decoder, PDA, color grading tool, video projector, audio-visual receiver, display (such as television), digital cinema projector, media player and other devices suitable for the purposes described herein. Those skilled in the relevant art will recognize that aspects of the system can be practiced through other communication, data processing or computer system configurations, including: Internet devices, handheld devices (including personal digital assistants (PDAs)), wearable computers, various cellular or mobile phones, multi-processor systems, microprocessor-based or programmable consumer electronics (e.g., video projectors, audio-visual receivers, displays (such as televisions), etc.), set-top boxes, color grading tools, network PCs, microcomputers, mainframe computers, etc.

[0189] Embodiments of the present invention may be implemented using specially designed hardware, configurable hardware, a programmable data processor configured by providing software (which may optionally include "firmware") that can be executed on the data processor, a special-purpose computer or a data processor that is specially programmed, configured or constructed to perform one or more steps of the method explained in detail herein and / or a combination of two or more thereof. Examples of specially designed hardware include: logic circuits, application specific integrated circuits ("ASICs"), large-scale integrated circuits ("LSIs"), very large-scale integrated circuits ("VLSIs"), etc. Examples of configurable hardware include: one or more programmable logic devices, such as programmable array logic ("PAL"), programmable logic arrays ("PLAs"), and field programmable gate arrays ("FPGAs"). Examples of programmable data processors include: microprocessors, digital signal processors ("DSPs"), embedded processors, graphics processors, mathematical coprocessors, general-purpose computers, server computers, cloud computers, mainframe computers, computer workstations, etc. For example, one or more data processors may implement the methods described herein by executing software instructions in a program memory accessible to the processor.

[0190] Although the processes or block diagrams described herein are presented in a given order, alternative examples may perform routines with steps, or employ systems with block diagrams, in a different order, and certain processes or block diagrams may be deleted, moved, added, subdivided, combined, and / or modified to provide alternatives or sub-combinations. Each process or block diagram may be implemented in a variety of different ways. In addition, although processes or block diagrams are sometimes shown to be performed sequentially, these processes or block diagrams may also be performed in parallel, or may be performed at different times.

[0191] Furthermore, while elements are sometimes shown as being performed sequentially, they may also be performed concurrently or in a different order. The following claims are therefore to be interpreted as including all such variations within their intended scope.

[0192] The present invention can also be provided in the form of a program product. The program product may include any non-transitory medium, which carries a set of computer-readable instructions, which, when executed by a data processor, causes the data processor to perform the method of the present invention. The program product according to the present invention can take various forms. The program product may include, for example, non-transitory media, such as magnetic data storage media (including floppy disks, hard disk drives), optical data storage media (including CD ROMs, DVDs), electronic data storage media (including ROMs, flash RAMs, EPROMs, hard-wired or pre-programmed chips (e.g., EEPROM semiconductor chips)), nanotechnology memories, etc. The computer-readable signals on the program product may be selectively compressed or encrypted.

[0193] When a component (e.g., a software module, a processor, a component, a device, a circuit, etc.) is mentioned above, unless otherwise specified, reference to the component (including reference to "means") should be interpreted as including any component that performs the function of the component as an equivalent of the component (i.e., functional equivalent), including components that are not structurally equivalent to the disclosed structure that performs the function in the exemplary embodiments shown in the present invention.

[0194] Therefore, it is intended that the following appended claims and claims introduced hereafter be interpreted to include all such modifications, permutations, additions, omissions and sub-combinations that can be reasonably inferred. The scope of the claims should not be limited by the preferred embodiments listed in the examples, but should be given the broadest interpretation consistent with the entire specification.

Claims

1. A method for determining a high-resolution texture for rendering a target facial expression, the method comprising: (a) obtaining a plurality of P training facial poses, each of the P training facial poses comprising: V high-resolution vertices and training pose positions of the V high-resolution vertices in a three-dimensional (3D) coordinate system; and at least one two-dimensional (2D) texture; (b) acquiring a target facial expression, the target facial expression comprising the V high-resolution vertex and a target expression position of the V high-resolution vertex in the 3D coordinate system; (c) determining a target feature map for the target facial expression based at least in part on the target facial expression and a neutral pose selected from the P training poses, the target feature map comprising a plurality of target feature map parameters; (d) determining a training feature map, the training feature map comprising a plurality of training feature map parameters for the P training facial poses, each of the training feature maps being based at least in part on one of the P training facial poses and the neutral pose; (e) for each high-resolution vertex in the plurality of high-resolution vertices v in the V high-resolution vertices, determining a plurality of blending weights t based at least in part on the following factors: v : Using the P to train an approximate model of facial posture training; and a similarity measure It represents the similarity between the target feature map at the vertex v and each of the training feature maps; (f) For each pixel n in a plurality of N pixels in a two-dimensional space: (f.1) determining one or more corresponding vertices of the plurality of high-resolution vertices v corresponding to the pixel n based at least in part on the 2D coordinates of the pixel n in the 2D space and a mapping of the plurality of high-resolution vertices v to the 2D space, the mapping providing the 2D coordinates of the plurality of high-resolution vertices v in the 2D space; and (f.2) based at least in part on a blending weight t of the one or more corresponding vertices v To determine a set of per-pixel mixing weights r for the pixel n n , the per-pixel mixing weight r n including a weight for each of the P training face poses; and (g) for each high-resolution pixel in the 2D rendering of the target facial expression, based at least in part on the per-pixel blending weight r n The 2D textures of the P training face poses are interpolated to provide a target texture for the high-resolution pixels.

2. The method of claim 1 or any other claim herein, wherein determining the training feature map for the P training pose comprises determining a plurality of W low-resolution handle vertices, wherein the W handle vertices are a low-resolution subset of the V high-resolution vertices, wherein W <V。 3. The method of claim 2 or any other claim herein, wherein determining the training feature maps for the P training poses comprises, for each of the P training facial poses: determining a training feature map geometry corresponding to the training facial pose, the training feature map geometry comprising a plurality of F feature edges defined between corresponding pairs of training pose positions of a plurality of low-resolution W-handle vertices of the training pose; determining the plurality of training feature map parameters as a plurality of F training feature map parameters corresponding to the F feature edges, each training feature map parameter of the plurality of F training feature map parameters being based at least in part on a corresponding feature edge of the training facial pose and a corresponding feature edge of the neutral pose; Thus, the P training feature graphs are obtained, and each of the P training feature graphs includes a corresponding plurality of F training feature graph parameters.

4. The method of claim 3 or any other claim herein, wherein determining the plurality of F training feature map parameters corresponding to the F feature edges comprises: For each training feature map parameter in the plurality of F training feature map parameters, the training feature map parameter is determined using an equation of the following form f i =(||p i，1 -p i，2 ||-l i ) / l i (1) Where: f i is with the i th The i-th feature edge th Training feature map parameters; p i，1 and p i，2 It is defined as i The training pose position of the handle vertex of the endpoint, l i is the position corresponding to the i-th position in the neutral posture th The length of the feature edge.

5. The method of claim 2 or any other claim herein, wherein determining the training feature maps for the P training poses comprises, for each of the P training facial poses: determining a training feature map geometry corresponding to the training face pose, the training feature map geometry comprising a plurality of F feature edges defined between training pose positions of corresponding pairs of low-resolution handle vertices of a plurality of low-resolution W handle vertices of the training pose; determining the plurality of training feature map parameters, each of the plurality of training feature map parameters being based at least in part on some or all of the F feature edges of the training face poses and some or all of the feature edges of the neutral pose; Thus, the P training feature maps are obtained, and each of the P training feature maps includes a corresponding plurality of training feature map parameters.

6. The method of any one of claims 1 and 5 or any other claim herein, wherein determining the training feature map of the P training poses comprises, for each of the P training facial poses: determining the plurality of training feature map parameters, each of the plurality of training feature map parameters being based at least in part on one or more raw parameters, the one or more raw parameters being determined at least in part based on the training facial pose and the neutral pose; Thereby, the P training feature maps are obtained, each of which includes a corresponding plurality of training feature map parameters.

7. The method of claim 6 or any other claim herein, wherein determining the plurality of training feature map parameters comprises determining one or more of: a deformation gradient based at least in part on the training face pose and the neutral pose; pyramid coordinates based at least in part on the training face poses and the neutral pose; triangle parameters based at least in part on the training face poses and the neutral pose; and 1-ring neighborhood parameters based at least in part on the training face poses and the neutral pose.

8. The method of any one of claims 2 to 4 or any other claim herein, wherein determining the target feature map comprises: Determining a target feature map geometry corresponding to the target facial expression, the target feature map geometry comprising a plurality of F feature edges defined between corresponding pairs of target expression positions of low-resolution handle vertices among a plurality of low-resolution W handle vertices of the target expression; determining the plurality of target feature map parameters as a plurality of F target feature map parameters corresponding to the F feature edges, each target feature map parameter of the plurality of F target feature map parameters being based at least in part on a corresponding feature edge of the target facial expression and a corresponding feature edge of the neutral posture; Thereby, a target characteristic map including the plurality of F target characteristic map parameters is obtained.

9. The method of claim 8 or any other claim herein, wherein determining the plurality of F target feature map parameters corresponding to the F feature edges comprises, for each target feature map parameter in the plurality of F target feature map parameters, determining the target feature map parameter using an equation of the form, f i =(||p i，1 -p i，2 ||-l i ) / l i (1) in: f i is with the i th The i-th feature edge corresponding to th Target feature map parameters, p i，1 and p i，2 It is defined as i The endpoints of the handle vertices are the target expression positions, and l i is the i-th position in the neutral posture th The length of the feature edge.

10. The method of claim 5 or any other claim herein, wherein determining the target feature map comprises: Determining a target feature map geometry corresponding to the target facial expression, the target feature map geometry comprising a plurality of F feature edges defined between target expression positions of corresponding pairs of low-resolution handle vertices of a plurality of low-resolution W handle vertices of the target facial expression; determining the plurality of target feature map parameters, each target feature map parameter of the plurality of target feature map parameters being based at least in part on some or all of the feature edges of the target facial expression and some or all of the feature edges of the neutral posture; Thereby, the target feature map including the multiple target feature map parameters is obtained.

11. The method of any one of claims 6 and 7 or any other claim herein, wherein determining the target feature map comprises: determining a plurality of target feature map parameters, each of the plurality of target map feature parameters being based at least in part on one or more raw parameters determined based on the target facial expression and the neutral posture; Thereby, the target feature map including the multiple target feature map parameters is obtained.

12. The method of claim 11 or any other claim herein, wherein determining the plurality of target feature map parameters comprises determining one or more of: deformation gradients based at least in part on the target facial expression and the neutral pose; pyramid coordinates based at least in part on the target facial expression and the neutral pose; triangle parameters based at least in part on the target facial expression and the neutral pose; and 1-ring neighborhood parameters based at least in part on the target facial expression and the neutral pose.

13. A method according to any one of claims 1 to 12 or any other claim herein, wherein determining the one or more corresponding vertices from the plurality of high-resolution vertices v corresponding to the pixel n is based at least in part on proximity of the 2D coordinates of the one or more corresponding vertices to the 2D coordinates of the pixel n.

14. The method of claim 13 or any other claim herein, wherein: Determining the one or more corresponding vertices from the plurality of high-resolution vertices v corresponding to the pixel n includes determining three vertices having two-dimensional coordinates closest to the two-dimensional coordinates of the pixel n, thereby defining a triangle around the pixel n in the two-dimensional space.

15. The method of claim 13 or any other claim herein, wherein determining the one or more corresponding vertices from the plurality of high-resolution vertices v corresponding to the pixel n comprises determining the barycentric coordinates of the triangle relative to the 2D coordinates of the pixel n.

16. The method of claim 13 or any other claim herein, wherein: Determining the one or more corresponding vertices from the plurality of high-resolution vertices v corresponding to the pixel n includes determining a vertex whose 2D coordinates are closest to the 2D coordinates of the pixel n.

17. A method according to any one of claims 1 to 16 or any other claim herein, comprising selecting a plurality of high-resolution vertices v from the V high-resolution vertices as a union of one or more high-resolution vertices corresponding to each pixel n in the plurality of N pixels; and wherein, Selecting a plurality of high-resolution vertices v from the V high-resolution vertices is performed before step (e), so that step (e) is performed only for the plurality of high-resolution vertices v selected from the V high-resolution vertices.

18. The method of any one of claims 1 to 16, comprising selecting the plurality of high-resolution vertices v from the V high-resolution vertices as all V high-resolution vertices.

19. The method of claim 15 or any other claim herein, wherein the set of per-pixel blending weights r is determined for the pixel n. n comprising a blending weight t based at least in part on the three vertices of a triangle defining the pixel n in the two-dimensional space. v and the barycentric coordinates of the triangle relative to the two-dimensional coordinates of the pixel to determine the set of per-pixel blending weights r of the pixel n n .

20. The method of any one of claims 15 and 19 or any other claim herein, wherein the set of per-pixel blending weights r is determined for the pixel n. n Including execution form r n =γ A t A +γ B t B +γ C t C operation, where t A , t B , t C represents the blending weights t of the three vertices of the triangle defined in the 2D space around the pixel n determined in step (e) v , and γ A , γ B , γ C Represents the barycentric coordinates of the triangle relative to the 2D coordinates of the pixel n.

21. The method of claim 16, wherein determining the set of per-pixel blending weights r for the pixel n n The set of per-pixel blending weights r for the pixel n is included n Determine the mixing weight t v , the mixing weight t v is determined in step (e) for a vertex whose 2D coordinates are closest to the 2D coordinates of the pixel n.

22. The method of any one of claims 1 to 21 or any other claim herein, wherein the 2D space is the 2D rendered 2D space.

23. The method of any one of claims 1 to 21 or any other claim herein, wherein the 2D space is different from the 2D space of the 2D rendering.

24. The method of claim 23 or any other claim herein, wherein the 2D space is a UV space.

25. The method of any one of claims 23 to 24 or any other claim herein, wherein the resolution of the plurality of N pixels in the 2D space is lower than the resolution of pixels in the 2D rendering.

26. The method of any of claims 23 to 25 or any other claim herein, wherein interpolating the 2D texture of the P training face pose is based at least in part on positions of high resolution pixels mapped to the 2D space.

27. A method according to any one of claims 1 to 26 or any other claim herein, comprising determining the approximate model based at least in part on a training feature map of the P training facial poses.

28. The method of claim 27 or any other claim herein, wherein determining the approximate model comprises, for each high-resolution vertex v in the V high-resolution vertices: The solution form is w = dφ -1 The equation of , where: d is the P-dimensional identity matrix; and φ is a PxP dimensional matrix of weighted radial basis functions (RBFs), wherein each element of φ is based at least in part on a training feature map parameter for comparing the P training feature maps; Thereby obtaining a PxP dimensional matrix of RBF weights w corresponding to the high-resolution vertex v; Thus, an approximate model is defined, which includes a set of V RBF weight matrices w.

29. The method of claim 28 or any other claim herein, wherein the training feature map parameters of the P training feature map are determined according to the method of any one of claims 3 to 7.

30. The method of claim 29 or any other claim herein, wherein: For each high-resolution vertex v in the V high-resolution vertices, the weight of the weighted radial basis function (RBF) in the matrix φ is determined at least in part based on a proximity mask, which assigns uniform values ​​to feature edges surrounding the vertex v and assigns attenuated values ​​to feature edges farther away from the vertex v.

31. The method of claim 30 or any other claim herein, wherein the proximity mask assigns exponentially decaying values ​​to feature edges that are farther from the vertex v.

32. A method according to any one of claims 30 to 31 or any other claim herein, wherein the proximity mask is determined according to an equation of the form: in: l i is the i-th neutral posture th The length of the feature edge; L v，i is the i-th distance from the vertex v to the neutral pose th The sum of the distances between the endpoints of the feature edge; and β is a configurable scalar parameter that controls the decay rate.

33. The method of any one of claims 30 to 32 or any other claim herein, comprising: if determining that the i th If the calculated proximity mask of the feature edge is less than the threshold, the i-th th The proximity mask of feature edges is set to zero.

34. The method of claim 30 or any other claim herein, wherein the proximity mask comprises assigning non-zero values ​​to a configurable number of feature edges that are relatively closer to the vertex v, and assigning zero values ​​to other feature edges that are relatively farther away from the vertex v.

35. The method of any one of claims 29 to 34 or any other claim herein, wherein the PxP dimensional matrix φ of the weighted RBF is in the kth th Column and l th The element at the row (φ v，k，l ) is given by an equation of the form: in: γ is the RBF kernel function; f k，i is the kth th Training posture i th Training feature map parameters of feature edges; f l，i It is the first th Training posture i th The training feature map parameters of the feature edge; α v，i Based on the i th The feature edge is assigned to the i-th vertex v based on its proximity to the high-resolution vertex v. th The weight of the feature edge.

36. The method of claim 35 or any other claim herein, wherein the RBF kernel function γ is a biharmonic RBF kernel function.

37. The method of claim 27 or any other claim herein, wherein determining the approximation model comprises training an approximation model based at least in part on training feature map parameters of the P training feature maps to solve a sparse interpolation problem.

38. The method of claim 37 or any other claim herein, wherein the training feature map parameters of the P training feature map are determined according to the method of any one of claims 3 to 7.

39. The method of any one of claims 35 to 36 or any other claim herein, wherein: Determining the target feature map parameters of the target feature map according to the method of any one of claims 8 to 12; and The method comprises determining a similarity measure It means that the similarity between the target feature map at the vertex v and each P training feature map is a P-dimensional similarity vector where the similarity vector is given by an equation of the form kth th element Where: γ is the RBF kernel function; f t，i is the i-th target feature map th The target feature map parameters of the feature edge; f k，i For the kth th The i-th training feature map of the training posture th The training feature map parameters of the feature edge; α v，i According to the i th The proximity of the feature edge to the high-resolution vertex v is assigned to the i-th th The weight of the feature edge; i (i∈1,2,...F) is an index describing the number of edges in each of the training feature graphs and the target feature graph.

40. The method of claim 39 or any other claim herein, wherein determining the plurality of blending weights t v Including execution form operation, where w is the RBF weight matrix of the high-resolution vertex v, is a P-dimensional similarity vector, representing the similarity between the target feature map at the vertex v and each training feature map in the P training feature maps.

41. The method of any of claims 23 to 26 or any other claim herein, wherein interpolating the 2D texture of the P training face pose comprises: A texture query is performed on the 2D texture of the P training face poses based on the high-resolution pixels in the 2D rendering to obtain an interpolated texture value (tex1, tex2, tex3, tex4, tex5, tex6, tex7, tex8, tex9, tex10, tex11, tex12, tex13, tex14, tex15, tex16, tex17, tex18, tex19, tex20, tex210, tex22, tex23, tex ... P ), each interpolated texture value is interpolated between texture values ​​at a plurality of texels of the 2D texture of a corresponding training face pose of the P training face pose; Based on the high-resolution pixels in the 2D rendering, the 2D space is subjected to a weight texture query to obtain a set of interpolated per-pixel blending weights r*((r1, r2...r P ), whose per-pixel mixing weight r of multiple pixels n in the 2D space n Interpolate between.

42. The method of claim 41 or any other claim herein, wherein interpolating the 2D texture of the P training face pose comprises determining the target texture of the high-resolution pixels according to an equation of the form (texture), texture=tex1·r1+tex2·r2+…tex P ·r P in:( tex1, tex2, tex3...tex P ) is the interpolated texture value of each training face pose in the P training face poses; and (r1, r2...r P ) is a set of interpolated per-pixel blending weights r*.

43. The method of any one of claims 41 to 42 or any other claim herein, wherein texture querying the 2D texture of the P training face pose based on high resolution pixels in the 2D rendering comprises mapping the high resolution pixels in the 2D rendering to UV space to determine 2D coordinates of the high resolution pixels in UV space.

44. The method of any one of claims 41 to 43 or any other claim herein, wherein: Performing a weighted texture query on the 2D space based on the high-resolution pixels in the 2D rendering includes mapping the high-resolution pixels in the 2D rendering to the 2D space to determine 2D coordinates of the high-resolution pixels in the 2D space.

45. The method of claim 22 or any other claim herein, wherein interpolating the 2D texture of the P training face poses comprises: A texture query is performed on the 2D texture of the P training face pose based on the high resolution pixels in the 2D rendering to obtain the P training face pose (tex1, tex2, tex3...tex P ), each interpolated texture value being interpolated between texture values ​​at a plurality of texels of the 2D texture of a corresponding training face pose in the P training face poses.

46. ​​The method of claim 45 or any other claim herein, wherein interpolating the 2D texture of the P training face poses comprises determining a target texture for high resolution pixels according to an equation of the form, texture=tex1·r1+tex2·r2+…tex P ·r P in:( tex1, tex2, tex3...tex P ) is the interpolated texture value of the training face pose in the P training face poses; (r1, r2...r P ) is the per-pixel blending weight r of the set of interpolated high-resolution pixels in the 2D space of the 2D rendering n .

47. The method of any one of claims 45 to 46 or any other claim herein, wherein texture querying the 2D texture of the P training face pose based on high resolution pixels in the 2D rendering comprises mapping the high resolution pixels in the 2D rendering to UV space to determine 2D coordinates of the high resolution pixels in UV space.

48. The method of any one of claims 1 to 47 or any other claim herein, wherein: The method is for rendering an animation sequence including a plurality of animation frames corresponding to a plurality of target facial expressions at an animation frame rate; performing steps (a), (d) and (f.1) on the animation sequence as a pre-computation step prior to one or more of steps (b), (c), (e), (f.2) and (g); and As part of step (b), steps (c), (e), (f.2) and (g) are performed in real time for the animation sequence after obtaining a corresponding target facial expression from a plurality of target facial expressions.

49. The method of claim 48 or any other claim herein, wherein: The set of per-pixel blending weights r for each pixel n n a vector of P elements; and The method comprises assigning the per-pixel mixing weight r n The corresponding elements of are provided to a graphics processor in color channels of one or more corresponding images having N pixels.

50. The method of any one of claims 23 to 26, 41 to 44, or any other claim herein, wherein: The method is for rendering an animation sequence including a plurality of animation frames corresponding to a plurality of target facial expressions at an animation frame rate; performing steps (a), (d) and (f.1) for the animation sequence as a pre-calculation step prior to one or more of steps (b), (c), (e), (f.2) and (g); and As part of step (b), performing steps (c), (e), (f.2) and (g) in real time for the animation sequence after obtaining a corresponding target facial expression from a plurality of target facial expressions; The set of per-pixel blending weights r for each pixel n n a vector of P elements; and The method comprises assigning the per-pixel mixing weight r n The corresponding elements of are provided to the graphics processor in color channels of one or more corresponding low-resolution images having N low-resolution pixels, whose resolution is lower than the resolution of the image being rendered.

51. The method of any one of claims 49 to 50 or any other claim herein, wherein: The color channels of the one or more corresponding low-resolution images include red, blue, and green (RGB) color channels.

52. The method of any one of claims 49 to 51 or any other claim herein, wherein: The plurality of P training postures include P training postures, and the number of P is a multiple of 3.

53. A computer-implemented method for training a facial pose approximation model, the method being operable to determine similarity of a target facial expression to a plurality of training facial poses at each high-resolution vertex of a set V of high-resolution vertices defining a topology in common with the target facial expression and the plurality of training facial poses, the method comprising: obtaining a plurality P of training face poses, each of the P training face poses comprising a training pose position of V high-resolution vertices in a three-dimensional (3D) coordinate system; For each of the P training face poses: determining a training feature map comprising a corresponding plurality of training feature map parameters, wherein each of the corresponding plurality of training feature map parameters is based at least in part on one or more original parameters, the one or more original parameters being determined at least in part based on the training facial pose and a neutral pose selected from the P training poses; Thus, P training feature maps are obtained, each of the P training feature maps includes a corresponding plurality of training feature map parameters; and For each high-resolution vertex v among the V high-resolution vertices: The solution form is w = dφ -1 The equation of , where: d is the P-dimensional identity matrix; and φ is a PxP dimensional matrix of weighted radial basis functions (RBFs), wherein each element 0 of φ is based at least in part on a training feature map parameter for comparing the P training feature maps; Thereby obtaining a PxP dimensional matrix of RBF weights w corresponding to the high-resolution vertex v; An approximate model comprising a set of V RBF weight matrices w is thereby defined.

54. The method of claim 53 or any other claim herein, comprising determining a plurality of low-resolution W-handle vertices, wherein the W-handle vertices are a low-resolution subset of the V-high-resolution vertices, wherein W <V。 55. The method of claim 54 or any other claim herein, wherein determining the training feature map including a corresponding plurality of training feature map parameters comprises: determining a training feature map geometry corresponding to the training face pose, the training feature map geometry comprising a plurality of F feature edges defined between training pose positions of corresponding pairs of low-resolution handle vertices of a plurality of low-resolution W handle vertices of the training pose; The plurality of training feature map parameters are determined, each of the plurality of training feature map parameters being based at least in part on some or all of the F feature edges of the training face poses and some or all of the feature edges of the neutral pose.

56. The method of claim 54 or any other claim herein, wherein determining the training feature map including a corresponding plurality of training feature map parameters comprises: determining a training feature map geometry corresponding to the training face pose, the training feature map geometry comprising a plurality of F feature edges defined between corresponding pairs of low-resolution handle vertices of a plurality of low-resolution W handle vertices of the training pose; determining the plurality of training feature map parameters as a plurality of F training feature map parameters corresponding to the F feature edges, each training feature map parameter of the plurality of F training feature map parameters being based at least in part on a corresponding feature edge of the training facial pose and a corresponding feature edge of the neutral pose; Thereby, the P training feature maps are obtained, and each of the P training feature maps includes a corresponding plurality of F training feature map parameters.

57. The method of claim 56 or any other claim herein, wherein determining a plurality of F-training feature map parameters corresponding to the F-training edges comprises: For each training feature map parameter in the plurality of F training feature map parameters, the training feature map parameter is determined using an equation of the following form f i =(||p i，1 -p i，2 ||-l i ) / l i (1) Where: f i is with the i th The i-th feature edge corresponding to th Training feature map parameters; p i，1 and p i，2 It is defined as i The training pose position of the handle vertex at the endpoint, l i is the corresponding i-th position in the neutral posture th The length of the feature edge.

58. The method of claim 53 or any other claim herein, wherein determining a training feature map comprising a corresponding plurality of training feature map parameters comprises determining one or more of: deformation gradients based at least in part on the training face poses and the neutral pose; pyramid coordinates based at least in part on the training face poses and the neutral pose; triangle parameters based at least in part on the training face poses and the neutral pose; and 1-ring neighborhood parameters based at least in part on the training face poses and the neutral pose.

59. The method of any one of claims 55 to 57 or any other claim herein, wherein: For each high-resolution vertex v in the V high-resolution vertices, the weight of the weighted radial basis function (RBF) in the matrix φ is determined at least in part based on a proximity mask, which assigns uniform values ​​to feature edges surrounding the vertex v and assigns attenuated values ​​to feature edges farther away from the vertex v.

60. The method of claim 59 or any other claim herein, wherein the proximity mask assigns exponentially decaying values ​​to feature edges that are farther from the vertex v.

61. The method of any one of claims 59 to 60 or any other claim herein, wherein the proximity mask is determined according to an equation of the form: in: l i is the neutral posture i th The length of the feature edge; L v，i is the distance from the vertex v to the neutral pose i th is the sum of the neutral pose distances of the endpoint locations of the feature edges; and β is a configurable scalar parameter that controls the decay rate.

62. The method of any one of claims 59 to 61 or any other claim herein, comprising: if determining that the i th If the calculated proximity mask of the feature edge is less than the threshold, the i-th th The proximity mask for feature edges is set to zero.

63. A method according to claim 59 or any other claim herein, wherein the proximity mask comprises assigning non-zero values ​​to a configurable number of feature edges that are relatively closer to the vertex v, and assigning zero values ​​to other feature edges that are relatively farther away from the vertex v.

64. The method of any one of claims 55 to 57 and 59 to 63 or any other claim herein, wherein the PxP dimensional matrix φ of the weighted RBF is in the kth th Column and l th The element at the row (φ v，k，l ) is given by an equation of the form: in: γ is the RBF kernel function; f k，i is the kth th Training posture i th Training feature map parameters of feature edges; f l，i It is the first th Training posture i th The training feature map parameters of the feature edge; α v，i Based on the i th The feature edge is assigned to the i-th vertex v based on its proximity to the high-resolution vertex v. th The weight of the feature edge.

65. The method of claim 64 or any other claim herein, wherein the RBF kernel function γ is a biharmonic RBF kernel function.

66. Use of an approximation model according to any one of claims 53 to 65 to determine a high resolution texture for rendering a target facial expression.

67. Use of an approximation model according to claim 66, comprising the features of any one of claims 1 to 52.

68. A system comprising one or more processors configured by appropriate software to perform the method of any one of claims 1 to 67 and / or any part thereof.

69. Include any block diagram, action, combination of blocks and / or actions, or sub-combination of blocks and / or actions described herein.

70. An apparatus and / or system comprising any feature, combination of features or sub-combination of features described herein.

Images