Fast and deep facial deformation

Abstract

To provide a method and a system that generate the mesh deformation of a facial model.SOLUTION: According to at least one embodiment, a method for generating a mesh deformation of a facial model includes the steps for: generating a first plurality of deformation maps by applying a plurality of first neural network-trained models; extracting a plurality of first vertex offsets based on the first deformation maps; and applying the first vertex offsets to a neutral mesh of the facial model to generate the mesh deformation of the facial model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Application No. 17 / 065,423, filed October 7, 2020, and U.S. Provisional Patent Application No. 63 / 022,398, filed May 8, 2020, the contents of all of which are incorporated herein by reference in their entireties. [Background technology]

[0002] Computer animation and other computer modeling applications combine two-dimensional (2D) or three-dimensional (3D) models of objects or characters with their corresponding programmed or keyframed movement. These models consist of geometric vertices, faces, and edges in a three-dimensional coordinate system. One form of computer animation involves attaching the 3D model to a transformation hierarchy that approximates a skeleton to represent the object's or character's movement. The skeleton is surrounded by a surface mesh that represents the boundary of the object or character. The 3D surface mesh can have high resolution and can contain polygons or faces of various sizes that make up the surface of the mesh, which can be shaped and deformed as required by the animator or other operator to achieve the desired result.

[0003] When comparing film-quality character rigs with rigs designed for real-time use, there is a clear difference in the quality of mesh deformations. Real-time rigs are often constrained by computational resources and trade realism for performance. Film-quality rigs do not have these limitations, and character riggers may make their rigs as complex as necessary to achieve realistic deformations.

[0004] Specifically, the level of detail included in character rigs for interactive applications such as video games and virtual reality is limited by computational constraints (e.g., mobile devices may have less processing power than networks of computer servers). This type of rig must run in real time and therefore must be able to evaluate at a rate that supports user interactivity without excessive delay. Because of this limitation, character rigs often lack a high level of realism. Film-quality character rigs, on the other hand, are not limited by computational constraints and their mesh deformations can appear more realistic.

[0005] Film-quality character rigs are not intended for interactive applications because film-quality rigs require greater computational power. With significant effort in optimizing and parallelizing rig evaluation, a single film-quality rig can run in real time on high-end hardware. While using these high-quality rigs is desirable to improve the realism of interactive applications, plugging computationally intensive rigs directly into interactive applications is impractical due to the interactive real-time requirements.

[0006] Therefore, existing systems and methods fail to provide a solution that reduces the computations required for mesh deformation for film-quality rigs for use in real-time applications. Therefore, it would be advantageous to determine mesh deformation for film-quality rigs (or near-film-quality rigs) for use in real-time applications.

[0007] Furthermore, film-quality characters typically display highly complex and expressive facial deformations, and the underlying rigs used to animate a character's facial deformations are often computationally expensive, requiring high-end hardware to deform the mesh at interactive rates. Summary of the Invention

[0008] Aspects of the present disclosure are directed to rigs for a character's face (or head). For example, according to at least one embodiment, a deep learning-based approximation method for complex film-quality facial rigs is disclosed. The disclosed method significantly reduces deformation evaluation time and provides a differentiable model for use with inverse kinematics. According to at least one embodiment, the method uses a convolutional neural network (CNN) to approximate mesh deformations of the character's face.

[0009] Embodiments of the present disclosure include systems and methods for approximating mesh deformations of character facial rigs. The embodiments of the present disclosure lead to significant reductions in evaluation time compared to original deformation calculations for film-quality character rigs. Various embodiments of the present disclosure include a data-driven approach to learning computationally inexpensive approximations for character facial rigs that compress computations sufficiently so that film-quality rigs can be approximated in real-time interactive applications.

[0010] According to at least one embodiment, a method for generating a mesh deformation of a facial model includes generating a first plurality of deformation maps by applying a first plurality of neural network-trained models, extracting a first plurality of vertex offsets based on the first plurality of deformation maps, and applying the first plurality of vertex offsets to a neutral mesh of the facial model to generate a mesh deformation of the facial model.

[0011] According to at least another embodiment, a system for generating a mesh deformation of a facial model includes one or more controllers configured to generate a first plurality of deformation maps by applying a first plurality of neural network training models, extract a first plurality of vertex offsets based on the first plurality of deformation maps, and apply the first plurality of vertex offsets to a neutral mesh of the facial model to generate the mesh deformation of the facial model.

[0012] According to at least another embodiment, a machine-readable non-transitory medium has stored thereon machine-executable instructions for generating a mesh deformation of a face model, the instructions including generating a first plurality of deformation maps by applying a first plurality of neural network training models, extracting a first plurality of vertex offsets based on the first plurality of deformation maps, and applying the first plurality of vertex offsets to a neutral mesh of the face model to generate a mesh deformation of the face model.

[0013] According to at least another embodiment, a method for generating an approximation of a deformation of a face model includes providing a first plurality of vertices to a first network to generate a first rig parameter pose; providing a second plurality of vertices to a second network to generate a second rig parameter pose; and processing the rig parameter pose and the second rig parameter pose to generate a composite rig parameter pose, wherein the composite rig parameter pose corresponds to the approximation of the deformation of the face model.

[0014] These and other aspects and features of the present disclosure will become more apparent from the following description of the embodiments, taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0015] [Figure 1A] FIG. 10 is a diagram showing a mesh of a character model. [Figure 1B] FIG. 10 is a diagram showing a mesh of a character model. [Figure 2] 10A-10C illustrate side-by-side comparisons of face mesh deformations including deformations approximated according to embodiments of the present disclosure. [Figure 3] FIG. 1 is a diagram of an approximation model according to at least one embodiment. [Figure 4A] FIG. 1 illustrates an example of a neural network for a coarse approximation model according to one or more embodiments. [Figure 4B] FIG. 1 illustrates an example of a neural network for a refined approximation model according to one or more embodiments. [Figure 5] FIG. 1 illustrates an example of rigid body approximation according to at least one embodiment. [Figure 6] FIG. 1 illustrates example poses from training data. [Figure 7] FIG. 1 is an illustration of an inverse kinematics (IK) model according to at least one embodiment. [Figure 8]FIG. 10 illustrates an example of mesh segments used during coarse modeling in accordance with at least one embodiment. [Figure 9] FIG. 10 illustrates an example of mesh segments used during refined modeling in accordance with at least one embodiment. [Figure 10] FIG. 10 is a diagram showing approximate transformations for a character pose including wrinkles. [Figure 11] FIG. 10 shows a side-by-side comparison of normalized error visualization. [Figure 12] FIG. 10 shows a side-by-side comparison of the ground truth mesh and approximate deformation for several example control point configurations. [Figure 13] FIG. 1 shows a frame from a recording and the resulting deformed mesh from the input. [Figure 14] FIG. 10 illustrates an example of transferring one mesh segment of a coarse approximation model onto a new mesh with a different topology. [Figure 15] 1 is a flowchart illustrating a method in accordance with at least one embodiment of the present disclosure. [Figure 16] 1 is a flowchart illustrating a method in accordance with at least one embodiment of the present disclosure. [Figure 17] FIG. 1 is a diagram of a system implemented in accordance with one embodiment of the present disclosure. [Figure 18] FIG. 1 is a diagram of an apparatus implemented in accordance with one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0016] In the following detailed description, reference may be made to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific embodiments of the present invention. It will be understood by those skilled in the art that other embodiments may be utilized and structural, electrical, and procedural changes may be made without departing from the scope of the present invention. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same elements.

[0017] While the present disclosure is presented in the context of 3D animation applications, other implementations of the systems, media, and methods described herein are contemplated, including, but not limited to, various interactive geometric modeling applications involving the transformation of geometric models in 2D or 3D coordinate systems, as well as the production and modification of geometric models, including, but not limited to, rigging, animation, architecture, automotive design, consumer product design, virtual reality applications, augmented reality applications, game development, visual effects, 3D printing, etc. Any reference in this disclosure to a 3D model or a geometric model or components thereof in 3D space will be understood to include the disclosure as applicable to 2D models and 2D spaces.

[0018] In this disclosure, the terms "character," "object," or "model" may include any suitable computer representation or geometric model of part or all of an entity, such as a character, a scene, an inanimate object, or any other virtual, modeled, and / or animated entity. An object may be composed of, for example, geometric vertices, edges, and / or faces in a 2D or 3D coordinate system.

[0019] A "point," "vertex," "edge," or "component" may include one of a set of elements used to determine the shape, movement, appearance, and / or absolute or relative position of a 2D or 3D model, including, but not limited to, the vertices, points, edges, and faces of a 2D or 3D model. Any description in this disclosure of one of a point, vertex, edge, or face is understood to apply equally and / or analogously to other of points, vertices, edges, or faces, where appropriate, as understood by one of ordinary skill in the art, and the use of one term over another within this disclosure is not to be considered a limitation solely on the term used.

[0020] By way of background, most character rigs are designed with a motion system and a deformation system. The motion system is responsible for mapping the rig parameters for a pose to a skeleton made up of bones and joints. The deformation system then maps the skeleton to the character's final mesh for the input pose. Figure 1A shows an example of a 3D character 100 in a rest pose that can be configured into a rig, in this case a "T" pose. A rig can include skeleton bones and joints that provide points of control and articulation for an artist or other operator.

[0021] Figure 1B is a diagram illustrating an example of points, or vertices, of a character mesh 101 corresponding to the 3D character depicted in Figure 1A. As shown in Figure 1B, adjacent vertices of the mesh will often be grouped together and assigned to the same bones of the skeleton. Furthermore, more vertices 102, i.e., vertices covering a larger area of ​​the mesh, can be assigned to larger bones of the skeleton with less articulation detail (e.g., the large bones of the arms and legs), while fewer vertices 103, i.e., vertices covering a smaller area of ​​the mesh, can be assigned to smaller bones with more articulation detail, such as the small bones of the wrist, hand, and fingers.

[0022] Film-quality characters typically exhibit highly complex and expressive facial deformations. The underlying rigs used to animate the deformations of a character's face (or more generally, a character's head) are often computationally expensive, requiring high-end hardware to deform the mesh at interactive rates. Disclosed herein, according to at least one embodiment, is a method using neural networks to approximate the mesh deformations of a character's face. According to at least one particular embodiment, one or more of such neural networks can be (or can include) a convolutional neural network (CNN). For tested models, the approximation can run up to 17 times faster than the original facial rig while maintaining a high level of fidelity to the original rig.

[0023] Also disclosed, according to at least one embodiment, is an extension to the approximation to handle high-frequency deformations, such as fine skin wrinkles. While the original animation rig implementation relies on an extensive set of proprietary libraries, making it difficult to install outside of an in-house development environment, the approximation according to the disclosed embodiments can rely on the widely available and easily deployed TensorFlow library. In addition to enabling high-frame-rate evaluation on more modest hardware and a wide range of computing environments, the significant speed improvement also enables interactive inverse kinematics on the animation rig. The technique and its applicability will be demonstrated through interactive character posing and real-time facial performance capture.

[0024] Character facial rigs for video games and other real-time applications are often controlled by a set of bones or blendshapes. These rigs can be computed quickly, but typically sacrifice fine detail for speed. Representing subtle deformations with such real-time rigs is difficult, often necessitating additional layers of computation on top of the underlying rig. These additions, which increase the level of detail in mesh deformations, include pose-space deformations and wrinkle maps. However, despite these improvements, the level of detail in film-quality facial rigs is clearly superior compared to real-time rigs. The primary reason film-quality facial rigs include more sophisticated mesh deformations is because they are not constrained by the same real-time requirements.

[0025] Facial rigs for film require a significant amount of computation to create realistic, expressive mesh deformations. These facial rigs can run at rates of 10–30 frames per second (FPS) when evaluated on high-end machines, but will run much slower on consumer-level devices. Furthermore, entities such as animation studios typically develop in-house rigging software on which to build characters. Such rigs are limited to custom animation engines, and porting characters for use outside of the in-house software can be difficult and time-consuming. For this reason, film-quality characters are rarely used outside of the films for which they are designed due to computational limitations and portability.

[0026] Methods have been proposed to approximate body deformations for film-quality character rigs. However, these methods use the underlying skeleton to approximate the deformations. Unlike body rigs, facial rigs do not have extensive skeletal structures available for deformation approximation. For example, facial features such as those corresponding to the areas around the nose, eyes, or mouth may not have extensive skeletal structures available.

[0027] Aspects of the present disclosure are directed to addressing this limitation. For example, according to at least one embodiment, a method for approximating the deformation of a facial mesh is disclosed. The approximation accurately calculates the deformation of the facial mesh, including the skin, interior of the mouth, teeth, and other structures. The approximation can use a neural network to exploit the types of deformations seen in the facial mesh. For example, a CNN is one type of neural network that can be employed. However, it is understood that other types of neural networks can be used. The method can calculate the deformation in three separate parts: a first approximation (e.g., a coarse approximation), an additional approximation (e.g., a fine approximation), and a further approximation (e.g., an approximation to the rigid components of the mesh). The method can preserve high-frequency details of the mesh while running up to 17 times faster than a tested production rig (see FIG. 2).

[0028] FIG. 2 shows a side-by-side comparison of facial mesh deformations using a coarse approximation according to at least one embodiment (see FIG. 2(c)), a fine approximation according to at least one embodiment (see FIG. 2(b)), and an approximation generated by linear blend skinning (LBS) (see FIG. 2(d)). Techniques such as LBS calculate the deformation of a mesh from its rest pose as a weighted sum of skeletal bone transformations applied to each vertex. However, these deformations can suffer from volume loss, known as the "candy wrapper" problem, and there are certain deformations that these techniques cannot achieve, such as skin sliding, muscle bulging, and cloth wrinkling.

[0029] Figure 2(a) shows the final deformation achieved by computing the full mesh function using high performance computing hardware given unlimited (or extensive) time resources, which for the purposes of this discussion may be referred to as the "original deformation function" or "ground truth" deformation. Compared to the "ground truth" deformation in Figure 2(a), the most notable difference shown in the second row can be observed around the nose region of the mesh.

[0030] As mentioned above, the approximation model according to at least one embodiment can be implemented in TensorFlow, an open-source machine learning library, which allows the facial rig to exceed the inherent limitations of the original rigging software used to author the character and enables the model to be deployed on a wider variety of devices and applications. Furthermore, the rig approximation can be more easily evaluated on both CPUs and GPUs without additional effort. Because the model disclosed herein can be neural network-based, the corresponding approximation is fully differentiable. The usefulness of this property (or aspect) is demonstrated by developing an inverse kinematics-based posing application and a facial performance capture system.

[0031] Facial deformation systems for animated characters vary greatly in method and complexity. Often, facial models combine multiple deformation methods to achieve the final result. In these situations, one of the simplest and fastest ways to calculate deformation is based on an underlying skeleton. Skeletal subspace deformation (SSD), also known as linear blend skinning (LBS), is popular due to its simplicity and speed. Due to well-known drawbacks of SSD, including the candy wrapper effect, improvements such as multi-weight envelope and dual quaternion skinning have been investigated, which improve quality without significantly impacting evaluation speed. This class of methods is often used as the base deformation system for a character's body, but is often combined with other methods for rigging a character's face. A more common facial deformation system is blendshapes, which linearly combine a set of artist-created facial expressions. While this method is also fast to evaluate, it has limitations on its own for film-quality character rigging, requiring the keying of hundreds of blendshapes for each frame of animation. Another approach to constructing face models relies on physics-based deformation to facilitate the generation of more realistic and lifelike poses. For complex face models, all of these and other techniques can be combined, but this generally results in high cost and slow evaluation speed.

[0032] Real-time applications may require building a facial deformation model that preserves detail without incurring significant computational costs. One approach utilizes pose-space deformations in a hybrid approach that uses SSd to compute a base deformation and learns a model to compute high-fidelity nonlinear details, such as wrinkles, that are applied on top of the base. To efficiently compute physics-based deformations, one approach refines rig-space physics for real-time results on production-quality character rigs. These approaches can be sufficient to achieve high performance for systems built on them. However, at least one aspect of the present disclosure is directed to finding efficient representations for existing high-quality rigs, where computing deformation models on low-power hardware can be slow, without the need to optimize complex character rigs.

[0033] Various techniques exist for approximating existing deformation models given a set of example poses. Many of these techniques aim to build more computationally efficient representations of the deformation function. One skinning decomposition technique finds bone transformations and skin weights for a skeletal subspace deformation model given a set of example poses. Similarly, other techniques also find SSD representations of the deformations, but organize them in a skeletal hierarchy to facilitate subsequent animation. Another technique learns a skinning mesh from example data via SSD for control point animation. For reasons explained above, bone-based deformation systems are not an optimal way to represent facial deformations. Therefore, such methods alone are not suitable for at least some aspects discussed herein.

[0034] Sphere-Meshes break down mesh animation into a set of animated spheres, which can later be keyframed for animation. This approach is also unsuitable for high-quality character animation due to the difficulty of representing fine details. Another approach specifically targeted at facial animation learns new parametric rig models, such as blendshapes, from example poses. Another approach creates facial rigs based on statistical models so that the appearance of the rig closely matches the appearance of the recorded actor. All of these methods learn entirely new rig representations with different controls than those present in the original rig. At least one aspect of the present disclosure is directed to approximating existing facial rigs and maintaining the same controls so that artists do not need to relearn the control parameters.

[0035] While previous work attempting to approximate existing rig functions often assumes an underlying blendshape model or underlying skeletal structure, according to at least one embodiment, Applicants' method does not make such strong assumptions about facial rigs. EigenSkin efficiently computes high-fidelity nonlinear deformations on the GPU via an error-optimal pose-dependent displacement base constructed from example meshes. The approach assumes an underlying SSD representation of a given rig and uses this in its computations. One approach learns an extended SSD skinning model with additional joints from an existing SSD rig. One approach assumes an underlying skeleton and uses the skeleton's deformations as a base onto which fine-detail nonlinear displacements are superimposed. According to at least one embodiment, the deformation model is learned without requiring (or otherwise utilizing) any assumptions about the skeletal system, which is appropriate for complex facial rigs.

[0036] To support inverse kinematics (IK) of facial rigs in real time, efficient and accurate inverses of rig functions may be required to calculate the pose of a character given a set of constraints. In computer animation and robotics, inverse kinematics refers to the mathematical process of calculating the variable joint parameters necessary to place the end of a kinematic chain (e.g., a robotic manipulator or the skeleton of an animated character) at a given position and orientation relative to the start of the chain. Based on these joint parameters, the position and orientation of the end of the chain (e.g., a character or robot hand) can be calculated directly, usually through multiple applications of trigonometric formulas. This process is called forward kinematics. However, in general, the inverse operation (i.e., inverse kinematics) can be more difficult.

[0037] Inverse kinematics is also used to reconstruct the motion of an object in the world from any other data, such as a film of that motion, or a film of the world as seen by a camera performing that motion, as occurs, for example, when the filmed motion of a human actor is replicated by an animated character.

[0038] Due to the complexity of facial rigs, traditional solutions to the IK problem are not easily applicable to film-quality face models because they require differentiable rig functions. There has been work on computing blendshape parameters from landmarks. However, aspects of the present disclosure are directed to enabling the inversion of arbitrary black-box rig functions.

[0039] Solutions to this problem have been explored. One approach utilizes an iterative optimization technique, however, it does not completely ignore the rig because it is designed to optimize the inversion of the pose-space deformation rig. Another approach has successfully inverted the black-box rig function using two nonlinear techniques: Gaussian process regression and feedforward neural networks. In contrast, at least one aspect of the present disclosure is directed to approximating the original rig function using deep learning methods. Due to the neural network, the gradient of the rig function can be estimated through the rig approximation and then used to estimate the inverse rig function.

[0040] Deep convolutional methods have been developed for data-driven mesh regression problems. These methods leverage the power and flexibility of deep neural networks for tasks ranging from face reconstruction and facial animation to cloth simulation. One approach to applying CNNs to meshes is based on defining mesh convolution operations. One approach introduces graph convolutional autoencoders, while another uses a similar idea to generate 3D faces. MeshCNN defines specialized convolution and pooling operations for triangular meshes. At least one aspect of this disclosure focuses on improving efficiency, and using such mesh convolutions would be computationally intensive in this regard. Traditional CNNs operate on 2D images and feature maps. To reconstruct 3D deformations using these models, a mapping must be created between the feature map space and vertex positions. One approach applies convolutions by parameterizing the mesh around a small local area. Another approach applies CNNs by projecting the mesh onto a spherical domain and "cropping" this projection. Other approaches use texture coordinates (or UV coordinates) to map vertex positions into 2D space. In this way, the network learns to predict a 2D feature map, but represents 3D coordinates. Convolutional neural networks are successful because the spatial coherence of vertex positions is preserved in the transformation space. Other approaches generate UV maps from perspective projections or scans. Because at least one aspect of Applicants' approach assumes a complete character rig, various embodiments disclosed herein use artist-created UV maps (or UV coordinates) to calculate vertex positions from the 2D feature space.

[0041] Features of various embodiments of the present disclosure will now be described in more detail.

[0042] Given a character's facial rig having a polygon mesh, let V be the set of vertex coordinates in the mesh with |V|=n vertices. Let p represent the character's rig parameters, and let V=r(p) be the rig function that maps the parameters to the deforming mesh. At least one embodiment of the present disclosure is directed to approximating this rig function r(p).

[0043] According to at least one embodiment, the approximation method uses artist-created texture coordinates of the face mesh.

number

[0044] According to at least one embodiment, a deformation map is a construct (e.g., a 3D image) having three channels of information. For purposes of explanation, it may be useful to think of a deformation map as similar to a color image having three channels of color information. For example, for a particular pixel, such a color image may have red color information, green color information, and blue color information. Similarly, according to at least one embodiment, a deformation map has x-, y-, and z-coordinate information for a particular vertex. The x-, y-, and z-coordinate information relate to the location of the vertex in the 3D image corresponding to the deformation map.

[0045] Thus, the deformation map holds x-, y-, and z-coordinate information for each of a plurality of pixels (e.g., each pixel in a 3D image). With respect to the vertices of a mesh, the location of a particular vertex may not necessarily coincide with the location of any particular pixel in the deformation map. Thus, according to at least one embodiment, texture coordinates corresponding to the vertex can be used to determine the location of the vertex based on the deformation map. As an example, the texture coordinates of a particular vertex can be used to sample the deformation map to identify particular pixels (e.g., adjacent pixels). For example, based on the texture coordinates, it can be determined that a particular vertex is located between (e.g., halfway between) the respective locations of two particular pixels in the deformation map. To determine the location of the vertex, a weighted sum of pixel information in the deformation map (e.g., a weighted sum of x-coordinate information of two pixels, a weighted sum of y-coordinate information of two pixels, and a weighted sum of z-coordinate information of two pixels) can be used to determine an offset relative to the location of the vertex in the mesh.

[0046] Therefore, deformation maps are sampled in texture coordinates to approximate vertex positions in the mesh. Many parameters of a facial rig deform local regions of the mesh, and rig parameters can be seen as local operations on the mesh. By design, CNNs perform local computations on feature maps. Assuming that local information about a mesh is stored in texture coordinates, CNNs may be an optimal way to approximate rig functions.

[0047] According to at least one embodiment, a method for generating an approximate mesh deformation of a face model includes two stages: coarse approximation and fine approximation (see FIG. 3 ). FIG. 3 is a diagram of an approximation model according to at least one embodiment. Rig parameters 302 are used as inputs to a neural network (e.g., a convolutional neural network) that generates a deformation map for each of one or more mesh segments. For example, one or more of the rig parameters 302 are input to a convolutional network in one or more coarse approximation models 304, which generates deformation map 306. As another example, one or more of the rig parameters 302 are input to a convolutional network in a fine approximation model 324, which generates deformation map 326. In one aspect, the rig parameters 302 are not the model itself, but rather are inputs used to influence the model and generate the deformation map. For example, the rig parameters 302 may include inputs for controlling movement of the lips, eyebrows, nose, etc.

[0048] For each deformation map, vertex offsets can be extracted by interpolating the deformation map at each vertex position in texture coordinate space. For example, bilinear interpolation can be employed. According to at least one embodiment, for each deformation map 306 generated by the coarse approximation model 304, vertex offsets are extracted by bilinear interpolating the deformation map at each vertex position in texture coordinate space. As previously described, a particular vertex can be located between (e.g., halfway between) the respective positions of two pixels in a deformation map (e.g., one of the deformation maps 306). A weighted sum of pixel information in the deformation map 360 can be used to determine the vertex position. For example, the average of the x-coordinate information of two pixels, the average of the y-coordinate information of two pixels, and the average of the z-coordinate information of two pixels can be used to determine the offset of a vertex position in a mesh.

[0049] In adder 310, the extracted offsets are added to the vertices of the neutral pose of mesh 312 to arrive at the desired deformation.

[0050] Bilinear interpolation is just one example of a form of interpolation that may be employed to extract the vertex offsets. It will be appreciated that other forms of interpolation (e.g., cubic interpolation) may be employed.

[0051] The coarse approximation model 304 can operate on the entire mesh 312. To reduce the corresponding execution time (e.g., so that the coarse approximation model runs faster), the coarse approximation model 304 includes multiple CNNs that output low-resolution deformation maps (e.g., deformation map 306 characterized by a relatively low resolution), which can result in loss of high-frequency details of the deformation.

[0052] To address this loss in detail, according to at least one particular embodiment, the approximated mesh deformation further includes a fine approximation model 324. The fine approximation model 324 includes a CNN that outputs a higher resolution deformation map (e.g., a deformation map 326 characterized by a higher resolution than that of the deformation map 306). As described in more detail below, the deformation map 326 is applied to less than all regions of the mesh 312. For example, the deformation map 326 may be applied to only selected regions of the mesh 312.

[0053] For example, for each deformation map 326 generated by the refined approximation model 324, vertex offsets are extracted by performing an interpolation of the deformation map at each vertex position in texture coordinate space. In general, the structure of the deformation map 326 may be similar to that described above with reference to the deformation map 306. Similarly, the extraction of vertex offsets based on the deformation map 326 may be similar to that described above with reference to the deformation map 306. Also, as described above with reference to the deformation map 306, the interpolation may be bilinear, cubic, or any other suitable form of interpolation. In this regard, at most, a particular subset 308 of all active vertices may be used. In adder 314, the extracted offsets are added to the output of adder 310. At the output of adder 314, the vertices of the deformed mesh 316 are provided.

[0054] Thus, the fine approximation model 324 can approximate these high frequency deformations by focusing only on dense regions of vertices in the mesh 312. To further improve the efficiency of the model, according to at least one further embodiment, segments of the mesh that undergo only rigid body rotations and translations within the rig function are identified. Such identified segments can be approximated with faster rigid body approximations instead of more complex CNN approximations (e.g., the CNN-based approximation of the coarse approximation model 304 and the CNN-based approximation of the fine approximation model 324).

[0055] 3 illustrates the approximation model including a coarse approximation model 304 and a fine model 324, it is understood that the approximation model need not include both. By way of example, according to at least one embodiment, the approximation model includes one or more coarse approximation models 304 but does not include a model similar to the fine model 324.

[0056] The coarse approximation model 304 will now be described in more detail with reference to at least one embodiment. A facial mesh (e.g., mesh 312 in FIG. 3) can be divided into multiple segments, which is common in artist-created facial rigs. Each vertex of the mesh is assigned to one mesh segment. In this manner, each vertex is assigned to one mesh segment. m denotes the total number of mesh segments, and V k and U k and denote the set of vertex positions and the set of texture coordinates, respectively, for a particular mesh segment k. While features are described herein with reference to meshes having multiple mesh segments, it is understood that the coarse approximation model 304 can also operate on non-segmented facial rigs. In this situation, m equals 1, and all vertices of the entire mesh will be assigned to exactly one segment.

[0057] The coarse approximation model 304 can calculate the deformation mesh by first generating deformation maps 306, where one deformation map 306 is generated for each mesh segment in the facial rig. Then, the vertex positions can be calculated based on the maps. For a mesh segment k, the corresponding function I k =f(p;θ k ) computes a deformation map for a mesh segment given rig parameters p (e.g., rig parameters 302). The function f can be a neural network (e.g., CNN) consisting of multiple dense and convolutional layers, where the neural network computes a deformation map based on θ k (see Figure 3), where θ k represents the optimal model parameters for mesh segment k.

[0058] Figure 4A illustrates an example of a neural network for a coarse approximation model according to at least one embodiment. Referring to Figure 4A, the neural network includes dense layers 402-1 and 402-2 and convolutional layers 404-1, 404-2, 404-3, and 404-4. Convolutional layers 404-1, 404-2, and 404-3 use 3x3 kernels, while the final convolutional layer (convolutional layer 404-4) uses a 1x1 kernel. All layers except the final convolutional layer use a leaky ReLU activation function, with no activation function applied to the final layer. All non-dense layers are square in the image plane.

[0059] Upsampling can be achieved by nearest neighbor interpolation (see Figure 3). Vertex offset Δ k is the texture coordinate U k At I k The deformation map is calculated by sampling the deformation map calculated by △ k =g(I k ;U k ) and outputs (or generates) vertex offsets. Each vertex is assigned to a single mesh segment, so the vertex offsets for the full mesh are △ = ∪ k∈{1,...,m} △ k Continuing to refer to Figure 3, a coarse approximation is calculated by adding (e.g., with adder 310) the vertex offsets to the vertices of the neutral pose of mesh 312 to calculate the final vertex positions of the mesh.

[0060] Given the approximated model, a loss function is defined to find the optimal model parameters θk. According to at least one embodiment, a loss function is proposed that penalizes both inaccuracies in the approximated vertex positions and inaccuracies in the facial normals on the mesh. Given a target mesh V and approximated vertex offsets Δ, the loss function can be defined in equation (1) as follows:

number

[0061] In the loss function of equation (1), n i denotes the normal of face i in mesh V, TIFF2025183216000004.tif13150 is the vertex position V 0 + △ and the normal of face i in an approximated mesh with a total of f faces in the mesh topology. L1 loss is used instead of L2 loss because it produces sharper features. The edge-to-edge mapping of vertex offsets from rig parameters (e.g., rig parameters 302 in FIG. 3) is learned without the need to supervise intermediate deformation maps. Furthermore, according to at least one embodiment, texture coordinates are not optimized; instead, artist-created coordinates are relied upon.

[0062] With respect to coarse approximations, examples of mesh segments for a particular mesh are described in more detail below with reference to FIG.

[0063] Because the coarse approximation model operates on separate mesh segments, the model may generate discontinuities across boundaries and / or seams between certain mesh segments. To address (e.g., minimize) this potential problem, the error function (e.g., the loss function in Equation (1)) may strongly penalize inaccurate surface normals and encourage smooth (or smoother) results along mesh segment boundaries. Penalizing normal error also suppresses low-amplitude, high-frequency errors that may be visually disturbing or distracting.

[0064] According to at least one embodiment, to aid in training the model, each network may be equipped with only a subset of rig parameters (e.g., subset of rig parameters 302). The subset includes all rig parameters that can deform any vertices in the mesh segment being approximated. All other rig parameters (e.g., inputs that do not affect the output) are excluded from the parameters input to the network. As a result, the network does not need to learn which parameters to ignore and avoids being adversely affected by noise provided by (or associated with) inputs that do not affect the output.

[0065] In the coarse approximation model, the resolution of the deformation map Ik is intentionally kept small (or relatively low) to reduce computational complexity. However, in texture coordinate space, vertices in dense regions of the mesh can be spaced apart by less than one pixel in the small deformation map. If high-frequency deformations, such as skin wrinkles, occur at such vertices, the coarse approximation model is unlikely to be able to accurately reproduce the deformations. The bottleneck in such situations is the resolution of the map output by the CNN (e.g., map 306 output by the coarse approximation model 304). To address this limitation, according to at least one embodiment, a fine model (e.g., the fine approximation model 324) is proposed to exclusively focus on specific regions of the mesh (e.g., dense regions of vertices).

[0066] The refined approximation model 324 will now be described in more detail with reference to at least one embodiment.

[0067] First, sets of vertices corresponding to regions of large (or larger) error in the approximation are identified. Vertex selection for the fine model is described in more detail below. Each identified set is then defined as a new mesh segment. For each vertex in such a new mesh segment, the texture coordinates are scaled to meet the full resolution of the fine deformation map (e.g., map 326 generated by the fine approximation model 324). As with the coarse approximation model, no vertex is assigned to multiple mesh segments (e.g., multiple new mesh segments defined for the purpose of the fine approximation).

[0068] Furthermore, not all vertices of a mesh (e.g., mesh 312 in FIG. 3) are assigned to new mesh segments with reference to the refinement step, e.g., only vertices located in regions of the mesh with high approximation errors are assigned to new mesh segments.

[0069] Figure 4B illustrates an example of a neural network for a fine approximation model in accordance with at least one embodiment. Referring to Figure 4B, the neural network includes dense layers 412-1 and 412-2 and convolutional layers 414-1, 414-2, 414-3, 414-4, 414-5, and 414-6. Convolutional layers 414-1, 414-2, 414-3, 414-4, and 414-5 use a 3x3 kernel, while the final convolutional layer (convolutional layer 414-6) uses a 1x1 kernel. All layers except the final convolutional layer use a leaky ReLU activation function, with no activation function applied to the final layer. All non-dense layers are square in the image plane.

[0070] m′ denotes the number of new mesh segments in the refinement step, and U k ′ denotes the new texture coordinates for a particular new segment k′. Similar to the notation used earlier for the coarse approximation model, the sampling of the fine approximation model for a new mesh segment k′ is expressed as δ k′ =g(f(p;θ k′ r);U k′ ′), and θ k′ r indicates the optimal model parameters for the new mesh segment k′. k′ approximates the residual between the vertex positions in the mesh (e.g., mesh 312 in FIG. 3) for the new mesh segment k′ and the output of the coarse model. For vertices not included in any of the new mesh segments, the fine approximation may be considered equal to zero, and this set of fine approximations is m′+1 = 0. Similarly to the vertex offsets obtained for the full mesh in the coarse approximation model (denoted as △), δ is the output tethered set δ k′ The fine model 324 can be trained using the same (or similar) loss function as described above with reference to the coarse approximation model (see Equation 1), where the loss is evaluated as L(V, △+δ).

[0071] With respect to fine approximation, examples of new mesh segments for a particular mesh are described in more detail below with reference to FIG.

[0072] According to at least one embodiment, the fine model generates a deformation map characterized by higher resolution relative to the deformation map generated by the coarse model. Alternatively, a global approximation can be performed by applying only the higher resolution fine model across the entire mesh and forgoing the use of coarse approximations. However, applying the fine model across the entire mesh would be much more computationally expensive due to the global increase in resolution and because the fine model uses deeper networks (see, e.g., the more convolutional networks in the fine model illustrated in FIG. 4B).

[0073] Vertex selection for fine models will now be described in more detail with reference to at least one embodiment.

[0074] To identify the set of vertices to be used for refinement, we can estimate the minimum approximation error for each vertex, given the resolution of the coarse deformation map and the texture coordinates of each mesh segment. We then cluster the texture coordinates, weighted by the estimate of the approximation error for each vertex. Vertices close to each cluster become the corresponding mesh segment in the refined model. Meanwhile, vertices far from (or far away from) the cluster centroid are omitted from the refinement stage.

[0075] The minimum approximation error can be estimated by first mapping the vertex positions through texture coordinates to a deformation map, and then sampling the deformation map at the coordinates to generate the vertex positions (sampled vertex positions). The map can be calculated by polyharmonic interpolation with a linear kernel by interpolating values ​​from texture coordinates to pixel coordinates. The vertex positions are calculated from the deformation map by bilinear interpolation. vi represents the original vertex positions, JPEG2025183216000005.jpg13150 can represent vertex positions sampled from the deformation map. A set of n samples, Vi = {v i 1 ,v i 2 ,...,v n i ), the approximation error can be estimated as shown in the following equation (2).

number

[0076] Then, k-means clustering is performed with the corresponding approximation error e iThis is done for the texture coordinates at each vertex, weighted by . The number of clusters can be determined using the elbow method, where each vertex is assigned to the nearest cluster centroid up to a user-specified distance. In the experiments conducted, vertices were assigned within a square whose length was 1 / 4 the width of the original texture coordinate space and centered on the cluster mean. This method worked well for the characters tested. As with the coarse approximation, a set of rig parameters can be calculated that can deform any vertex contained in these new mesh segments. Each fine model can be provided with only its specific input parameters.

[0077] Next, the computationally-efficient rigid body approximation feature will be described with reference to at least one embodiment.

[0078] In a character's face, there can be sections of the mesh that move rigidly, such as individual teeth, where the vertices of a given tooth can be thought of as moving together as a single unit. In the tested character, each tooth was modeled as a separate segment. Because the deformation of each tooth in the rig can be represented as a rotation and translation, approximating a linear transformation in the CNN model for each tooth would result in unnecessary computation. Instead, rigid movement can be estimated by computing a linear transformation from nearby vertices of the approximated mesh, as shown in Figure 5.

[0079] 5 illustrates an example of rigid body approximation according to at least one embodiment. Triangle 502 represents a rigid mesh segment, e.g., the rigid mesh segment identified by equation (3) presented below. Line 504 represents the nonlinearly deformed mesh segment, and the dots on line 504 represent vertices on the surface. In particular, dots 506 represent the set of vertices identified by equation (4) that best match the rigid transformation of triangle 502 over a large set of examples. Given the rest pose, as well as the positions of the vertices on nonlinear segment 504 in the deformed pose, the transformation R,t is calculated from the vertices represented by dots 506.

[0080] A transformation is then applied to triangle 502 to calculate its position in the deformed pose. Each rigid mesh segment is assigned to a subset of vertices approximated by the CNN model. The motion of these rigid segments is then estimated by solving the rigid transformation that best explains the motion of the corresponding subset of vertices from the CNN approximation. According to at least one embodiment, the rigid transformation is calculated after the coarse and fine approximations have been evaluated, since the calculation depends on the results of the approximation.

[0081] To identify the rigid transformation segments of the mesh, all k mesh segments provided by the facial rig author are considered. Then, n mesh transformation segments V = {V 1 ,V 2 ,...,V n}. The mesh is in a rest pose V 0 When , the vertex position V k 0 V k i The approximation error when rigidly transforming is calculated as shown in the following equation (3).

number

[0082] Equation (3) imposes a rigid rotation R on mesh segment k of sample i.k i and translation t i k This shows the difference in vertex positions when applying the error between samples.

number

[0083] V r i is the rigidly transformed mesh segment of sample i (i.e., e r <τ) can be expressed as R r i and t i r Let P denote the set of vertex indices of the mesh that are not included in the rigidly transformed segment. For each vertex j∈P, the transformation R over all samples i r i ,t i r The approximation error under is calculated using the following equation (4):

number

[0084] For a rigid mesh segment r, V δ 0 and V i δ is |V δ 0 The minimum approximation error ∈ r,j In the experiments performed, a value of c=10 was chosen. P Given the nonlinearly deformed vertices of , the vertex positions of the rigid mesh segment r are given by V r ′=V r 0 R′ δ +t′δ can be approximated as R′ δ and t' δ is the vertex position V′ δ This is the minimization factor of equation (3) for

[0085] The implementation of the coarse and fine approximation models will now be described in more detail with reference to at least one embodiment.

[0086] The respective models f(p;θ) for the coarse approximation and refinement stages k ) and f(p;θ k′ r All of the models can be implemented as deep neural networks with a series of fine layers following a convolutional layer. Figures 4A and 4B show an example of a neural network for a coarse approximation model and an example of a neural network for a fine approximation model, respectively. The networks are trained in two stages. In the first stage, the parameters θk corresponding to the coarse approximation are trained to minimize the loss L(V,Δ) expressed in equation (1). These models are trained with the Adam optimizer (see Diederik Kingma and Jimmy Ba. 2014 Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (12 2014)) using the momentum parameters proposed by the authors and a batch size of 8. The learning rate is set to 10. -3 After the model has converged, the learning rate is increased to 10 -4 After converging again, the learning rate is reduced to 10 -5 and run again until convergence. k are fully optimized, they are kept constant, and in the second stage, the fine model parameters θ k′ r is optimized with loss L(V, Δ+δ). The same (or similar) hyperparameters and training schedule are used for optimizing the fine model.

[0087] By training the approximate model, the rigid mesh segments (see Equation (3)) and the set of vertices assigned to each rigid mesh segment (see Equation (4)) are calculated using the original rig function. During model evaluation, the coarse and fine models are evaluated, and then the rigid transformation is calculated. The approximated vertex positions are used to calculate the rotation matrix and translation vector, which are then applied to the rigid mesh segments to create the resulting approximated mesh deformation.

[0088] To train a facial approximation model according to at least one embodiment, a large training data set may be required. The training data consists of pairs (p, V) of vertex positions of a deforming mesh output by rig parameters p and a rig function V=r(p). To generate the training data, existing animations can be augmented with multiplicative noise, and data balancing can be applied to prevent common poses found in the animation data from being over-represented in the training data.

[0089] Let A denote the set of poses from the training animation, and m denote the number of rig parameters in each pose. The training set can be expressed based on the following equation (5):

number

[0090] After the training set is generated, the data is balanced. The training data can be generated from existing animations, where certain facial expressions, such as neutral expressions, may occur more frequently than other poses in the data. A model trained using this dataset may overfit to frequently occurring facial expressions and may perform poorly when approximating other types of facial expressions. According to at least one embodiment, the training examples are sorted into bins, and random samples are drawn by uniformly randomly selecting bins and then uniformly randomly selecting samples within the bins.

[0091] To divide the data into bins, a small set of landmark vertices around key facial features such as the mouth, eyes, and nose can be manually labeled. For example, in our experiments, we manually identified roughly 20-30 landmark points for each character. For each pose p i ∈T, the landmark vertices V of the deformed mesh l i The landmark positions are collected. l 1 ,V l 2 ,...,V l |T| Principal component analysis (PCA) is used to project} into a one-dimensional space. This one-dimensional space is divided into intervals of equal length along the range of the projected data. Samples are then sorted into bins along the intervals. When extracting training samples, bins are selected uniformly at random, and samples are selected uniformly at random from each bin. In the experiments conducted, the data was divided into 16 bins.

[0092] Production facial character rigs are typically constructed in such a way that computing the gradient of vertex positions with respect to the rig parameters ∂V / ∂p is difficult and very time-consuming. Using the approximation model described herein, at least by embodiments, the estimation of the gradient becomes possible and potentially trivial due to automatic differentiation, a common feature in deep learning libraries. One useful application of this gradient is in inverse kinematics, where rig parameters are estimated to optimally deform a mesh to match user-specified control point positions.

[0093] Common solutions to inverse kinematics formulate it as an iterative optimization problem. These types of solutions require multiple gradient evaluations before converging on the optimal rig parameters. While an approximate model can be used to estimate ∂V / ∂p, calculating the gradient multiple times through the model for an iterative optimization method requires too much computation to run in real time. Instead, according to at least one embodiment, a feedforward neural network is disclosed. The feedforward neural network takes IK control points as input and outputs the corresponding rig parameters. During training, the network utilizes the approximate gradient but does not require ∂V / ∂p when evaluating with new inputs. As a result, the feedforward network can more easily calculate the desired rig parameters in real time.

[0094] A model of a feedforward neural network will now be described with reference to at least one embodiment.

[0095] C can denote the set of vertex indices corresponding to IK control points, and r C (p):R m →R |C|×3 is a rig parameter p given a subset of vertices V C As a result, the inverse kinematics problem can be expressed as equation (6) below.

number

number

[0096] The loss function used to train the model contains both a point-matching component to ensure that the deformed mesh closely matches the control points, and a regularization component to avoid large rig parameters that create unnatural poses. This loss is expressed as follows:

number

number

[0097] Ideal IK approximation model f IK This avoids learning erroneous correlations between specific rig parameters and control points. For example, if a user adjusts a control point above a character's left eye, the approximation model should avoid changing rig parameters related to the area around the mouth. According to at least one embodiment, this property is achieved by designing the IK approximation model as a combination of multiple networks. The control points are divided into separate sets based on facial regions. For example, all of the points above a character's right eye define one subset, and all of the points above the mouth define another subset. These points can then be divided manually.

[0098] For example, the control points can be divided into k subsets, C j can denote the control points in a particular subset j. According to at least one embodiment, the IK approximation model is composed of k separate feedforward networks in total. The input to network j is the control points JPEG2025183216000019.jpg7150, and the output is a set of rig parameters that can deform any of the vertices corresponding to the control points. The rig parameters can be estimated by multiple models. In this case, the final estimate can be the average of the outputs. It is understood that more sophisticated methods can be used to calculate the final values ​​of the rig parameters predicted by multiple networks.

[0099] However, for the rigs used in the various experiments conducted, value averaging worked well. For a character's face, only a small portion of the rig parameters can be shared between IK models. Regarding the shared parameters, almost all of these control large-scale deformations of the face, such as squashing or stretching the entire head. Because these controls cause large deformations across the entire area of ​​the mesh, IK models trained on control points of a small portion of the mesh will generally agree on parameter values ​​for this kind of global deformation. Therefore, simply averaging these parameters can produce reasonable results.

[0100] According to at least one embodiment, each network in the IK approximation model has 256 nodes in the first two layers and |R j can consist of three dense layers with nodes and |R j denotes the set of rig parameters estimated by IK model j. A leaky ReLU activation function is applied after the first and second layers. Also, no activation is applied to the output of the final layer, allowing the rig parameters to be output as any value.

[0101] 7 is a diagram of an IK model according to at least one embodiment. Control points 702 are divided into discontinuous subsets and provided to separate dense neural networks. For example, the control points are divided into subset 704-1 (corresponding to the eye region), subset 704-2 (corresponding to the contralateral eye region), and subset 704-3 (corresponding to the nose and mouth region). The control points of subset 704-1 are input to dense neural network 706-1. Similarly, the control points of subset 704-2 are input to dense neural network 706-2, and the control points of subset 704-3 are input to dense neural network 706-3.

[0102] Each of networks 706-1, 706-2, and 706-3 outputs a respective subset of poses. For example, network 706-1 outputs subset 708-1 of poses. Similarly, network 706-2 outputs subset 708-2 of poses, and network 706-3 outputs subset 708-3 of poses. The valid values ​​from the outputs (from output subsets 708-1, 708-2, and 708-3) are averaged together to generate a final averaged rig parameter pose 710. Similar to the facial approximation model, the IK model can be optimized with Adam using the same training schedule and balanced dataset described above.

[0103] As explained, the IK model is trained using control points from the deforming mesh calculated through the rig function. Therefore, the training data can only contain examples of control points that can be accurately matched with the appropriate rig parameters. However, when evaluating the IK model, the user may set control points that the rig cannot accurately match. To account for such cases, noise can be added to the control points during training. If the training samples are TIFF2025183216000020.tif7150, the new sample is tif13150. This new data point can be created by adding uniformly random noise to the position of each control point. Based on experimental data, it was observed that δ = 4 mm produces reasonable results. The IK model is trained with this new data VC, while all other aspects of the model training remain the same (or similar).

[0104] The results produced by the methods according to one or more embodiments will now be described in more detail.

[0105] One or more methods of approximation work, for example, in film-quality facial rigs used in computer-animated movie productions. The rig can deform through a combination of a freeform molding system and a curve-based pose interpolation system. The deformation system can be layered for coarse-to-fine control of the mesh to facilitate art-directable facial rigging of the corresponding character. The rig can be implemented as a node-based computational graph—e.g., a computational graph with 10,000 or more nodes—used to calculate the facial deformation. The nodes implement various functions, such as basic arithmetic operators and spline interpolation. The rig system also supports custom nodes that can execute any code.

[0106] We illustrate our results with reference to four example facial rigs. Three of these rigs are custom facial rigs used in the feature film "How to Train Your Dragon: The Hidden World" for Hiccup, Valka, and Toothless. The fourth example facial rig is from Ray, a publicly available open-source character published by the CGTarian Animation and VFX Online School.

[0107] The results of one or more methods of approximation are compared against the results of a linear blend skinning (LBS) approximation and a dense, feed-forward version of the approximation model disclosed herein. Based on these comparisons, it can be observed that one or more of the disclosed methods preserve high-frequency detail that is lost in the LBS approximation, and that one or more of the disclosed methods produce results that are more accurate than those produced by the dense version for three of the four example facial rigs. Furthermore, unlike the LBS approximation, one or more of the disclosed models preserve a mapping from rig parameters to a deforming mesh, which allows for approximating new animations without requiring access to the original rig functions.

[0108] Table 1 shows the statistics for each model trained on four example facial rigs. The models were not used to approximate the character's hair or eyes. However, the models were used to approximate the interior of the mouth as well as the teeth.

[0109] Table 1 TIFF2025183216000022.tif77153

[0110] FIG. 8 shows example mesh segments for the facial models of Hiccup, Valka, and Toothless (e.g., mesh segments used during coarse modeling), and FIG. 9 shows example mesh segments used during the fine stage of approximation (e.g., refined modeling).

[0111] More specifically, according to one set of examples, FIG. 8(a) shows coarse mesh segments for Hiccup's facial model, FIG. 8(b) shows coarse mesh segments for Valka's facial model, and FIG. 8(c) shows coarse mesh segments for Tooth's facial model. Each coarse mesh segment is represented as a contiguous region of the same shade of gray. For example, FIG. 8(a) shows regions 802, 804-1, 804-2, 806, and 808. Region 802 corresponds to the largest region of Hiccup's face. Region 806 corresponds to the inside of Hiccup's mouth, and region 808 corresponds to Hiccup's scalp. Regions 804-1 and 804-2 correspond to Hiccup's ears.

[0112] As another example, Figure 8(b) illustrates regions 812, 814-1, 814-2, 816, and 818. Region 812 corresponds to the largest area of ​​Valka's face. Region 816 corresponds to the inside of Valka's mouth, and region 818 corresponds to Valka's scalp. Regions 814-1 and 814-2 correspond to Valka's ears.

[0113] Similarly, FIG. 8(c) shows regions corresponding to segments for a coarse approximation of the Toothless face model.

[0114] As mentioned above, Figure 9 shows an example of mesh segments used in the refinement stage of the approximation. With respect to Figure 9, areas shown as shaded with a solid line pattern indicate segments that are not used in the refinement model.

[0115] More specifically, according to one set of examples, FIG. 9(a) shows refined mesh segments for Hiccup's facial model, FIG. 9(b) shows refined mesh segments for Valka's facial model, and FIG. 9(c) shows refined mesh segments for Tooth's facial model. Each refined mesh segment is represented as a contiguous region of the same shade of gray. For example, FIG. 9(a) shows regions 902, 904, 906, and 908, each of which is a subregion of region 802 in FIG. 8(a). Region 904 corresponds to Hiccup's forehead region. Region 902 corresponds to the first eye region, and region 906 corresponds to the other eye region. Region 908 corresponds to the area around Hiccup's nose and mouth. With respect to FIG. 8(a), regions illustrated using a solid line pattern correspond to unused (e.g., not further analyzed) segments in the refined model.

[0116] As another example, Figure 9(b) shows regions 912, 916, and 918, each of which is a subregion relative to region 812 in Figure 8(b). Region 912 corresponds to a first eye region, and region 916 corresponds to an adjacent eye region. Region 918 corresponds to the region around Valka's nose and mouth. With respect to Figure 8(b), regions illustrated with a solid line pattern correspond to unused (e.g., not further analyzed) segments in the refined model.

[0117] Similarly, Figure 9(c) shows the regions of Toothless's face model that correspond to the segments for the coarse approximation.

[0118] Referring back to the results listed in Table 1, the dense version runs faster than the disclosed approximation method and, in some cases, is more accurate than the disclosed method at approximating artist-created animations. However, the speedup of the dense approximation comes at the expense of more (e.g., a larger number) model parameters, which leads to higher memory storage costs, as seen in Table 1. When the dense model fails, undesirable visible artifacts appear in the deformed mesh, as seen in the facial meshes of Hiccup and Valka in Figure 11. These artifacts appear as high-frequency noise on the surface of the mesh and are caused by the dense approximation modeling each component of each vertex as an independent output. In contrast, the approximation method according to one or more embodiments models local neighborhoods within the mesh through the use of CNNs, and inaccuracies in the approximation are less likely to appear as high-frequency noise as in the dense approximation. Furthermore, the disclosed approximation method is more accurate than the dense approximation for poses generated through inverse kinematics for all characters.

[0119] The accuracy of the disclosed approximation method was compared with that of an LBS model and a dense feedforward network with fully connected layers instead of convolutional layers. The LBS weights and bone transformations were performed using the method of Le and Deng (Binh Huy Le and Zhigang Deng, Smooth Skinning Decomposition with Rigid Bones. ACM Trans. Graph. 31, 6, Article 199 (Nov. 2012)). For each mesh segment, we trained a dense model to approximate the vertex offsets, and a separate network. Each model contained two hidden layers with 256 nodes each, and the final output layer generated the offsets for each vertex of the mesh segment. Because this dense network was not constrained by the resolution of the deformation map, we did not train additional fine-grained models. However, rigid segments were deformed using the rigid body approximation method described previously. This dense model was developed by Stephen W. Bailey, Dave Otte, Paul Dilorenzo, and James FO'Brien. 2018. It is closest to the approach described by [ACM Trans. Graph. 37, 4, Article 119 (July 2018)]. With respect to this approach, the main difference with the dense model is that the facial mesh is not linearly deformed by a set of bones before applying the dense neural network.

[0120] For each character, we collected all available animations for the rig and randomly split the data into training and test data at 90% and 10%, respectively. The training data could be generated using only the poses in the training set according to Equation (5). For Ray's rig, no existing facial animations were available. Therefore, the rig parameters for each pose were independently sampled from a uniform distribution covering a user-specified range of values ​​to create the training and test sets. This random sampling method does not work for training approximate models of other character rigs due to their more complex mesh deformations. To train the approximate and detailed models disclosed herein, we generated 50,000 samples for each character. For the LBS models, 16, 24, and 32 bones were fitted to the mesh, with eight non-zero weights per vertex. We also generated 1,000 samples to estimate the vertex weights. Due to memory and computational constraints, we used fewer training examples.

[0121] Test sets for Hiccup, Valka, and Toothless are constructed by extracting all unique poses from the test data that were not used in training. In Table 2, we measure both the vertex position error (unit: mm) and the facial normal error (unit: degrees). The vertex position error is the average distance between the approximate vertex position and the target vertex position across the entire test set. The face normal error is the angle between the approximate face normal in the mesh and the target face normal. in particular

number

[0122] Table 2 TIFF2025183216000024.tif129153

[0123] According to the results shown in Table 2, most approximations achieve submillimeter accuracy on average. However, the average vertex position error is not a good indicator of the accuracy of the fine-scale details of the approximation. FIG. 2 shows the approximated deformation for Toothless's pose including wrinkles, and FIG. 10 shows the approximated deformation for Hiccup's pose including wrinkles. For comparison, FIG. 10(a) shows the ground truth mesh of Hiccup with forehead wrinkles. FIGS. 10(b) and 10(c) illustrate approximated deformations based on fine and coarse approximations of forehead wrinkles, respectively, according to one or more disclosed embodiments. FIG. 10(d) illustrates the approximated deformation based on LBS approximation.

[0124] As shown in Table 2, the fine approximation produces the smallest normal error for Hiccup, Valka, and Ray. However, for Toothless, the fine model produces the smallest error. This small error indicates that the fine and fine approximations can reproduce the fine details of the deformed mesh compared to the coarse and LBS approximations. Figure 11 shows a side-by-side comparison of the normal error. Figure 11 visually demonstrates the difference between the ground truth mesh and the rig approximation method evaluated via the original rig function. The heatmap on the right half of each approximation visualizes the angle between the normal vector on the approximation and the corresponding normal on the ground truth mesh. Smaller angle errors are preferable.

[0125] The approximation model disclosed herein was implemented in Python using TensorFlow. These execution times were evaluated on both a high-end machine using both a CPU and a GPU, as well as a consumer-quality laptop. The high-end machine used a 28-thread Intel Xeon E5-2697 v3 processor running at 2.60 GHz and an NVIDIA Quadro K5200 GPU. The laptop used an 8-thread Intel Core i7-7700HQ processor running at 2.80 GHz and an NVIDIA GeForce GTX 1060. The rotation of the rigid body segments was calculated by minimizing Equation (3) using SVD. When evaluating the full approximation on the GPU, this minimization problem was solved on the CPU due to the slow TensorFlow implementation of SVD on GPUs. Model training times consisted of 2–4 hours for generating training data using the original rig evaluation engine, followed by 2–3 hours for training the coarse approximation model and 2–3 hours for training the fine approximation model.

[0126] We compared the timing of the approximate models disclosed herein with the original rig evaluation software for Hiccup, Valka, and Toothless. The rigs for these three characters were designed for Libee, a multi-threaded rig evaluation engine. Character artists optimized these rigs to run as fast as possible on the engine. Unlike the methods disclosed herein with reference to various embodiments, Libee can evaluate character rigs only on the CPU. Table 3 shows the evaluation times using Libee and the methods disclosed herein running on both the CPU and GPU. The disclosed models were timed by taking the average execution time over 1,000 evaluations for a single pose.

[0127] Table 3 shows the average evaluation times in milliseconds on both high-end and consumer-quality machines. The coarse approximation is measured by evaluating the coarse model and the rigid transformation. The full approximation is measured by evaluating the coarse model, the fine model, and the rigid transformation. We also note that the neural network is evaluated on the GPU, while the rigid component is always evaluated on the CPU.

[0128] Table 3 TIFF2025183216000025.tif83153

[0129] The results shown in Table 3 indicate that the approximated models disclosed herein run 5 to 17 times faster than the original rig evaluation engine. On high-end machines, the approximation runs slower on the GPU because the model is evaluated at a single pose and the convolutions operate on low-resolution feature maps. This underutilization of the GPU results in poor performance. Furthermore, the GeForce GPU on the consumer-quality machine evaluates the approximated models faster than the Quadro GPU on the high-end desktop. This difference is likely due to the Quadro's slower clock speed compared to the GeForce GPU.

[0130] The approximation method disclosed herein, according to at least one embodiment, provides a differentiable model that maps rig parameters to a deforming mesh, which can be used in IK applications. A demonstration of the use of such an approximation method will now be described with reference to an interactive posing application and a facial landmark-based performance capture system.

[0131] First, the interactive posing application will be described in more detail.

[0132] A real-time posing application was developed in which a user manipulates a sparse set of control points, and an IK model, as disclosed above with reference to various embodiments, calculates rig parameters that deform a mesh to match the control points. The user drags points on the screen, and the mesh is updated interactively. The control points are provided to the system as 2D image coordinates. The IK model is trained to match the points by projecting the mesh onto the image plane, and the point loss term in Equation (9) is expressed in terms of distance in image coordinates. The mesh is projected onto the image plane by orthogonal projection with the camera oriented along the Z axis. Therefore, distance in image coordinates can be calculated using only the X and Y coordinates of the vertex positions.

[0133] The IK model is trained on meshes generated from the same extended dataset used to train the approximation model, and takes 1-2 hours to train, excluding the time to generate the mesh from the original rig function.

[0134] We compared the disclosed approximation method with a dense neural network approach. We trained IK models using both coarse and fine rig approximation methods (r-(p)). Experiments showed that IK models trained with gradients from the dense models of Hiccup, Valka, and Ray produced poses where the dense approximation produced highly inaccurate deformations with obvious visual artifacts. For the Tooth rig, we evaluated the dense model using poses generated from an IK model trained with gradients from the dense approximation method. To evaluate this model, we collected 25 user-generated control point configurations. These control point configurations are not guaranteed to match the original rig exactly. The IK model then calculates rig parameters for the control points. Finally, we generated a mesh using the approximation method and generated a ground truth mesh using the original rig function evaluated with the same rig parameters. We measured the vertex-to-vertex distance error and face-to-face normal error between the approximated mesh and the ground truth mesh. For Toothless, the approximate model is fed with poses generated from an IK model trained on the gradients. For Hiccup, Valka, and Ray, both the approximate and dense models are fed with poses generated from IK models trained on the gradients from the methods disclosed herein.

[0135] As can be seen in Table 4, the method disclosed herein more closely matches the ground truth mesh evaluated on the rig parameters output by the IK model. Table 4 shows the posing error measured in mm and degrees. For Toothless, the IK model is trained using gradients from the corresponding approximation. For Hiccup, Valka, and Ray, the IK model is trained using gradients from a method according to one or more embodiments to generate rig parameters for both the approach described herein and the dense method.

[0136] Table 4 TIFF2025183216000026.tif60153

[0137] Figure 12 shows a side-by-side comparison of the ground truth mesh and the approximate deformation for several example control point placements. A comparison of the mesh deformed by the rig parameters calculated through the IK model is shown. Dots 1202 represent the control points provided to the IK.

[0138] For Hiccup, Valka, and Toothless, the larger difference in accuracy between the disclosed and fine approximations can be explained by the type of poses output by the IK model. The IK model was trained in an unsupervised setting, and the distribution of poses output by the model does not exactly match the distribution of poses from the training data. As a result, some poses output by the IK model differ from the original training data. The higher accuracy achieved for these poses suggests that the disclosed approximation model generalizes to new poses better than the fine model. Ray's results further support this conclusion. Both Ray's CNN model and the fine model are trained on uniformly randomly sampled poses. Any pose output by the IK model will fall somewhere within this distribution. As can be seen from the results, the average approximation error of Ray's CNN and the fine model is comparable when evaluated on a uniformly random set of poses (see Table 2) and on the set of poses output by the IK model (see Table 4).

[0139] Next, we will explain in more detail the performance capture system using facial landmarks.

[0140] Real-time monocular facial performance capture systems rely on a differentiable rig for mapping video recordings to animation sequences. One paper provides a survey of current methods in monocular facial tracking. Because the appearance of an actor and an animated character do not match, the developed system animates the character by tracking a sparse set of facial landmark points. To track the actor's facial landmarks, we used an implementation of the method described in Zhen-Hua Feng, Josef Kittler, Muhammad Awais, Patrik Huber, and Xiaojun Wu, "Wing Loss for Robust Facial Landmark Localization with Convolutional Neural Networks," 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition (2017), pp. 2235-2245. This model was trained on the same dataset described in that method. The developed face tracking system used 54 of the 68 landmark points from the dataset. Corresponding points on the face model were manually identified.

[0141] To animate the mesh, we tracked the movement of landmarks detected in the recording and used an IK model to estimate the rig parameters needed to match the new landmark configuration. Because the actor's facial proportions may differ from those of the animated character, we tracked the difference between the actor's facial expression and the actor's neutral pose. This difference was then applied to the control points of the IK model. Specifically, let l0 be the landmark point detected on the image of the actor with a neutral expression, and l be the coordinate of the detected landmark point in the actor's current expression. Then, the control point c given to the IK model is calculated as c = c0 + l - l0, where c0 is the control point position of the mesh in a neutral expression. Figure 13 shows a frame from the recording and the deformed mesh resulting from the input. Figure 13 shows an example of the developed facial performance capture system. Facial landmarks are detected on the input image. This landmark information is passed to the IK model, which calculates the rig parameter values. The rig parameters are then passed to the disclosed approximation model to generate the deformed target mesh.

[0142] The method disclosed herein, according to at least one embodiment, provides fast (or faster) and accurate (or more accurate) approximations of film-quality facial rigs. Based on such approximations, fine-grained mesh deformation details can be preserved when bone-based approximations cannot. Furthermore, the disclosed method can provide differentiable rig approximations, which enables a wide range of potential new applications for character rigging. As examples, a real-time IK-based posing method and a performance facial capture system built on top of an IK solver have been described. Furthermore, once a model is trained, the original rig functions may no longer be required to evaluate mesh deformations. Because the approximation can be implemented using open-source machine learning libraries, the model can be more easily distributed and deployed on many different systems without requiring the complex or proprietary software originally used to build the facial rig. Thus, the approximated model disclosed herein provides a common format that allows facial rigs to be shared without relying on the original rigging software. Furthermore, the approximated model parameters can be considered a form of rig obfuscation, such that the underlying rigging techniques used to create the character are hidden when the model is shared.

[0143] Thus, the methods disclosed herein can be agnostic to the application and / or method by which the facial rig is built. For example, a particular rig can be a Maya rig or a rig built in an application that is proprietary to a particular studio or company. The approximation model disclosed herein provides a common format that allows facial rigs to be shared regardless of the original rigging software. Here, the common format can include model parameters obtained when the model is trained on any particular rig. In this way, a level of interoperability can be achieved such that the common format for approximating facial rigs can be more easily moved and / or shared between different digital content creation (DCC) applications and / or tools.

[0144] Because the method disclosed herein, according to at least one embodiment, is built on convolutional layers, the model is not necessarily limited to a single mesh topology. An approximated model trained on one mesh can deform new meshes not seen during training. The approximated rig can be transferred to a new facial mesh as long as the texture coordinates of the facial features in the new mesh match those of the original mesh. In this case, the approximated model outputs a deformation map using the same set of input rig parameters. Vertex offsets for the new mesh are calculated by sampling the deformation map with new texture coordinates corresponding to the new mesh. Figure 14 shows an example of transferring one mesh segment from a coarsely approximated model onto a new mesh with a different topology. In this example, the texture coordinates are manually aligned to those of the original mesh. For example, Figure 14 shows a rigid approximation of Hiccup transferred to a new mesh with a different topology. A single mesh segment from the coarse approximation is applied to the new mesh on the right. The facial mesh on the right is from the freely available Mathilda Rig.

[0145] The approximation method according to at least one embodiment outputs vertex offsets in the world coordinate system. As a result, if the facial proportions of the mesh differ significantly from the original model, the deformations applied to the new mesh may appear undesirable. A different parameterization of the offsets output by the approximation model can help alleviate this problem, allowing the methods disclosed herein to transfer the approximation from one rig to a facial mesh with significantly different proportions.

[0146] In the embodiments described herein, vertex normals are calculated separately and are not considered as part of the approximation model. However, in certain real-time applications, recalculating normals from a deformed mesh is avoided to save computation time. While the disclosed technique has not been experimented with approximating vertex normals, this technique can be easily extended to approximate normals. Instead of outputting three channels of deformation maps, the network can output an additional channel for normal direction, and an additional loss term can be included to train the model to output accurate normal vectors. Due to the low resolution of the intermediate feature maps, this technique may only be suitable for approximating vertex or face normals. Other high-resolution maps, such as normal maps or ambient occlusion maps, may need to be created using other means.

[0147] In at least one embodiment, texture coordinates provided with each character rig were used to interpolate vertex offsets from a deformation map. While these coordinates work well for mapping textures to mesh surfaces, they may not be well suited to the approximation methodology disclosed herein. For example, the texture coordinates for the upper and lower lips of a character's mouth may be close to each other. The vertex of the lower lip may move far away from the upper lip when the mouth opens. If the texture coordinates are close enough, the vertices of both lips may be located at the same pixel in the deformation map. In this case, the lips may appear to stick together when the mouth opens, resulting in inaccurate deformation. To avoid such problems, new deformation map coordinates can be generated specifically for this approximation task rather than relying on existing texture coordinates.

[0148] FIG. 15 is a flowchart of a method 1500 for generating a mesh deformation of a face model according to at least one embodiment.

[0149] At block 1502, a first plurality of deformation maps is generated by applying a first plurality of neural network training models.

[0150] In at least a further embodiment, each of the first plurality of neural network training models includes a convolutional neural network (CNN) model.

[0151] 3, the deformation map 306 is generated by applying coarse approximation models 304. Each of the coarse approximation models 304 may include a CNN model.

[0152] At block 1504, a first plurality of vertex offsets are extracted based on the first plurality of deformation maps.

[0153] In at least a further embodiment, extracting the first plurality of vertex offsets includes performing an interpolation of at least one deformation map of the first plurality of deformation maps at the plurality of vertex locations, where the interpolation may include bilinear or cubic interpolation.

[0154] 3, vertex offsets are extracted based on a deformation map 306. The extraction may include performing at least one interpolation of the deformation map 306 at one or more vertex positions (e.g., UV coordinates). The interpolation performed may be bilinear or cubic.

[0155] At block 1506, the first plurality of vertex offsets is applied to the neutral mesh of the facial model to generate a mesh deformation of the facial model.

[0156] In at least a further embodiment, applying the first plurality of vertex offsets to the neutral mesh of the face model includes adding the first plurality of vertex offsets to values ​​of the plurality of vertices of the neutral mesh.

[0157] 3, vertex offsets are applied to a neutral mesh 312 of a face model to generate an approximate mesh deformation 316. The vertex offsets may be applied by adding (e.g., with adder 310) the vertex offsets to the vertices of the neutral pose of mesh 312.

[0158] In at least a further embodiment, at block 1508, a second plurality of deformation maps is generated by applying a second plurality of neural network training models, the second plurality of deformation maps having a greater resolution than the first plurality of deformation maps.

[0159] 3, deformation map 326 is generated by applying refined approximation model 324. Deformation map 326 has a greater resolution than deformation map 306.

[0160] In at least a further embodiment, at block 1510, a second plurality of vertex offsets is extracted based on the second plurality of deformation maps.

[0161] 3, the vertex offsets are extracted based on the deformation map 326. The extraction may include performing at least one interpolation of the deformation map 326 at one or more vertex positions (e.g., UV coordinates). The interpolation performed may be bilinear or cubic.

[0162] In at least a further embodiment, at block 1512, the second plurality of vertex offsets are applied to the neutral mesh of the face model to generate a mesh deformation of the face model.

[0163] In yet a further embodiment, a first plurality of vertex offsets is applied to the neutral mesh by applying the first plurality of vertex offsets to the values ​​of all vertices of the neutral mesh. A second plurality of vertex offsets is applied to the neutral mesh by applying the second plurality of vertex offsets to the values ​​of at most a subset of the vertices of the neutral mesh to generate a mesh deformation. The subset of vertices of the neutral mesh can correspond to one or more regions of the neutral mesh that exhibit a level of approximation error that is above a certain threshold.

[0164] For example, referring to FIG. 3, the vertex offsets (extracted based on the deformation map 326 ) are applied to the neutral mesh 312 of the face model to generate the approximate mesh deformation 316 .

[0165] Vertex offsets (derived based on deformation map 306) can be applied to neutral mesh 312 by applying these vertex offsets to the values ​​of all vertices of neutral mesh 312. Vertex offsets (derived based on deformation map 326) can be applied to neutral mesh 312 by applying these vertex offsets to the values ​​of at most a subset of the vertices of neutral mesh 312 to generate approximate mesh deformation 316. The subset of vertices of neutral mesh 312 can correspond to one or more regions of the neutral mesh that exhibit a level of approximation error that is above a certain threshold.

[0166] In at least a further embodiment, the first plurality of vertex offsets are applied to the neutral mesh of the face model by adding the first plurality of vertex offsets to values ​​of the vertices of the neutral mesh to generate a plurality of intermediate vertex values, and the second plurality of vertex offsets are applied to the neutral mesh by adding the second plurality of vertex offsets to at most a subset of the plurality of intermediate vertex values ​​to generate the mesh deformation.

[0167] 3, the vertex offsets (extracted based on the deformation map 306) are applied to the neutral mesh 312 by adding (e.g., in adder 310) the vertex offsets to the values ​​of the vertices of the neutral mesh to generate a plurality of intermediate vertex values ​​(see, e.g., the output of adder 310). The vertex offsets (extracted based on the deformation map 326) are applied to the neutral mesh 312 by adding (e.g., in adder 314) the vertex offsets to at most a subset of the plurality of intermediate vertex values ​​to generate the approximate mesh deformation 316.

[0168] In at least a further embodiment, one or more rigid mesh segments of the face model that move rigidly during deformation are identified based on the first plurality of deformation maps and the second plurality of deformation maps at block 1514. A separate approximation of each of the one or more rigid mesh segments is performed.

[0169] Performing a separate approximation of each of the one or more rigid mesh segments may include performing rigid rotations and translations on the vertices of the rigid mesh segments.

[0170] For example, referring to Figure 5, a rigid mesh segment corresponding to triangle 502 is identified. A separate approximation of the mesh segment corresponding to triangle 502 is performed. Referring to Figure 5, a rigid rotation and translation can be performed on dot 506 of line 504.

[0171] FIG. 16 is a flowchart of a method 1600 for generating an approximation of a deformation of a face model according to at least one embodiment.

[0172] According to at least a further embodiment, at block 1602, a manual definition of a first plurality of vertices and a second plurality of vertices is received.

[0173] For example, referring to FIG. 7, a definition of control points 702 (including subsets 704-1 and 704-2) is received from a user.

[0174] The first plurality of vertices can include a first plurality of inverse kinematic (IK) control points, and the second plurality of vertices can include a second plurality of IK control points.

[0175] The first plurality of IK control points can correspond to a first region of the face model, and the second plurality of IK control points can correspond to a second region of the face model that is discontinuous with respect to the first region.

[0176] 7, subset 704-1 includes a first plurality of inverse IK control points corresponding to a first region corresponding to a first eye (e.g., a right eye) of the face model, and subset 704-2 includes a second plurality of inverse IK control points corresponding to a second region corresponding to a second eye (e.g., a left eye) of the face model.

[0177] At block 1604, the first plurality of vertices is provided to a first network to generate a first rig parameter pose. The first network may include a first dense neural network model. The first dense neural network model may include a first plurality of convolutional layers.

[0178] For example, referring to FIG. 7, the control points of subset 704-1 are input into dense neural network 706-1 to generate subset 708-1.

[0179] At block 1606, the second plurality of vertices is provided to a second network to generate a second rig parameter pose. The second network may include a second dense neural network model. The second dense neural network model may include a second plurality of convolutional layers.

[0180] For example, referring to FIG. 7, the control points of subset 704-2 are input into dense neural network 706-2 to generate subset 708-2.

[0181] At block 1608, the first rig parameter pose and the second rig parameter pose are processed to generate a composite rig parameter pose, which corresponds to an approximation of the deformation of the face model.

[0182] Processing the first rig parameter pose and the second rig parameter pose may include generating one or more average values ​​of the first rig parameter pose and the second rig parameter pose.

[0183] For example, referring to FIG. 7, the valid values ​​from the outputs (including output subsets 708-1, 708-2) are averaged together to produce a final averaged rig parameter pose 710.

[0184] By utilizing embodiments of the present disclosure, accurate approximations of deformations can be achieved in significantly less time and with significantly less computing resources than the original deformation functions, which require much more time and computing resources. Embodiments of the present disclosure achieve much more accurate results in less computing time, particularly when compared to ground truth deformation results, compared to other known deformation techniques.

[0185] 17 , there is shown a simplified block diagram of a system 600 for creating computer graphic images (CGI) and computer-aided animation, in which various embodiments of the present disclosure may be implemented or incorporated. The system 600 may include one or more terminals 601. The one or more terminals 601 may include hardware and software elements configured for designing CGI and assisting computer-aided animation. The terminals 601 may be implemented as a single computing device or a set of one or more computing devices corresponding to computer hardware and / or software.

[0186] Examples of terminals 601 may be desktop computers, laptop computers, workstation computers, mainframes, cluster computing systems, cloud computing terminals, embedded computing devices, computer graphics devices, gaming devices and consoles, video media playback devices, consumer electronic devices with programmable processors, or the like. One or more terminals 601 may be utilized in various stages of the production process, including pre-production, modeling, design, creation, editing, simulation, animation, rendering, post-production, finishing, publishing, etc., to generate recordings, computer files, tangible materials, etc., relating to or including images, image sequences, motion pictures, video, audio, or related effects associated with CGI and animation.

[0187] In one example of system 600, a user 602 can utilize one or more terminals 601 to design, create, or modify objects within a computer-generated interface that is displayed on a display associated with terminal 601.

[0188] The terminal may implement, include, or otherwise be in operative communication with systems and units including an object library 603, an object modeling unit 604, an object rigging unit 605, an object animation unit 606, an object rendering unit 607, and an object lighting unit 608. The object library 603 may include software and / or hardware elements configured to store and access information related to objects that may be accessed, requested, retrieved, and / or used by one or more terminals 601 to design, create, and modify 3D models of objects in CGI and animation processes.

[0189] The object modeling unit 604 may include software and / or hardware elements and information that are accessed, requested, obtained, and / or used by one or more terminals 601 to sculpt and design a 3D model to take on a desired appearance as dictated by a user 602, or other terminal operator, during the CGI and animation process.

[0190] The object rigging unit 605 may include software and / or hardware elements and information that can be accessed, requested, obtained, and / or used by one or more terminals 601 to design, create, or modify components of the 3D model, to define joint points and ranges of motion for various elements of the 3D model.

[0191] The object animation unit 606 may include software and / or hardware elements and information that are accessed, requested, obtained, and / or used by one or more terminals 601 to design, create, or modify animation aspects of the 3D model, including specifying the movement and position of various elements of the 3D model over time during animation, such as specifying animation paths, cues, etc., or generating key frames or intermediate frames for animated movement of the 3D model.

[0192] The object rendering unit 607 may include software and / or hardware elements and information that are accessed, requested, obtained, and / or used by one or more terminals 601 to design, create, or modify the final appearance of an animated 3D model, including generating one or more images of one or more computer-generated objects, including textures, colors, rigging, controls, etc.

[0193] The object lighting unit 608 may include software and / or hardware elements and information that are accessed, requested, obtained, and used by one or more terminals 601 to design, create, or modify lighting aspects of 3D models and animated scenes, including defining lighting sources, shading, reflections, refractions, textures, colors, etc.

[0194] One or more terminals 601 may be in communication with one or more server computers, which may be in operative communication with one or more data stores (e.g., databases, indexes, files, or other data structures). The one or more server computers may be connected to a data communications network, including a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a telephone network, a satellite or wireless communications network, or some combination of these or similar networks.

[0195] In selected embodiments, one or more server computers may implement one or more processes, systems, or units of the animation system, including object library 603, object modeling unit 604, object rigging unit 605, object animation unit 606, object rendering unit 607, object lighting unit 608, etc. One or more terminals 601, one or more server computers, or any other aspect of system 600 may be associated with or coupled to a display configured to display information, including a displayed interface for displaying, performing, executing, instructing, reviewing, and / or selecting operations of the present disclosure.

[0196] 18, an illustration of an exemplary computer 1700 is provided. One or more of the aspects of system 600 described above in FIG. 17, such as one or more terminals 600 or one or more server computers, can be configured as or include such a computer 1700. In selected embodiments, computer 1700 can include a bus 1703 (or multiple buses) or other communication mechanism, a processor 1701, a main memory 1704, a read-only memory (ROM) 1705, one or more additional storage devices 1706, and / or a communication interface 1702, or the like, or subcombinations thereof. Embodiments described herein can be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or other electronic units designed to perform the functions described herein, or selected combinations thereof. In all embodiments, the various components described herein may be implemented as a single component, or alternatively, may be implemented in various separate components.

[0197] A bus 1703 or other communication mechanism, including multiple such buses or mechanisms, can support communication of information within the computer 1700. A processor 1701 is coupled to the bus 1703 and can process information. In selected embodiments, the processor 1701 can be a specialized or dedicated microprocessor configured to perform particular tasks in accordance with the features and aspects disclosed herein by executing machine-readable software code that defines those tasks. A main memory 1704 (e.g., random access memory—RAM—or other dynamic storage device) is coupled to the bus 1703 and can store information and instructions executed by the processor 1701. The main memory 1704 can also store temporary variables or other intermediate information during the execution of those instructions.

[0198] A read only memory (ROM) 1705 or other static storage device may be coupled to the bus 1703 and may store static information and instructions for the processor 1701. Additional storage device 1706 (e.g., a magnetic disk, optical disk, memory card, etc.) may also be coupled to the bus 1703. The primary storage device 1704, the ROM 1705, and the additional storage device 1706 may comprise non-transitory computer-readable media that store information, instructions, or some combination thereof, such as instructions that, when executed by the processor 1701, cause the computer 1700 to perform one or more operations of the methods described herein. A communication interface 1702 may also be coupled to the bus 1703. The communication interface 1702 may provide or support two-way data communication between the computer 1700 and one or more external devices (e.g., other devices included within the computing environment).

[0199] In selected embodiments, computer 1700 may be connected to a display 1707 (e.g., via a bus). Display 1707 may use any suitable mechanism for communicating information to a user of computer 1700. For example, display 1707 may include or utilize a liquid crystal display (LCD), a light emitting diode (LED) display, a projector, or other display device to present information to a user of computer 1700 in a visual display. One or more input devices 1708 (e.g., an alphanumeric keyboard, a mouse, a microphone, a stylus pen) may be connected to bus 1703 for communicating information and commands to computer 1700. In selected embodiments, one input device 1708 may provide or support control over cursor positioning to enable selection and execution of various objects, files, programs, etc. provided by computer 1700 and displayed by display 1707.

[0200] Computer 1700 may be used to transmit, receive, decode, display, or the like, one or more video files. In selected embodiments, such transmitting, receiving, decoding, and displaying may be in response to processor 1701 executing one or more sequences of one or more instructions contained in main memory 1704. Such instructions may be read into main memory 1704 from another non-transitory computer-readable medium (e.g., a storage device).

[0201] Execution of the sequences of instructions contained in main memory 1704 may cause processor 1701 to perform one or more of the procedures or steps described herein. In selected embodiments, one or more processors in a multiprocessing arrangement may also be employed to execute the sequences of instructions contained in main memory 1704. Alternatively, or in addition, firmware may be used in place of or in combination with software instructions to implement procedures or steps in accordance with features and aspects disclosed herein. Thus, embodiments in accordance with features and aspects disclosed herein may not be limited to any specific combination of hardware circuitry and software.

[0202] Non-transitory computer-readable media may refer to any medium that participates in holding instructions for execution by processor 1701 or stores data for processing by a computer, and includes all computer-readable media, with the sole exception of transitory, propagating signals. Such non-transitory computer-readable media may include, but are not limited to, non-volatile media, volatile media, and temporary storage media (e.g., cache memory). Non-volatile media may include optical or magnetic disks, such as additional storage. Volatile media may include dynamic memory, such as main memory. Common forms of non-transitory computer-readable media include, for example, hard disks, floppy disks, magnetic tape, or any other magnetic media, CD-ROMs, DVDs, Blu-ray®, or other optical media, RAM, PROMs, EPROMs, FLASH®-EPROMs, any other memory card, chip, or cartridge, or any other computer-readable memory media.

[0203] In selected embodiments, communication interface 1702 may provide or support external, two-way data communication to or through a network link. For example, communication interface 1702 may be a wireless network interface controller or cellular radio that provides a data communication network connection. Alternatively, communication interface 1702 may comprise a local area network (LAN) card that provides a data communication connection to a compatible LAN. In any such embodiment, communication interface 1702 may send and receive electrical, electromagnetic, or optical signals that carry information.

[0204] The network link may provide data communication through one or more networks to other data devices (e.g., one or more terminals 601 as shown in system 600). For example, the network link may provide a connection to data devices through a local network of a host computer or to data devices operated by an Internet Service Provider (ISP). The ISP may in turn provide data communication services through the Internet. Thus, computer 1700 can send and receive commands, including program code, data, or a combination thereof, through one or more networks, network links, and communication interface 1702. Thus, computer 1700 can interface with or otherwise communicate with a remote server, or some combination thereof.

[0205] The various devices, modules, terminals, etc. discussed herein may be implemented on a computer by execution of software comprised of machine instructions read from a computer-readable medium, as described above. In certain embodiments, some hardware aspects may be implemented using a single computer, while in other embodiments, multiple computers, input / output systems, and hardware may be used to implement the system.

[0206] For a software implementation, certain embodiments described herein may be implemented with separate software modules, such as procedures and functions, that each perform one or more of the functions and operations described herein. The software code may be implemented in a software application written in any suitable programming language and may be stored in a memory and executed by a controller or processor.

[0207] The above-described embodiments and features are merely illustrative and should not be construed as limiting the present invention. The teachings of the present invention can be readily applied to other types of devices and processes. The description of such embodiments is intended to be illustrative, not limiting, of the claims. Many alternatives, modifications, and variations will be apparent to those skilled in the art. [Explanation of symbols]

[0208] 302 Rig Parameters 304 Coarse Approximation Model 306 Deformation Map 308 subset 310 Adder 312 mesh 314 Adder 316 Deformed Mesh 324 Microscopic Approximation Model 326 Deformation Map

Claims

1. 1. A method for generating a mesh deformation of a face model, comprising: generating a first plurality of deformation maps by applying a first plurality of neural network training models; extracting a first plurality of vertex offsets based on the first plurality of deformation maps; applying the first plurality of vertex offsets to a neutral mesh of the facial model to generate a mesh deformation of the facial model; A method comprising:

2. The method of claim 1 , wherein extracting the first plurality of vertex offsets comprises performing interpolation of at least one deformation map of the first plurality of deformation maps at a plurality of vertex locations.

3. The method of claim 2 , wherein the interpolation comprises bilinear or cubic interpolation.

4. The method of claim 1 , wherein applying the first plurality of vertex offsets to the face model neutral mesh comprises adding the first plurality of vertex offsets to values ​​of a plurality of vertices of the neutral mesh.

5. 10. The method of claim 1, wherein each of the first plurality of neural network training models comprises a convolutional neural network (CNN) model.

6. generating a second plurality of deformation maps by applying a second plurality of neural network training models, the second plurality of deformation maps having a greater resolution than the first plurality of deformation maps; extracting a second plurality of vertex offsets based on the second plurality of deformation maps; applying the second plurality of vertex offsets to a neutral mesh of the facial model to generate a mesh deformation of the facial model; The method of claim 1 further comprising:

7. applying the first plurality of vertex offsets to the neutral mesh includes applying the first plurality of vertex offsets to values ​​of all vertices of the neutral mesh; 7. The method of claim 6, wherein applying the second plurality of vertex offsets to the neutral mesh comprises applying the second plurality of vertex offsets to values ​​of at most a subset of vertices of the neutral mesh to generate the mesh deformation.

8. The method of claim 7 , wherein the subset of vertices of the neutral mesh corresponds to one or more regions of the neutral mesh that exhibit a level of approximation error that exceeds a certain threshold.

9. applying the first plurality of vertex offsets to the face model neutral mesh includes adding the first plurality of vertex offsets to values ​​of a plurality of vertices of the neutral mesh to generate a plurality of intermediate vertex values; 7. The method of claim 6, wherein applying the second plurality of vertex offsets to the neutral mesh comprises adding the second plurality of vertex offsets to at most a subset of the plurality of intermediate vertex values ​​to generate the mesh deformation.

10. identifying one or more rigid mesh segments of the face model that move rigidly during deformation based on the first plurality of deformation maps and the second plurality of deformation maps; performing a separate approximation of each of the one or more rigid mesh segments; The method of claim 6 further comprising:

11. The method of claim 10 , wherein performing a separate approximation of each of the one or more rigid mesh segments comprises performing rigid rotations and translations on vertices of the rigid mesh segments.

12. 1. A system for generating a mesh deformation of a face model, comprising: generating a first plurality of deformation maps by applying a first plurality of neural network training models; extracting a first plurality of vertex offsets based on the first plurality of deformation maps; applying the first plurality of vertex offsets to a neutral mesh of the facial model to generate a mesh deformation of the facial model; one or more controllers configured to system.

13. 13. The system of claim 12, wherein the one or more controllers are further configured to derive the first plurality of vertex offsets by performing interpolation of at least one deformation map of the first plurality of deformation maps at a plurality of vertex positions.

14. The system of claim 13 , wherein the interpolation comprises bilinear or cubic interpolation.

15. 13. The system of claim 12, wherein the one or more controllers are further configured to apply the first plurality of vertex offsets to the neutral mesh of the face model by adding the first plurality of vertex offsets to values ​​of vertices of the neutral mesh.

16. 13. The system of claim 12, wherein each of the first plurality of neural network training models comprises a convolutional neural network (CNN) model.

17. The one or more controllers: further configured to generate a second plurality of deformation maps by applying a second plurality of neural network training models, the second plurality of deformation maps having a greater resolution than the first plurality of deformation maps; The one or more controllers: extracting a second plurality of vertex offsets based on the second plurality of deformation maps; applying the second plurality of vertex offsets to a neutral mesh of the facial model to generate a mesh deformation of the facial model; The system of claim 12 further configured to:

18. the one or more controllers are further configured to apply the first plurality of vertex offsets to the neutral mesh by applying the first plurality of vertex offsets to values ​​of all vertices of the neutral mesh; the one or more controllers are further configured to apply the second plurality of vertex offsets to the neutral mesh by applying the second plurality of vertex offsets to values ​​of at most a subset of vertices of the neutral mesh to generate the mesh deformation.

20. The system of claim 17.

19. 20. The system of claim 18, wherein the subset of vertices of the neutral mesh corresponds to one or more regions of the neutral mesh that exhibit a level of approximation error above a particular threshold.

20. 1. A machine-readable non-transitory medium storing machine-executable instructions for generating a mesh deformation of a face model, the medium comprising: The instruction: generating a first plurality of deformation maps by applying a first plurality of neural network training models; extracting a first plurality of vertex offsets based on the first plurality of deformation maps; applying the first plurality of vertex offsets to a neutral mesh of the facial model to generate a mesh deformation of the facial model; a machine-readable non-transitory medium, including

Images