Generating nonlinear human shape models

Abstract

The present disclosure provides a statistical, articulated 3D human shape modeling pipeline within a fully trainable, modular deep learning framework. In particular, aspects of the present disclosure are directed to a machine-learned 3D human shape model having at least face and body shape components jointly trained end-to-end on a set of training data. Joint training of model components (e.g., including both face, hand, and the rest of the body components) enables improved synthetic consistency between generated face and body shapes.

Description

Technical Field

[0001] This disclosure generally relates to a three-dimensional (3D) human shape modeling pipeline. More specifically, this disclosure relates to systems and methods that include or use machine learning to create 3D human shape models, which have at least facial and body shape components that have been jointly trained end-to-end. Background Technology

[0002] Human movement, motion, and facial expressions hold significant practical importance and continue to receive attention across various technological fields. For example, creative capture in images and videos, immersive photography, and physical 3D spatial reasoning are all areas where improved human modeling has benefited. Therefore, models capable of accurately representing full-body details at the levels of pose, shape, facial expressions, and / or hand movements are essential for capturing and deeply analyzing subtle interactions that can only be fully understood in 3D.

[0003] While considerable progress has been made in locating human sketches in images and videos, and under certain conditions, in terms of elevating them to equivalent 3D skeletons and basic shapes, the overall exploration of reconstructing accurate models of the human body on a semantically meaningful surface level based on 3D physical space remains an area undergoing significant innovation.

[0004] The potential for advancements in model building, at least in the medium term, appears to lie somewhere between intuitive physical and semantic human modeling and the impact of capturing large-scale datasets. While many interesting and robust models for faces, hands, and bodies have been built over time, most (if not all) have been built in isolation rather than within the context of a complete human body. Consequently, they have inevitably failed to leverage the large-scale data analysis and model building processes that have recently emerged in the context of deep learning.

[0005] As examples, many recent full-body models, such as Adam, Frank, or SMPL-X, combine traditional components for the face, body, and hands, but typically focus on building consistent, joint parameterizations and appropriate scaling on already learned components. This makes it difficult to simultaneously fully utilize the structure in all the data, experiment with alternative representations of components or different losses, evaluate the final impact, and innovate. Summary of the Invention

[0006] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0007] One example aspect of this disclosure relates to a computing system characterized by having at least a machine learning-based 3D human shape model of facial and body shape components jointly trained end-to-end. The computing system includes one or more processors and one or more non-transitory computer-readable media for co-storing the machine learning 3D human shape model. The machine learning 3D human shape model includes a machine learning facial expression decoder model trained to process facial expression embeddings associated with the human body to generate facial expression data of the human body. The machine learning 3D human shape model includes a machine learning pose space deformation model trained to process a set of pose parameters to generate pose-related shape adjustments of the human body. The machine learning 3D human shape model includes a machine learning shape decoder model trained to process static shape embeddings associated with the human body to generate identity-based static shape data of the human body. The machine learning 3D human shape model is trained to generate a set pose mesh of the human body based at least in part on facial expression data, pose-related shape adjustments, and identity-based static shape data. All models in the machine learning facial expression decoder model, the machine learning pose space deformation model, and the machine learning shape decoder model have been jointly trained end-to-end, at least in part, based on a reconstruction loss function that compares a training set pose mesh generated by a machine learning 3D human shape model used to train the body with a shape scan registered with one or more ground truths of the training body.

[0008] Another example aspect of this disclosure relates to a computer-implemented method for end-to-end joint training of a 3D human shape model for machine learning. The method includes: obtaining one or more ground-value registered shape scans of a training body via a computing system including one or more computing devices, wherein the shape scans of the one or more ground-value registered shape scans of the training body include at least a full-body scan of the ground-value registered shape with an arbitrary pose and a ground-value registered facial detail scan. The method includes encoding the registered full-body scan with an estimated static pose using a shape encoder model via the computing system to obtain a static shape embedding associated with the training body. The method includes decoding the static shape embedding using a shape decoder model via the computing system to obtain identity-based static shape data of the training body. The method includes encoding data derived from the ground-value registered facial detail scan using a face encoder model via the computing system to obtain a facial expression embedding associated with the training body. The method includes decoding the facial expression embedding using a face decoder model via the computing system to obtain facial expression data of the training body. The method includes generating a training setting pose mesh for the training body via the computing system based at least in part on the identity-based static shape data, the facial expression data, and a set of pose parameters corresponding to the arbitrary pose. The method includes evaluating a reconstruction loss function that compares a training set pose mesh used to train the body with a ground truth registered full-body scan and a ground truth registered facial detail scan. The method also includes jointly training a shape encoder model, a shape decoder model, a face encoder model, and a face decoder model, at least in part based on the reconstruction loss.

[0009] Another example aspect of this disclosure relates to one or more non-transitory computer-readable media that jointly store a machine learning-based 3D human shape model, comprising: a machine learning facial expression decoder model trained to process facial expression embeddings associated with a human body to generate facial expression data of a human body; and a machine learning shape decoder model trained to process shape embeddings associated with a human body to generate identity-based shape data of a human body. The machine learning-based 3D human shape model is trained to generate a set pose mesh of a human body based at least in part on facial expression data, a set of pose parameters, and identity-based shape data. The machine learning facial expression decoder model and the machine learning shape decoder model have been jointly trained end-to-end, at least in part, based on a reconstruction loss function that compares the training set pose mesh generated by the machine learning-based 3D human shape model used to train the body with shape scans of one or more ground truth registrations of the training body.

[0010] Other aspects of this disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0012] A detailed discussion of embodiments for those skilled in the art is set forth in the description with reference to the accompanying drawings, wherein:

[0013] Figure 1 An example evaluation of an example human shape model according to an example embodiment of the present disclosure is depicted.

[0014] Figure 2 A block diagram depicts an example machine learning human shape model according to an example embodiment of the present disclosure.

[0015] Figure 3 Close-up scans of the head and hands are depicted according to an exemplary embodiment of this disclosure.

[0016] Figure 4 An example evaluation of an example human shape model according to an example embodiment of the present disclosure is depicted.

[0017] Figure 5 A block diagram depicts an example pose space deformation model according to an example embodiment of the present disclosure.

[0018] Figure 6 An example evaluation of an example human shape model according to an example embodiment of the present disclosure is depicted.

[0019] Figure 7 An example analysis of the coding techniques according to exemplary embodiments of the present disclosure is described.

[0020] Figure 8 An example evaluation of an example human shape model according to an example embodiment of the present disclosure is depicted.

[0021] Figure 9 An example evaluation of an example human shape model according to an example embodiment of the present disclosure is depicted.

[0022] Figure 10 Human pose and shape reconstruction from monocular images is depicted according to an example embodiment of the present disclosure.

[0023] Figure 11A A block diagram of an example computing system according to an example embodiment of the present disclosure is depicted.

[0024] Figure 11B A block diagram of an example computing device according to an example embodiment of the present disclosure is depicted.

[0025] Figure 11C A block diagram of an example computing device according to an example embodiment of the present disclosure is depicted.

[0026] The repeated reference numerals in multiple figures are intended to identify the same features in various implementations. Detailed Implementation

[0027] 1. Introduction

[0028] In general, this disclosure relates to a statistically articulated 3D human shape modeling pipeline within a fully trainable, modular deep learning framework. Specifically, aspects of this disclosure target machine learning-based 3D human shape models, which have at least facial and body shape components jointly trained end-to-end on a set of training data. The joint training of the model components (e.g., including facial and body components) enables improved synthetic consistency between the generated facial and body shapes.

[0029] More specifically, in some implementations, a set of training data may include high-resolution full 3D body scans of humans captured in various poses, optionally accompanied by additional close-ups of their head and facial expressions and / or hand joint movements. One example training dataset may include over 34,000 different human configurations. In some implementations, each of these scans may be registered to one or more initial, artist-designed, gender-neutral quadrilateral grids of assemblies to obtain a registered shape scan of the human body.

[0030] Some example machine learning 3D human shape models described herein may include multiple sub-models or other modular components. As examples, a human shape model may include: a machine learning facial expression decoder model trained to process facial expression embeddings associated with the human body to generate facial expression data of the human body; a machine learning pose space deformation model trained to process a set of pose parameters to generate pose-related shape adjustments of the human body; and / or a machine learning shape decoder model trained to process static shape embeddings associated with the human body to generate identity-based static shape data of the human body. For example, in some implementations, the decoder model may be trained as part of a corresponding autoencoder (e.g., a variational autoencoder) trained to receive an input shape mesh (e.g., a facial mesh or a body static shape mesh) and generate corresponding embeddings (e.g., facial expression embeddings or static shape embeddings).

[0031] Additional example models that may be included in a machine learning-based 3D human shape model may include: a machine learning joint center prediction model trained to process identity-based static shape data to generate multiple predicted joint centers for multiple joints of a human skeleton representation, and / or a machine learning hybrid skinning model trained to process facial expression data, pose-related shape adjustments, identity-based static shape data, and one or more predicted joint centers to generate a set pose mesh of the human body.

[0032] According to one aspect of this disclosure, some or all of the models included above or otherwise in a 3D human shape model in machine learning can be jointly trained end-to-end based on a shared loss function. Thus, in some implementations, all model parameters, including nonlinear shape space, pose space deformation correction, skeletal joint center prediction, and / or hybrid skinning functions based on variational autoencoders, can be trained in a single consistent learning loop.

[0033] Simultaneous training of all models on dynamic 3D scan data (e.g., over 34,000 different human configurations) improves the overall model's ability to capture relevance and ensures consistency across various components (e.g., modeled faces, bodies, and / or hands). In other words, by jointly training training data that may include facial, body, and / or hand scans using facial, body, and / or hand components, the resulting model can more naturally and consistently support facial expression analysis, as well as body shape and pose estimation (with detailed hand details).

[0034] This disclosure provides two exemplary, fully trainable, gender-neutral general human models, constructed and trained as described herein, with two different resolutions—a medium-resolution GHUM consisting of 10,168 vertices and a low-resolution GHUM (item) with 2,852 vertices. Example experimental data for these two example human models are also provided, demonstrating improved quality and consistency with reduced error. As an example, Figure 1 Example evaluations of GHUM and GHUML on GHS3D data are provided, with heatmaps of the two models on the left. Rendered plots show registration for different body poses of subjects (back row), and GHUML and GHUM fits for the middle and front rows, respectively. Both models demonstrate good quality estimation, with GHUM exhibiting lower error.

[0035] Therefore, aspects of this disclosure pertain to an end-to-end learning pipeline for constructing a full-body, statistical human shape and pose model capable of driving body shape, facial expressions, and / or hand movements. An end-to-end pipeline and a unified loss function are provided, enabling computational systems to perform deep learning within a minimal human skeletal parameterization with anatomical joint angle constraints. This allows for the simultaneous training of all model components, including nonlinear shape space, pose space deformation corrections, skeletal joint center estimators, and / or hybrid skinning functions. The model can be trained using high-resolution full-body scans as well as close-ups of moving faces and / or hands to ensure maximum detail and design consistency among body part components.

[0036] Furthermore, a newly collected 3D dataset of general human shapes, GHS3D, is described, consisting of over 30,000 realistic dynamic human scans. Example embodiments also utilize over 4,000 full-body scans from Caesar. The medium-resolution model GHUM and a specially designed (non-downsampled) low-resolution model GHUML are provided, and their relative performance for registration and constrained 3D surface fitting is evaluated under different linear and nonlinear models (PCA or variational autoencoders for body shape and facial expressions). Shape and pose recovery from images is also illustrated.

[0037] The systems and methods described herein offer numerous technical effects and benefits. As an example of a technical effect, the systems and methods of this disclosure can provide more realistic two-dimensional or three-dimensional renderings or models of human shapes, including human shapes with improved consistency among body, facial, and / or hand features. Specifically, model components can learn to be consistent with each other, or they can learn patterns or relationships across domains. Therefore, the systems and methods of this disclosure enable computational systems to perform improved human modeling functions.

[0038] As another example of technical effectiveness, by jointly training multiple model components, the total training time required to generate a human-shaped model can be reduced. More specifically, previous methods trained facial and body parts separately and then attempted to combine them later, resulting in two distinct training processes that required additional work to facilitate the combination, which could still be inconsistent. The proposed method jointly trains all model components in an end-to-end process, thereby making the trained and resulting models more consistent.

[0039] Exemplary embodiments of this disclosure will now be discussed in more detail with reference to the accompanying drawings.

[0040] 2. Example Implementation Overview

[0041] Given an unstructured point cloud The training set of human body scans, with variations in the number of points P, provides a statistical body model that enables learning representations of the variability of body shape and the natural deformations caused by joint movement. The technique involves a body model X that can have a consistent topology of V vertices, specified, for example, by a template mesh (assembled) provided by the artist, and α can be a set of variables controlling the body's deformation caused by shape and joint movement. For example... Figure 2 As illustrated, to learn a data-driven human model from scanned data Y, a body template can first be registered to a point cloud to obtain a new, registered ground truth mesh with the same topology, which is labeled as...

[0042] Then the registered truth grid X can be... * The data is fed into an end-to-end training network, where model parameters α are tuned to produce outputs that closely match the inputs, resulting from joint movement and shape adjustments. Various techniques can be used, including direct model parameter tuning of the point cloud via Iterative Closest Point (ICP) loss (the same as those used for registration), or with a proxy mesh X. * Alignment. Align input meshes with the same model topology X * As a target, this makes the process quite fast and shows better training loss.

[0043] therefore, Figure 2 The illustration provides an overview of the construction of an example end-to-end statistical 3D articulated human shape model. Training inputs can include a set of high-resolution 3D body scans, including static (or "A") poses and arbitrary poses demonstrating various joint movements and soft tissue deformations. Additionally, detailed close-up scans of the head with facial expressions and / or close-up scans of hands capturing different gestures and object grasping can be collected. These scans are typically shown in Y-shape.

[0044] Body landmarks can be automatically identified by rendering photorealistic 3D reconstructions (e.g., obtained using multi-view stereo triangulation) from multiple virtual viewpoints (e.g., which may differ from the original camera group used to collect data), detecting those landmarks in the generated images, and triangulating the landmark detection in the images to obtain their corresponding 3D reconstructions. Under the most conformal possible surface prior, a loss combining sparse landmark correspondences and dense iterative nearest neighbor (ICP) residuals (e.g., implemented as point scan to mesh face distance) can be used to progressively register the artist-designed full-body articulated mesh to the point cloud. The ground truth shape scan for registration is typically X... * As shown.

[0045] Figure 2 The illustrated example human shape model can have a body φb Deep variational autoencoder (VAE) and facial expression φ f The offset VAE is a nonlinear shape space. The example human shape model also includes a trainable pose space deformation function D modulated by a skeleton K with J joints, a center predictor C, and a hybrid skinning function M.

[0046] In some implementations, during training, all high-resolution scans of the same subject (full body as well as close-ups of the face and hands) can be used (see [link to implementation]). Figure 3 The residuals are appropriately masked using a filter F. For model building, N captured subjects can be used, with B full-body scans, F close-up hand scans, and H close-up head scans. In some implementations, during learning, the training algorithm can alternate between minimizing the loss function with respect to pose estimation in each scan θ and optimizing it relative to other model parameters (φ, γ, ψ, ω).

[0047] In operation, for pose and shape estimation, the model can be controlled by parameters α = (θ, β), including kinematic pose θ and body shape and facial expression β = (β... f ,β b The VAE latent space of ) where the encoder-decoder is composed of φ = (φ f ,φ b (This is given.)

[0048] 2.1 Example Human Model Representation

[0049] An example implementation of this disclosure represents a human model as a hinged mesh specified by a skeleton K with J joints and skin deformed using linear blended skin (LBS) to explicitly encode joint motion. In addition to skeletal joint motion, nonlinear models can be used to drive facial expressions. The model X with J joints can be formulated as M(α=(θ,β),φ,γ,ω), or more specifically as…

[0050]

[0051] in, It is a static shape based on identity in the "A" pose (see Figure 2 ), where β is a low-dimensional static shape embedding vector encoding the variability of body shape (which can use different low-dimensional representations including PCA or VAE); similarly, It utilizes low-dimensional facial expression embedding β f To control facial expressions with neutral head posture; It depends on the skeletal joint center of the body shape; It is a vector of skeleton pose parameters, which consists of the Euler angles of each joint (up to) 3 rotation DOFs and 3 translation variables of the root. This includes the skinning weights per vertex (e.g., in some implementations, these may be influenced by at most a few joints (e.g., I=4)); and finally, the pose-dependent correction blend shape. Add to static shapes to fix skinning artifacts.

[0052] Use the artist-defined assembly template grid (V ghum =10,168,V ghuml =2852, J=63) generate two example implementations of the proposed human model, referred to as GHUM and GHUML, respectively. For GHUM and GHUML, Figure 2 The illustrated pipeline estimates all parameters (θ, φ, γ, ψ, ω), although the mesh topology and joint level K are assumed to be fixed. This level is anatomically (and therefore minimally) parameterized so that biomechanical joint angle constraints can also be utilized during optimization. Vertex x i ∈X can be written as

[0053]

[0054]

[0055] Among them, T j (θ,c) is the world transformation matrix of joint j, integrated by traversing the kinematic chain from root to j. It can be multiplied by the static pose. The inverse of the world transformation matrix is ​​used to construct the transformation from a static to a set pose mesh.

[0056] 3. Example of end-to-end statistical model learning

[0057] This section provides a description of an example end-to-end neural network-based pipeline for optimizing skin weights ω and learning from surface meshes X registered from multiple subjects and multiple poses. * Embedded still shapes from full-body and close-up facial and hand scans. Facial expression embedding The joint center c(β) related to the same shape, and the mixed shape related to the posture. (see Figure 2 As a result of ICP registration, some example reconstruction losses can be formulated using the Euclidean distance error per vertex under one-to-one correspondence:

[0058]

[0059] Here, F is a filter that considers different types of data (e.g., full-body scans rather than close-ups). In some implementations, to construct X(α), the pose θ can be jointly estimated with statistical shape parameters. As an example, block coordinate descent can be performed, alternating between estimating the pose parameter θ based on the BFGS layer under the current shape parameters and updating other model parameters with a fixed θ. For example, the skin can be initialized from default values ​​provided by the artist, with all other parameters set to 0. Each submodule can update the parameter α based on the global loss (4).

[0060] 3.1 Example Variational Body Shape Autoencoder

[0061] In some example implementations, multi-subject shape scans can be obtained by registering the model to the Caesar dataset (4329 subjects) and captured scans in GHS3D, in a static or neutral "A" pose. As an example, Figure 3 The figure shows a close-up of a head and face scan. Some example implementations estimate the whole-body shape in a neutral A-pose by fusing a body scan with close-up hand and head scans. These example implementations can take advantage of the additional head and hand shape details compared to body shape estimation from a single body scan.

[0062] Given the estimated static shape for multiple subjects A compressed latent space can be established for body shape variations. As an example, instead of simply establishing a PCA subspace, a deep nonlinear variational autoencoder with a low-dimensional latent subspace can be used to represent body shape. Because in some implementations, mesh joint activity is estimated, the input scan to the autoencoder(s) is... They were all well aligned in the “A” pose without significant perturbations from rigidity changes and pose joint movements.

[0063] In some example implementations, the encoder and decoder can use parametric ReLU activation functions because they can model the identity transformation or standard ReLU with respect to certain parameters. As a standard practice, a variational encoder can output the mean and variance (μ, Σ), which can be transformed to the latent space using reparameterization tricks to obtain the sampled code β. b In some implementations, a simple distribution can be used. Furthermore, the Kullback-Leibler divergence can be integrated into the loss function to regularize the latent space. Therefore, an example formula is as follows:

[0064]

[0065]

[0066] Where encoder S E The variance of average body shape is captured in the latent vector β. b And decoder S D From β b Construct a static shape to match the target static shape of the input. Specifically, some example implementations initialize the first and last layers of the encoder and decoder to separate PCA subspaces. Where L is the dimension of the latent space. All other fully connected layers can be initialized to the same level, including PReLU units. In some example implementations, the submatrix of the log-variance entries can be initialized to 0, and the biases can be set to sufficiently large negative values. In this way, the network will be effectively initialized from a linear model while minimizing additional parameters, for example, compared to PCA.

[0067] 3.2 Example Variational Facial Expression Autoencoder

[0068] Variational body shape autoencoders can represent the variance of various body shapes, including facial shapes. To additionally support complex facial expressions (as opposed to merely anthropomorphic head and face static variations), additional facial modeling can be optionally introduced. For example, this model can be constructed from thousands of facial expression motion sequence scans in GHS3D. In addition to a 3-DOF articulated chin, two 2-DOF eyelids, and two 2-DOF eyeballs, parameters of the articulated joints on the head, including skin weights and pose space deformation, can be updated along with the rest of the pipeline.

[0069] For facial movements caused by facial expressions rather than joint movements, a nonlinear embedding β can be established in the same network structure as the variational body shape autoencoder. f By removing all articulated joint movements (including neck, head, eyes, and chin), the input to the VAE can be facial expressions in a neutral head posture. (for GHUM V) f =1,932 and 585 for GHUML). In some implementations, in order to unpose the registered head mesh to neutral, the articulated joint motion θ of the head shape (expressionless) matched to the registration can be fitted as closely as possible. The displacement field between the posed head and the registration is considered in the facial expression, and the effect of the articulated joint θ can be eliminated before comparison.

[0070] 3.3 Example Skinned Model

[0071] After applying a nonlinear shape and facial expression model, the optimal skinning function can be statistically estimated from multi-subject and multi-pose mesh data. Specifically, the same date term as in (4) can be used, but now the optimization variable can be the joint center estimator C(ψ). The parameters are the pose-related corrections for the body shape D(θ,γ) and / or the skin weights ω.

[0072] One possible choice for the skeletal joint centers is to place them at the average position on the ring connecting the boundary vertices of the two mesh components (segments) most affected by the joint. The average value of the boundary vertices... The skeleton is forced to lie within the convex hull of the mesh surface to allow the center to adapt to different body proportions. However, to achieve better skinning, the estimated... However, a linear regressor ΔC can be built on top of this: Learning joint center correction from body shape

[0073]

[0074] In some implementations, instead of learning joint centers globally by aggregating all mesh vertices, estimation can be performed locally only from those vertices skinned by the joint. This results in significantly fewer trainable parameters, decreasing from 3N×3J to 3N×3I, for example, I=4 in practice. Sparsity can also be encouraged through L1 regularization, as well as alignment of the bone orientation with the template. To avoid singularities and prevent joint centers from shifting off the surface, the magnitude of the center correction can be regularized.

[0075] In some implementations, a data-driven pose-dependent corrector (PSD) can be learned to repair skin artifacts that result from complex soft tissue deformation. This is then applied to a stationary shape. The nonlinear mapping D can be estimated: However, pose space correction at mesh vertices should intuitively derive from neighboring joints. Therefore, some example implementations use fully connected ReLU activation layers to extract more compressed feature vectors than the input (e.g., 32 units), from which pose space deformation can be linearly regressed.

[0076] As an example, Figure 5 A sample pose space deformation architecture sketch and illustration are shown, illustrating the benefits of PSD, here around non-passive joint activity points, such as the right hip and thigh, and the chest and armpit. For simplicity, θ is used as the input feature here, instead of...

[0077] Furthermore, in some instances, It is sparse, and joints can only generate local deformation corrections by patching their skinned meshes. Compared to SMPL's dense linear regressors, the proposed network produces deformations with significantly fewer trainable parameters of similar quality. The system can regularize the magnitude of pose space deformation to a smaller value, preventing overfitting to the target through PSD correction. This can be achieved as a simple L2 penalty, such as...

[0078]

[0079] High-frequency local PSD is generally undesirable and is likely due to overfitting. Therefore, smooth pose space deformation can be encouraged.

[0080]

[0081] Where N(i) are the neighboring vertices of vertex i, and l i,j It is based on the cotangent Laplace weight.

[0082] Even with reduced PSD regularization and a smaller number of trainable weights, overfitting can still occur. Unlike SMPL or MANO, which specifically build pose space deformations for certain regions (body or hand), in some implementations of this disclosure, the PSD model is built for the entire human model, jointly trained on high-resolution close-ups of body, hand, and head data. Therefore, the body data has limited variation in hand and head movements, while the head and hand data lack movement of the rest of the body. Consequently, there is a large joint motion space where all joints can move without affecting the loss. This is undesirable. To prevent overfitting, the input pose feature vector can be filtered or masked into four feature vectors, including head, body, left hand, and right hand joints. Each feature vector can be fed into the same ReLU layer, and the output can be summed before the next regressor. Thus, an example loss is as follows:

[0083]

[0084] This forces the PSD outside the masked area to be smaller, thus causing the network's corrections to be biased towards a limited global impact. However, deformations of shared surface regions corresponding to areas between the head, hands, and the rest of the body are learned from all relevant data.

[0085] In some implementations, to estimate skin weights, a linearly blended skin layer can be used at the end of the pipeline, given a pose θ and a static shape with pose correction for facial expressions. The output is a set pose mesh controlled by trainable skinning weight parameters ω (2). Each skinned vertex can optionally be maximized by a certain number (e.g., I = 4) of joints in the template. The system can then regularize ω to approximate the initial artist-drawn value to be smoothed in space. And / or non-negative and normalized skinning weights per vertex

[0086]

[0087]

[0088]

[0089] Alternatively, you can add the following to weakly regularize the final skinned mesh to make it smooth.

[0090]

[0091] Pose estimator. Given a body shape estimate and current skinning parameters, the pose θ can be reoptimized on the training set. To limit the search space, enhance consistency, and avoid unnatural local minima, anatomical joint angle constraints available in the anthropometric skeleton can be utilized. This problem can be efficiently solved using an L-BFGS solver with box constraints, and the gradients are evaluated by TensorFlow's automatic differentiation.

[0092] 4. Example Experiment

[0093] This section describes example experiments conducted on the example implementations of the systems and methods described in this paper.

[0094] Dataset. In addition to the Caesar dataset (4,329 subjects) containing diverse body and facial shapes, the example experiment described in this paper also used multiple 3DMD systems operating at 60Hz to capture 40 motion sequences with 55 body poses, 60 hand poses, and facial expressions from 32 subjects (16 women and 16 men). Subjects had BMIs ranging from 17.5 to 39, heights from 152 cm to 192 cm, and ages from 22 to 47. For all multi-pose data, we used 3 subjects for evaluation and 4 subjects for testing, based on freestyle motion sequences containing poses not typically found in the training set. Each facial capture sequence started with a neutral face and progressed to a specified facial expression, and each sequence lasted approximately 2 seconds.

[0095] Registration samples from the data, such as Figure 6 As shown. Specifically, Figure 6The image shows sample registration from Caesar (top left) and GHS3D data. Note the quality of the registration, which captures facial details and soft tissue deformations in other body parts as a result of joint movement.

[0096] Registration. Table 1 reports the registration of the point cloud using ICP and (extended) Chamfer distance. ICP error is measured as the distance between the point and the plane to the nearest registered grid surface, while Chamfer is estimated as a bidirectional point-to-point distance.

[0097] Table 1: Registration errors of GHUM and GHUML on Caesar and GHS3D (including details on the face, hands, and other body parts).

[0098]

[0099]

[0100] The proposed registration technique has low error and preserves local point cloud details. Figure 6 ).

[0101] Model Evaluation. Both full-resolution and low-resolution human models (GHUM and GHUML) were built using our end-to-end pipeline. The two models share the same set of skeletal joints but have 10,168 pairs of 2,852 mesh vertices (1,932 pairs of 585 vertices for facial expressions). Example experiments evaluated the mesh X to the registration X on the test data for both models. * The vertex-based average Euclidean distance. The figures are reported in Table 2 and visualized therein. Figure 4 , Figure 1 and Figure 9 (See Hand Evaluation in Sup.Mat). We compare the outputs of the two models with the registration mesh under their corresponding topologies. Both models can tightly represent various body shapes (e.g., modeled as VAEs), produce natural facial expressions (e.g., represented as facial VAEs), and are smooth and natural for various body shapes and poses without noticeable skinning artifacts (e.g., generated by optimized skinning parameters).

[0102] Table 2: Average Euclidean error (mm) based on vertex used for registration.

[0103]

[0104] GHUM vs. GHUML. The low-resolution model preserves global features of the body shape and correctly skins the body and face for motion. Compared to GHUM, GHUML loses some details such as lip deformation, muscle bulges in the arms and fingers, and wrinkles caused by adipose tissue. In terms of performance, GHUML is 2.5× faster than GHUM in feedforward evaluation mode.

[0105] Figure 4 An evaluation of Caesar is shown. Figure 4 The left side shows the Euclidean distance error per vertex for the registration of GHUM and GHUML. Figure 4 The right side, from top to bottom, shows the registration of GHUM and GHUML. The VAE-based model accurately represents body shape. Compared to GHUML, GHUM retains additional muscle or soft tissue details of the lower back.

[0106] Figure 9 It shows Figure 1 The evaluation and rendering in the dataset focus on the reconstruction of the hands using GHUM and GHUML. Similar conclusions are found in... Figure 1 As shown. Note that GHUM retains additional deformation details around the curved area of ​​the palm compared to GHUML.

[0107] VAE evaluation. For body shape, the proposed VAE supports both 16-dim and 64-dim latent representations, with the former having a higher reconstruction error of 1.72× (the reports in Table 2 and figures are based on the 64-dim representation). In some examples, 20-dim embeddings can be used for facial expression VAEs.

[0108] Figure 7 The reconstruction error of facial expressions as a function of latent dimension is shown for both VAE and PCA. The reconstruction error of the 20-dimensional VAE is similar to that using a 96-dimensional linear PCA basis, at the cost of a slower performance of 2.9 × 10⁻⁶. Specifically, Figure 7 An analysis of VAE and PCA models is presented, illustrating the advantages of nonlinearity in low-dimensional domains.

[0109] GHUM vs. SMPL. In Figure 8 The visual quality of GHUM and SMPL was compared. Specifically, Figure 8 The registration of GHUM and SMPL for each of the two poses is shown from left to right. GHUM produces poses with comparable visual quality, although fewer pelvic artifacts are noted in this motion sequence.

[0110] GHUM and SMPL have different mesh and skeleton topologies, and SMPL lacks hand and facial joints. For comparison, captured motion sequences (all poses, not in our training dataset) were obtained from GHS3D, and the captured sequences were registered with SMPL and GHUM meshes respectively. When evaluating errors, one-to-one point-to-plane Euclidean distances were used (e.g., to avoid sensitivity to surface slippage during registration), and errors were evaluated only on the body region to ensure a fair comparison with SMPL. The average reconstruction error from GHUM was 4.4 mm, while SMPL had an error of 5.37 mm, and the visual skinning quality of GHUM was observed to be comparable to that of SMPL.

[0111] 3D pose and shape reconstruction from monocular images. This section illustrates image inference for GHUM. In this case, the model's kinematic priors (for the hands and other body parts, excluding the face) were trained using data from Human3.6M, CMU, and GHS3D. No image predictor was used for pose and shape. Instead, initialization was performed under six different kinematic configurations, and parameter α was optimized under anatomical joint angle constraints. As losses, skeletal joint reprojection error and semantic body part alignment were used. Results are as follows: Figure 10 shown. Specifically, Figure 10 This paper demonstrates monocular 3D human pose and shape reconstruction using GHUM with a loss of semantic body part alignment, relying on nonlinear pose and shape optimization.

[0112] 5. Example devices and systems

[0113] Figure 11A A block diagram of an example computing system 100 according to an exemplary embodiment of the present disclosure is depicted. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0114] User computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0115] User computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 can store data 116 and instructions 118 executable by the processor 112 to cause the user computing device 102 to perform operations.

[0116] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be, or may otherwise include, various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Reference Figure 2 Discuss example machine learning model 120.

[0117] In some implementations, one or more machine learning models 120 may be received from server computing system 130 via network 180, stored in user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 may implement multiple parallel instances of a single machine learning model 120.

[0118] Additionally or optionally, one or more machine learning models 140 may be included in, or otherwise stored and implemented by, a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine learning model 140 may be implemented by the server computing system 140 as part of a web service (e.g., a body shape modeling and / or rendering service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0119] User computing device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). Touch-sensitive components can be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means that a user can use to provide user input.

[0120] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138 executed by processor 132 to cause server computing system 130 to perform operations.

[0121] In some implementations, the server computing system 130 includes one or more server computing devices or is implemented by one or more server computing devices. When the server computing system 130 includes multiple server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or combinations thereof.

[0122] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, model 140 may be, or may otherwise include, various machine learning models. Example machine learning models include neural networks or other multi-layered nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. (Reference) Figure 2 Discuss example model 140.

[0123] User computing device 102 and / or server computing system 130 can train models 120 and / or 140 via interaction with training computing system 150, which is communicatively coupled to network 180. Training computing system 150 may be separate from server computing system 130, or it may be part of server computing system 130.

[0124] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices or is implemented using one or more server computing devices.

[0125] The training computing system 150 may include a model trainer 160 that uses various training or learning techniques, such as, for example, error backpropagation, to train machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130. For example, a loss function may be backpropagated through the models(s)(s)(s)(s)(s)(s)(s)(s)) to update one or more parameters of those models(s)(s)(s)(s)(s)(s)(s)(s)(s)(s)(s)) based on the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over multiple training iterations.

[0126] In some implementations, backpropagation of the execution error may include truncated backpropagation over time. The model trainer 160 may perform various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0127] Specifically, model trainer 160 can train machine learning models 120 and / or 140 based on a set of training data 162. Training data 162 may include, for example, full-body, hand, and / or facial scans and / or ground truth registrations of these scans.

[0128] In some implementations, training examples can be provided by the user computing device 102 if the user has provided consent. Therefore, in such an implementation, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process can be referred to as model personalization.

[0129] Model trainer 160 includes computer logic for providing the required functionality. Model trainer 160 can be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0130] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or a combination thereof, and can include any number of wired or wireless links. Generally, communication on Network 180 can be carried over any type of wired and / or wireless connection using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0131] Figure 11A The figure illustrates an example computing system that can be used to implement this disclosure. Other computing systems may also be used. For example, in some implementations, user computing device 102 may include model trainer 160 and training dataset 162. In such implementations, model 120 can be trained and used locally at user computing device 102. In some such implementations, user computing device 102 may implement model trainer 160 to personalize model 120 based on user-specific data.

[0132] Figure 11B A block diagram depicts an example computing device 10 implemented according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0133] The computing device 10 includes multiple applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0134] like Figure 11B As illustrated, each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In other implementations, the API used by each application is application-specific.

[0135] Figure 11C A block diagram depicts an example computing device 50 implemented according to an example embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0136] Computing device 50 includes multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may communicate with the central intelligence layer (and the multiple models stored therein) using an API (e.g., a common API across all applications).

[0137] The central intelligence layer includes many machine learning models. For example, such as... Figure 11C As illustrated, a corresponding machine learning model (e.g., a model) can be provided for each application and managed by a central intelligent layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligent layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligent layer is included in, or implemented by, the operating system of computing device 50.

[0138] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data warehouse for computing device 50. For example... Figure 11C As illustrated, the central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0139] 6. Additional disclosures

[0140] The technologies discussed here involve servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and received from these systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and partitions of tasks and functions among components. For example, the processes discussed here can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0141] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of explanation and not as a limitation of this disclosure. Those skilled in the art will readily make changes, modifications, and equivalents to these embodiments upon understanding the foregoing. Accordingly, this disclosure does not exclude such modifications, variations, and / or additions to the subject matter that will be apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may be used with another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such changes, variations, and equivalents.

Claims

1. A computer-implemented method for jointly training a 3D human shape model using machine learning in an end-to-end pipeline, the method comprising, for one or more training iterations: Shape scans of one or more ground-value registered training bodies are obtained through a computing system including one or more computing devices, wherein the shape scans of the one or more ground-value registered training bodies include at least a full-body scan with ground-value registration in arbitrary pose and a facial detail scan with ground-value registration, and wherein, The truth-registered full-body scan and the truth-registered facial detail scan are separated from each other; The registered whole-body scan with an estimated static pose is encoded using a shape encoder model by a computational system to obtain a static shape embedding associated with the training body. The static shape embedding is decoded using a shape decoder model by a computing system to obtain identity-based static shape data of the training body; The facial encoder model is used by the computational system to encode the data derived from the ground truth registration facial detail scan to obtain a facial expression embedding associated with the training body. The facial expression embedding is decoded using a facial decoder model by a computing system to obtain the facial expression data of the training body; The training setting posture grid for the training body is generated by a computational system based at least in part on facial expression data, static shape data based on identity, and a set of posture parameters corresponding to any posture. The reconstruction loss function is evaluated by comparing the training pose mesh generated for the training body with a full-body scan with arbitrary poses and a facial detail scan with ground truth registration. The shape encoder model, shape decoder model, face encoder model, and face decoder model are jointly trained, at least in part, based on reconstruction loss; and Provides a 3D human shape model using machine learning, comprising at least a shape decoder model and a face decoder model, wherein the reconstruction loss function includes filters indicating which vertices of the training set pose mesh are compared with a ground truth registered full-body scan having the arbitrary pose, and which vertices of the training set pose mesh are compared with a ground truth registered facial detail scan, the ground truth registered facial detail scan being separate from the ground truth registered full-body scan having the arbitrary pose.

2. The computer-implemented method according to claim 1, wherein: Generating the training set pose mesh of the training body through a computational system includes: processing the set of pose parameters using a pose space deformation model to generate pose-related shape adjustments for the training body; and The pose space deformation model is trained at least in part based on the reconstruction loss, together with the shape encoder model, shape decoder model, face encoder model, and face decoder model.

3. The computer-implemented method according to claim 2, wherein: The training posture grid for the training body is generated by the calculation system, including: The computational system uses a joint center prediction model to process static shape data based on identity to generate multiple predicted joint centers for multiple joints of the skeletal representation of the trained body; and The computational system uses a hybrid skinning model to process facial expression data, pose-related shape adjustments, identity-based static shape data, and one or more predicted joint centers to generate the training setting pose mesh for the training body; and At least in part, the joint center prediction model and the hybrid skinning model are jointly trained based on the reconstruction loss, the shape encoder model, the shape decoder model, the face encoder model, the face decoder model, and the pose space deformation model.

4. The computer-implemented method according to claim 3, wherein, The hybrid skinning model includes a linear hybrid skinning model with multiple learned weights for the plurality of joints, respectively.

5. The computer-implemented method according to any one of the preceding claims, wherein, The shape scans of the one or more ground-value registered bodies for the training body also include ground-value registered hand detail scans, and wherein a reconstruction loss function evaluates the difference between the training set pose grid and the ground-value registered hand detail scans.

6. The computer-implemented method according to claim 1, wherein the joint training comprises alternating between: (1) estimating the set of pose parameters and (2) updating the parameters of the shape encoder model, shape decoder model, face encoder model and face decoder model while the set of pose parameters is fixed.

7. The computer-implemented method according to claim 1, wherein, The reconstruction loss function uses filtering and one-to-one correspondence between shape scans of the one or more ground truth registrations of the training body to evaluate the Euclidean distance error for each vertex.

8. A computational system characterized by a 3D human shape model with at least facial and body shape components jointly trained in an end-to-end pipeline using machine learning, the computational system comprising: One or more processors; as well as One or more non-transitory computer-readable media, collectively storing a machine learning-based 3D human shape model, the machine learning-based 3D human shape model comprising: A machine learning facial expression decoder model, which is trained to process facial expression embeddings associated with the human body to generate facial expression data of the human body; A machine learning pose space deformation model, trained to process a set of pose parameters to generate pose-related shape adjustments of the human body; and A machine learning shape decoder model, trained to process static shape embeddings associated with the human body, to generate identity-based static shape data of the human body; The machine learning-based 3D human shape model has been trained to generate a set pose mesh of the human body based at least in part on facial expression data, pose-related shape adjustments, and identity-based static shape data. The machine learning facial expression decoder model, the machine learning pose space deformation model, and the machine learning shape decoder model have all been jointly trained end-to-end based, at least in part, on the reconstruction loss function. The reconstruction loss function compares the training setting pose mesh generated by the machine learning 3D human shape model used to train the body with one or more ground-value registered shape scans of the trained body. Wherein, the shape scans of the training body for one or more ground-value registrations include ground-value registered full-body scans and ground-value registered facial detail scans, and wherein the ground-value registered full-body scans and the ground-value registered facial detail scans are separate from each other; and The reconstruction loss function includes a filter that indicates which vertices of the training set pose mesh are compared with the ground truth registered full-body scan having an arbitrary pose, and which vertices of the training set pose mesh are compared with the ground truth registered facial detail scan, which is separate from the ground truth registered full-body scan having the arbitrary pose.

9. The computing system according to claim 8, wherein, The machine learning-based 3D human shape model also includes: A machine learning-based joint center prediction model, trained to process isomorphic static shape data, to generate multiple predicted joint centers for multiple joints in a skeletal representation of the human body; and A machine learning hybrid skinning model is trained to process facial expression data, pose-related shape adjustments, identity-based static shape data, and one or more predicted joint centers to generate a set pose mesh of the human body. Among them, the machine learning joint center prediction model and the machine learning hybrid skinning model have been trained end-to-end, at least in part, based on the reconstruction loss and the machine learning facial expression decoder model, the machine learning pose space deformation model and the machine learning shape decoder model.

10. The computing system according to claim 9, wherein, The machine learning hybrid skinning model includes a linear hybrid skinning model with multiple learned weights for the multiple joints respectively.

11. The computing system according to claim 8, wherein, The shape scans of the training body that are one or more ground value registered also include ground value registered hand detail scans, which are separate from the ground value registered whole-body scans and the ground value registered facial detail scans.

12. The computing system according to any one of claims 8-11, wherein, The machine learning facial expression decoder model is trained as part of a facial expression variational autoencoder, which is trained to receive a training head grid, encode the training head grid to generate facial expression embeddings, and process the facial expression embeddings to generate facial expression data of the human body.

13. The computing system according to claim 8, wherein, The machine learning shape decoder model is trained as part of a shape variational autoencoder, which is trained to receive a training body static shape mesh, encode the training body static shape mesh to generate static shape embeddings, and process the static shape embeddings to generate identity-based static shape data of the human body.

14. The computing system according to claim 8, wherein, The set of pose parameters includes four feature vectors for the head, body, left hand, and right hand joints, respectively, and the machine learning pose space deformation model includes a rectified linear unit layer configured to jointly receive the four feature vectors and output a single set of outputs.

15. The computing system according to claim 8, wherein, For each of the multiple training examples, the machine learning 3D human shape model is trained by alternating between: (1) estimating the set of pose parameters and (2) updating other parameters of the machine learning 3D human shape model while keeping the set of pose parameters fixed.

16. The computing system according to claim 8, wherein, The reconstruction loss function uses a one-to-one correspondence to evaluate the Euclidean distance error of each vertex.

17. The computing system of claim 8, wherein the one or more non-transitory computer-readable media further stores instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations including: Obtain new facial expression embeddings, a new set of pose parameters, and new static shape embeddings associated with new human bodies; as well as A new set of pose meshes is generated for the new human body, based at least in part on the new facial expression embedding, the new set of pose parameters, and the new static shape embedding.

18. The computing system of claim 17, wherein obtaining the new facial expression embedding, the new set of pose parameters, and the new static shape embedding comprises: New facial expression embeddings, a new set of pose parameters, and new static shape embeddings are generated from monocular images depicting the new human body.

19. One or more non-transitory computer-readable media that collectively store a machine learning-based three-dimensional human shape model, said machine learning-based three-dimensional human shape model comprising: A machine learning facial expression decoder model, which is trained to process facial expression embeddings associated with the human body to generate facial expression data of the human body; as well as A machine learning shape decoder model, trained to process shape embeddings associated with the human body, to generate identity-based shape data of the human body; Specifically, a 3D human shape model trained using machine learning is used to generate a predefined pose mesh for the human body, based at least in part on facial expression data, a set of pose parameters, and shape data based on identity; and Specifically, the machine learning facial expression decoder model and the machine learning shape decoder model have been jointly trained end-to-end, at least in part, based on a reconstruction loss function. This reconstruction loss function compares a training-set pose mesh generated by a 3D human shape model used for training the body with shape scans of one or more ground-value registered models of the trained body. Wherein, the shape scans of the training body for one or more ground-value registrations include ground-value registered full-body scans and ground-value registered facial detail scans, and wherein the ground-value registered full-body scans and the ground-value registered facial detail scans are separate from each other; and The reconstruction loss function includes a filter that indicates which vertices of the training set pose mesh are compared with the ground truth registered full-body scan having an arbitrary pose, and which vertices of the training set pose mesh are compared with the ground truth registered facial detail scan, which is separate from the ground truth registered full-body scan having the arbitrary pose.

Images