Creating variants of animated avatar model using low resolution cage

Abstract

Some embodiments relate to methods, systems, and computer readable media for creating a template avatar. In some embodiments, the method includes obtaining a template avatar, the template avatar including template geometries obtained from a grid of the template avatar, generating a template cage associated with the template avatar, as a low resolution approximation wrapped around the template geometries, creating a target cage from the template cage by modifying the template cage based on a user's input, and generating a template pattern associated with the template avatar as a low resolution approximation wrapped around the template geometries. And deforming the template geometry with the target cage to generate a target avatar as a variant of the template avatar. The method further includes adjusting the binding and skin of the target avatar such that the target avatar can be animated. Variants of a template avatar can be created more efficiently and more effortlessly using these techniques.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 465,621, filed May 11, 2023, entitled “CREATION OF VARIANTS OF AN ANIMATED AVATAR MODEL USING LOW-RESOLUTION CAGES”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The implementation generally relates to computer graphics, and more specifically, but not exclusively, to methods, systems, and computer-readable media for creating and animatening variants of template avatars. Background Technology

[0003] Creating visually stunning animated avatars is a time-consuming and complex process involving advanced expertise in three-dimensional (3D) modeling, character rigging, and animation. It is challenging to leverage user-generated content (UGC) or support developers with limited experience in creating 3D characters when developing scenarios for building animated avatars.

[0004] In view of the above situation, some implementation methods have been proposed.

[0005] The background description provided herein is intended to present the context of this disclosure. The work done by the present inventor, with respect to what is described in this background section, and to any aspects of the specification that may not have constituted prior art at the time of filing, whether express or implied, should not be considered prior art to this disclosure. Summary of the Invention

[0006] Implementations of this application relate to creating high-quality avatar variants. For example, various techniques are used to create high-quality avatars while minimizing manual labor. These techniques include obtaining information related to a template avatar, including template geometry, generating a template cage approximating the template avatar, creating a target cage from the template cage based on user input, and deforming the template geometry with the target cage to generate the target avatar.

[0007] A system of one or more computers can be configured to perform specific operations or actions by means of software, firmware, hardware, or a combination thereof installed on the system, which, when operated, cause the system to perform the actions. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause that device to perform the actions.

[0008] According to one aspect, a computer-implemented method for creating variants of a template avatar is provided, comprising: obtaining a template avatar including template geometry obtained from a mesh of the template avatar; generating a template cage associated with the template avatar as a low-resolution approximation wrapped around the template geometry; creating a target cage from the template cage by modifying the template cage based on input from a user; and deforming the template geometry with the target cage to generate the target avatar as a variant of the template avatar.

[0009] This article describes various implementation methods of the computer implementation approach.

[0010] In some implementations, the computer-based method further includes adjusting the binding and skinning of the target avatar to enable the target avatar to be animated.

[0011] In some implementations, the template avatar also includes a template head of the template avatar, wherein the target avatar includes a target head of the target avatar, and adjusting the binding and skinning includes one or more of the following: determining the pose of the target head based on a specific pose of the template head; and determining the facial expression of the target head based on a specific facial expression of the template head.

[0012] In some implementations, adjusting the binding and skinning of the target avatar includes: converting the mesh of the template avatar into a flat panel mesh; deforming the flat panel mesh into a deformable neutral body based on the neutral pose of the target avatar; performing retargeting on the deformable neutral body to obtain a deformable skeleton of the template avatar; stitching the deformable skeleton to the shape of the target avatar to generate a stitched skeleton with binding and skinning; and stitching the deformable skeleton to the shape of the target avatar to generate a stitched skeleton with binding and skinning.

[0013] In some implementations, the computer-based method further includes: defining a spatial deformation function that maps points in the 3D world coordinates of the flat panel mesh to 3D points in the deformable neutral body; and using the spatial deformation function to deform the flat panel mesh into a deformable neutral body.

[0014] In some implementations, the template avatar is associated with multiple poses encoded by a Facial Action Coding System (FACS), and performing redirection includes: performing shape solving operations using a spatial deformation function to map the corresponding poses among the multiple poses, thereby generating a set of deformable pose shapes; and performing joint solving operations, which include using a deformable neutral body and the set of deformable pose shapes as real shapes to construct a linear hybrid skinned skeleton for the target avatar.

[0015] In some implementations, deforming the template geometry of the template avatar with a target cage to generate the target avatar includes using at least one surface-based deformation technique.

[0016] In some implementations, using at least one surface-based deformation technique includes: performing a wrap deformation to provide a wrap deformation version of the template avatar, and selecting a sparse subset of delta based on the wrap deformation version of the template avatar.

[0017] In some implementations, at least one surface-based deformation technique includes variational optimization.

[0018] In some implementations, variational optimization includes radial basis function optimization to find the displacement field, and applying the displacement field to the template avatar to generate the target avatar.

[0019] In some implementations, the computer-based method further includes performing at least one of implicit surface tracking or Laplacian fitting.

[0020] In some implementations, performing implicit surface tracing includes: generating a first implicit surface based on a template cage; adding the embedding equivalents of the vertices of the template avatar to the first implicit surface; generating a second implicit surface based on a target cage; and projecting the vertices of the target avatar toward the corresponding equivalents in the second implicit surface based on the first implicit surface and the embedding equivalents of the first implicit surface.

[0021] In some implementations, Laplace fitting includes solving a Poisson problem based on a modified Laplace operator to reconstruct the target avatar. The modified Laplace operator is designed to reproduce the regular geometry of the target cage by generating a surface of the target avatar that satisfies the delta fitting constraint.

[0022] According to another aspect, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium stores instructions that, in response to execution by a processing device, cause the processing device to perform operations including: obtaining a template avatar, the template avatar including template geometry obtained from a mesh of the template avatar; generating a template cage associated with the template avatar as a low-resolution approximation surrounding the template geometry; creating a target cage from the template cage by modifying the template cage based on user input; and deforming the template geometry with the target cage to generate a target avatar as a variant of the template avatar.

[0023] This document describes various implementations of non-transitory computer-readable media.

[0024] In some implementations, the operation also includes adjusting the binding and skinning of the target avatar to make the target avatar animateable.

[0025] In some implementations, adjusting the binding and skinning of the target avatar includes: converting the mesh of the template avatar into a flat panel mesh; deforming the flat panel mesh into a deformable neutral body based on the neutral pose of the target avatar; performing retargeting on the deformable neutral body to obtain a deformable skeleton of the template avatar; stitching the deformable skeleton to the shape of the target avatar to generate a stitched skeleton with binding and skinning; and performing skinning diffusion on the stitched skeleton after stitching to obtain the target avatar.

[0026] In some implementations, deforming the template geometry of the template avatar with a target cage to generate the target avatar includes using at least one surface-based deformation technique.

[0027] According to another aspect, a system is disclosed, comprising: a memory storing instructions thereon; and a processing device coupled to the memory, the processing device being configured to access the memory, wherein the instructions, when executed by the processing device, cause the processing device to perform operations, the operations including: obtaining a template avatar, the template avatar comprising template geometry obtained from a mesh of the template avatar; generating a template cage associated with the template avatar as a low-resolution approximation surrounding the template geometry; creating a target cage from the template cage by modifying the template cage based on user input; and deforming the template geometry with the target cage to generate a target avatar as a variant of the template avatar.

[0028] This article describes various implementations of the system.

[0029] In some implementations, the operation also includes adjusting the binding and skinning of the target avatar to make the target avatar animateable.

[0030] In some implementations, adjusting the binding and skinning of the target avatar includes: converting the mesh of the template avatar into a flat panel mesh; deforming the flat panel mesh into a deformable neutral body based on the neutral pose of the target avatar; performing retargeting on the deformable neutral body to obtain a deformable skeleton of the template avatar; stitching the deformable skeleton to the shape of the target avatar to generate a stitched skeleton with binding and skinning; and performing skinning diffusion on the stitched skeleton after stitching to obtain the target avatar.

[0031] According to another aspect, parts, features, and implementation details of a system, method, and non-transitory computer-readable medium may be combined to form additional aspects, including aspects that omit and / or modify some or part of a single component or feature, including additional components or features and / or other modifications, and all such modifications are within the scope of this disclosure. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of an example system architecture for creating variations of template avatars and animates, according to some implementation methods.

[0033] Figure 2 This is a flowchart of an example method for creating and animates variants of a template avatar, according to some implementations.

[0034] Figure 3 This is a flowchart of an example method for adjusting the binding and skinning of a target avatar according to some implementations.

[0035] Figure 4 An example of a workflow for creating a head cage and the resulting animable head variants, according to some implementations, is shown.

[0036] Figure 5 Examples of flat panel grids and skeletons according to some implementations are shown.

[0037] Figure 6 An example of an automated processing pipeline for deforming the geometry of a flat facial panel, according to some embodiments, is shown.

[0038] Figure 7 More details of an automated processing pipeline according to some implementations are shown.

[0039] Figure 8 More details of an automated processing pipeline according to some implementations are shown.

[0040] Figure 9 An example of shape migration by utilizing an existing dynamic head is shown according to some implementations.

[0041] Figure 10 An example of a two-step method, including shape transfer and linear blended skin (LBS) skeleton solving, is shown according to some implementations.

[0042] Figure 11 Example functions for performing shape migration via spatial deformation according to some implementations are shown.

[0043] Figure 12 This illustrates a skeleton transfer technique that automatically transfers skeletons, skins, and poses to a target given a correspondence between neutral facial expressions, according to some implementations.

[0044] Figure 13 This is a block diagram of an example computing device according to some implementation methods. Detailed Implementation

[0045] In the following detailed description, reference is made to the accompanying drawings, which form an integral part of this document. In the drawings, like reference numerals generally identify like parts unless the context otherwise requires. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein. As generally described herein and shown in the drawings, aspects of this disclosure can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are considered herein.

[0046] The embodiments described in this specification, such as "some embodiments," "implementations," and "example embodiments," may include specific features, structures, or characteristics; however, each embodiment does not necessarily include specific features, structures, or characteristics. Furthermore, these phrases do not necessarily refer to the same embodiment. Additionally, when a specific feature, structure, or characteristic is described in connection with an embodiment, whether explicitly stated or not, that feature, structure, or characteristic may be implemented in combination with other embodiments.

[0047] This disclosure specifically relates to techniques for deforming a "template" avatar (or "avatar skeleton") to a target shape, and for automatically migrating complex binding elements of the template avatar to the target shape. This disclosure provides a way to adjust an existing avatar skeleton to generate new variations. This adjustment can be achieved through an intuitive workflow that deforms a pre-existing "template" avatar skeleton to the target shape and automatically migrates complex binding elements to the target avatar.

[0048] The implementation uses a low-resolution cage encased around the stencil avatar geometry to infer surface correspondences and establish a deformation field between the existing stencil avatar skeleton and the new target variant of the stencil avatar skeleton. The proposed workflow for creating new avatar variants comprises two stages. The first stage can be a manual stage. For example, in the manual stage, the creator can (e.g., via some digital content creation (DCC) tool or procedurally via scripts) create and / or sculpt a cage to define a new rough shape for the part. This is the only manual step that needs to be performed by the creator, requiring a significantly lower level of expertise than other methods.

[0049] The second stage can be an automation stage. For example, in the automation stage, automation techniques can deform the underlying geometry and transfer facial expressions. Implementation methods can also adjust joints and skinning to create a target facial skeleton optimized for mobile device runtime performance.

[0050] Automation drastically reduces the time required to create character skeletons and poses from long cycles (e.g., about a month) to extremely short cycles (e.g., seconds). Workflows exist that deform the head by sculpting a cage, generating a head icon that automatically renders facial expressions. In such workflows, the user sculpts an existing cage into the desired shape. Automation then deforms the original geometry to the deformation target, adjusting joints and skinning simultaneously.

[0051] Once the user has sculpted the cage into the desired shape, the following main steps of an automated technique are executed to create animated avatar variants. During cage deformation, this technique deforms the underlying avatar components, causing their rough shape to match the shape changes of the cage. Various methods have been considered. These methods include, for example, spatial deformation via radial basis functions (RBF), wrapping deformation, and surface deformation. In some implementations, surface deformation can provide an efficient way to create animated avatar variants.

[0052] This surface deformation scheme incorporates three terms within an optimization framework. These terms include variational optimization, implicit surface tracing, and Laplacian fitting. Variational optimization imposes physical constraints to ensure reasonable results. Implicit surface tracing extends implicit skinning techniques to surface-based deformation. Laplacian fitting can be a variant of the Laplacian surface editing technique used to reproduce the properties of regular geometric primitives.

[0053] Cage deformation can be followed by pose / expression transfer. This technique adjusts joints and skinning to adapt the facial expressions and poses of the initial geometry to the plausible expressions / poses of the cage-deformed geometry. A key feature is the use of spatial deformers (e.g., radial basis functions (RBFs)) calculated from the cage deformation step to compute the target geometry vertices for each facial expression and body pose.

[0054] Pose / expression transfer can be followed by skeleton creation. For linear blendskinning (LBS) skeletons, target vertices computed from pose / expression transfer can be used as constraints for a known solver to adjust joint transformations and skinning. For blend shape skeletons, the implementation can compute the vertex delta for each pose / expression.

[0055] Examples of animated avatar skeletons and their corresponding cages may exist according to some implementation methods. Specifically, as shown in the examples, the avatar skeleton can be an animable / animated stencil avatar represented by a mesh. Multiple low-resolution cages are created and wrapped around the components of the avatar skeleton (e.g., head, torso, hands, feet, etc.). In other applications, cages can be used to wrap clothing and facial accessories around the avatar, while the implementation methods disclosed herein use cages to generate variations of the stencil avatar.

[0056] One aspect of the example may include pose / expression transfer techniques. These techniques adjust joints and skinning to adapt the facial expressions and poses of an initial geometry (e.g., head geometry) to a plausible expression / pose of a cage-deformed geometry (e.g., head variant geometry). One aspect of this technique is to compute the target geometry vertices for each facial expression and body pose using a spatial deformer (e.g., RBF) calculated from the cage deformation step.

[0057] On the other hand, it can be skeleton creation. For linear hybrid skinning (LBS) skeletons, automation techniques can use target vertices computed from pose / expression transfer as constraints for a known solver to adjust joint transformations and skinning. For hybrid shape skeletons, automation techniques can compute the vertex delta for each pose / expression. This paper provides further details on pose / expression transfer (e.g., shape solving) and skeleton creation (e.g., LBS skeleton solving) according to various implementations.

[0058] According to various implementations, a platform is provided in which each user can have an avatar with expressive and communicative capabilities. The platform can have a large number of avatars with static (e.g., non-animable) heads, which user groups use to represent themselves in virtual experiences.

[0059] The challenge in enabling one or more avatars to display animated facial expressions on a media platform lies in converting the static heads of these avatars into versions that support high-quality facial expressions. This introduces scalability challenges, as one or more static heads are typically constructed as a combination of static decals and simple head shape meshes. The avatar's face, mouth, eyes, and other facial features are often not represented using geometry, making them difficult to animate. An example media platform might have approximately 600 static decals and approximately 50 head shapes. This example implies that there could potentially be 30,000 different heads that can be animated.

[0060] Decoupling the static head from the decal and head shape allows for the decoupling of rigged, skinned, and animated faces from the head shape. Artists (and other creators) can generate rigged and animated faces on flat panel meshes, which effectively simplifies the creation of geometry, skeletons, and animations, bringing it closer to a two-dimensional (2D) problem. Through the implementation of the automated face stitching technique described in this paper, a pipeline can then deform and stitch these artist-generated flat face panels onto any head shape. This deformation and stitching supports the large-scale generation of combinations of animated faces and head shapes.

[0061] Implementations may provide a workflow for converting one or more avatars with static heads into avatars with animable facial expressions. According to some implementations, example static heads may exist on media platforms. Various head shapes and static decals may be provided.

[0062] Each specific static head can be the result of texturizing a selected head shape (e.g., one of about fifty available shapes) at runtime, with a defined UV mapping and static facial decals (e.g., about six hundred decals available). By decoupling the head shape from the static decals, the number of unique heads grows exponentially, thus providing users with a wider variety of static head avatars to choose from.

[0063] Manually converting a static head into a head with an animated facial expression can be a time-consuming process. This manual conversion might involve: modeling geometry with animable parts to make it resemble a static face decal, creating a joint skeleton, skinning the vertices, and finally posing the geometry with a facial expression. Manually converting each combination of head shape and face decal in the avatar / head catalog individually is not scalable. Therefore, the implementation disclosed herein employs a composite approach, decoupling the creation of the animable facial skeleton from the final head shape.

[0064] An automated processing pipeline stitches an animable face and its skeleton onto a head-shaped mesh. The creation of the animable face mesh and skeleton can be simplified to primarily flat, rectangular face surfaces. For example, some implementations may use flat panel face meshes and skeletons. A front view, side view, skeleton joints, and Facial Action Coding System (FACS) poses can co-create the facial expression when blended. While a few FACS poses (e.g., six poses) might be used in a single example, a much larger number (e.g., any number between eighty-five and one hundred and twenty poses) can be defined (again, as an example) for the skeleton retargeting solver.

[0065] Using the techniques discussed in this paper, no curvature specific to any particular head shape needs to be addressed during rigging and animation. These flat panel facial skeletons can be pseudo-3D (two-and-a-half-dimensional, 2.5D) because the mouth bag shape and internal teeth / tongue components have some depth, and the eye components and other facial features have detached surface parts. Animation for the eyes, eyebrows, and nose can be primarily 2D. This simpler approach greatly simplifies the rigging, skinning, and animation processes. By design, the main surfaces of the flat panel facial mesh and skeleton can be square, so the xy coordinates of the vertices are exactly the uv coordinates in UV space.

[0066] After creating the flat panel face mesh and skeleton, the next step is to automate the stitching of these flat panel face skeletons onto the existing head shape. The automated pipeline stitches the flat panel face mesh and skeleton onto the head shape mesh defined by UV mapping, similar to how actual textures are applied to the mesh. Given a flat face panel mesh and a target head shape, this automated pipeline generates a fully bound and animable stitched head.

[0067] In some implementations, an automated processing pipeline deforms the geometry of a flat face panel. This flat face panel is then redirected onto a head shape with the obtained UV mapping, resulting in a fully bound and animated stitched head with facial expressions.

[0068] Several technical challenges of this processing pipeline require solutions, including the following: Deforming flat panels onto an arbitrary shaped surface of a head-shaped mesh. If the flat panels are very thin, they can be directly mapped using UV coordinates and a signed normal distance from the facial surface. However, these flat panels are 2.5D flat panels. Furthermore, the flat panels may have internal mouth pockets with teeth and tongue components. Determining how these components deform when the facial surface is stitched onto a curved surface is a challenge.

[0069] The skeleton of the flat facial panel needs to be repositioned so that the final facial expression on the sutured head looks good. Because it is helpful to be able to suture the flat facial panel to any arbitrary head shape, the implementation can have a universal method capable of accommodating wide curvature differences and anisotropic scaling. This universal method allows for the use of a wide variety of head shapes.

[0070] In some implementations, the first part of the automated processing pipeline deforms a flat facial panel in a neutral pose into a head shape, and then redirects the skeleton. Redirecting the flat facial panel skeleton to the head shape can be done in two steps: a shape-solving step and a joint-solving step. The second part of the automated processing pipeline stitches the deformed facial skeleton onto the head shape to generate a stitched head skeleton. This automated processing pipeline can also apply some skin diffusion around the edges of the stitched areas of the deformed facial skeleton to achieve smooth deformation attenuation in the final result of the pipeline.

[0071] Examples of key weighting issues might include the following. One example is weight gaps. When weights don't extend evenly to the boundaries, they can "bleed out" around the internally fixed weights, causing unwanted distortion. The bottom corners of a panel can be fully weighted to the head to "fix" those corners.

[0072] Retaining this results in a “fixed pool” of weights. Weight gaps can be corrected by applying a narrow band diffusion centered on the stitch boundary and blending at a specified geodesic distance from the boundary, which may involve a specified “blending mask.” There may also be issues with excessively high internal weights. In some implementations, for head shapes where there isn’t much space between the skin surface and the inside of the mouth, the mouth pouch weights are often too high, causing collisions if the internal weights are not adjusted as part of the diffusion process.

[0073] Excessive weighting of the mouthpiece can be corrected by performing a scalar expansion on the point cloud of the mesh using a vector heatspan method, thus limiting the diffusion of weights from the surface to the interior. This avoids excessive weighting in R... 3 The "bleeding" problem caused by diffusion weights. Similarly, these new weights are then extended to the lower teeth and tongue, making their weights proportional to the weights of the surrounding mesh. For example, there might be a scalar expansion of the mouth pouch relative to the tongue and lower teeth. There might also be other cases where weights are overcorrected.

[0074] According to some implementation methods, various examples of stitching results may exist. For example, various flat facial panels can be stitched to one of multiple head shapes, and animation can be performed accordingly.

[0075] Therefore, the embodiments disclosed herein provide a method for stitching rigged, skinned, and posed 2.5D flat panels to a mesh surface. This method is similar to rendering textures on a mesh via UV mapping. High-quality results can be achieved by fine-tuning and considering many details within a specific set of flat face panels and head shapes.

[0076] It can maintain geometric consistency across all or almost all flat facial panels. Specifically, enforcing similarity symbolic distances for 2.5D components (e.g., eyes, eyebrows, mouth, teeth, and tongue) improves result quality. By enforcing these distances, the results become more predictable, thereby reducing collisions in the final stitching resources.

[0077] In the deformation neutral step, adding subdivision surface smoothing to the head shape while deforming the flat face panel helps create a higher quality stitching result. This method eliminates faceting present in the original head shape and reduces associated artifacts in the final stitched resource. Defining vertices that do not move between the neutral pose and any particular FACS pose is crucial for creating aesthetically pleasing and intuitive repositioning.

[0078] Regarding symmetry, aside from common issues like slight numerical asymmetry and intentional asymmetric design, unintended asymmetries introduced during shape and joint solving are managed through implementation methods. Asymmetries introduced during joint solving (e.g., joint static pose / vertex weights) can often be attributed to asymmetries in the shape solving input. However, since these asymmetries may include intended design asymmetries, simply correcting the shape solving output may not be sufficient to guarantee symmetrical joint solving results. Therefore, a reconstruction of similar intended symmetry is applied after solving.

[0079] Skinning diffusion, especially the subsequent normalization (including pruning small weights and limiting the number of joints / vertices), can amplify small asymmetries during the process. Ensuring that the results of the two solution operations (i.e., shape solving and joint solving) are symmetrical where symmetry is expected can reduce the problem of asymmetry amplification. However, weights still need to be symmetrical after diffusion to ensure correct results.

[0080] Regarding smoothing, the diffusion weight method can use the graph Laplacian operator. While this method can be fast, the results depend solely on the mesh topology. Using the cochet Laplacian operator addresses the topology problem but leaves continuity issues. Solving the biharmonic equations with the cochet Laplacian operator corrects the previous problem. This method can be achieved by minimizing the Hessian energy (E). H 2 To achieve better "shaped" results, rather than minimizing the Laplace energy (E) Δ 2 ).

[0081] Debugging can help decouple the steps of finding the shape through shape solving from the steps of obtaining the final linear hybrid skin skeleton through joint solving. The ability to visualize each individual step in the process of generating the final suture result allows the implementation to narrow down the scope to identify which step in the process failed or provided unacceptable results. Figure 1 -System Architecture

[0082] Figure 1 This is a diagram of an example system architecture for creating variations of template avatars and animates, based on some implementation methods. Figure 1 The same reference numerals are used to identify the same elements as in other figures. Characters following the reference numerals, such as "110," indicate that the text specifically refers to the element with that particular reference numeral. Reference numerals without characters such as "110" in the text refer to any or all elements in the figure that bear that reference numeral (e.g., "110" in the text refers to reference numerals "110a," "110b," and / or "110n" in the figure).

[0083] System architecture 100 (also referred to herein as the "system") includes an online virtual experience server 102, a data storage device 120, client devices 110a, 110b, and 110n (generally referred to herein as "client device 110"), and developer devices 130a and 130n (generally referred to herein as "developer device 130"). The virtual experience server 102, data storage device 120, client device 110, and developer device 130 are coupled via network 122. In some implementations, client device 110 and developer device 130 may refer to the same device or devices of the same type.

[0084] The online virtual experience server 102 may include (but is not limited to) a virtual experience engine 104, one or more virtual experiences 106, and a graphics engine 108. In some embodiments, the graphics engine 108 may be a system, application, or module that allows the online virtual experience server 102 to provide graphics and animation capabilities. In some embodiments, the graphics engine 108 may perform the following combined... Figure 2 and Figure 3 The flowchart shown describes one or more operations. Client device 110 may include virtual experience application 112 and input / output (I / O) interface 114 (e.g., input / output device). Input / output device may include one or more of a microphone, speaker, headphones, display device, mouse, keyboard, game controller, touch screen, virtual reality console, etc.

[0085] Developer device 130 may include virtual experience application 132 and input / output (I / O) interface 134 (e.g., input / output device). Input / output device may include one or more of the following: microphone, speaker, headphones, display device, mouse, keyboard, game controller, touch screen, virtual reality console, etc.

[0086] A system architecture 100 is provided for illustration. In different implementations, the system architecture 100 may include components that are compatible with... Figure 1 The same, fewer, more, or different elements configured in the same or different ways as shown.

[0087] In some implementations, network 122 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., Ethernet), or a wireless network (e.g., an 802.11 network). Networks, or wireless LAN (WLAN), cellular networks (e.g., 5G networks, long term evolution (LTE) networks, etc.), routers, hubs, switches, server computers, or combinations thereof.

[0088] In some implementations, data storage 120 may be a non-transitory computer-readable storage device (e.g., random access memory), a cache, a drive (e.g., a hard disk drive), a flash drive, a database system, or another type of component or device capable of storing data. Data storage 120 may also include multiple storage components (e.g., multiple drives or multiple databases) that may span multiple computing devices (e.g., multiple server computers). In some implementations, data storage 120 may include cloud-based storage.

[0089] In some implementations, the online virtual experience server 102 may include a server with one or more computing devices (e.g., a cloud computing system, rack server, server computer, physical server cluster, etc.). In some implementations, the online virtual experience server 102 may be a standalone system, may include multiple servers, or may be part of another system or server.

[0090] In some implementations, the online virtual experience server 102 may include one or more computing devices (such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc.), data storage (e.g., hard disks, memory, databases), networks, software components, and / or hardware components that can be used to perform operations on the online virtual experience server 102 and provide users with access to the online virtual experience server 102. The online virtual experience server 102 may also include website (e.g., web pages) or application backend software that can be used to provide users with access to content offered by the online virtual experience server 102. For example, a user can access the online virtual experience server 102 using a virtual experience application 112 on a client device 110.

[0091] In some implementations, virtual experience session data is generated via an online virtual experience server 102, a virtual experience application 112, and / or a virtual experience application 132, and stored in a data storage device 120. With the permission of the virtual experience participants, the virtual experience session data may include associated metadata, such as: virtual experience identifier; device data associated with the participant; participant demographic information; virtual experience session identifier; chat logs; session start time, session end time, and session duration for each participant; the relative location of the participant's avatar in the virtual experience environment; purchases made by one or more participants in the virtual experience; accessories used by the participant; and so on.

[0092] In some implementations, the online virtual experience server 102 may be a social network providing connectivity between users or a user-generated content system allowing users (e.g., end users or consumers) to communicate with other users on the online virtual experience server 102. In this user-generated content system, communication may include voice chat (e.g., synchronous and / or asynchronous voice communication), video chat (e.g., synchronous and / or asynchronous video communication), or text chat (e.g., 1:1 and / or N:N synchronous and / or asynchronous text-based communication). Records of some or all user communication may be stored in data storage 120 or within the virtual experience 106. Data storage 120 may be used to store chat logs (text, audio, images, etc.) exchanged between participants.

[0093] In some implementations, chat logs are generated via virtual experience applications 112 and / or 132 and stored in data storage 120. Chat logs may include chat content and associated metadata, such as the chat text content for each message with a corresponding sender and receiver; message formatting (e.g., bold, italics, uppercase, etc.); message timestamps; the relative positions of participants' avatars within the virtual experience environment; accessories used by virtual experience participants; etc. In some implementations, chat logs may include multilingual content, and messages in different languages ​​from different sessions within the virtual experience may be stored in data storage 120.

[0094] In some implementations, chat logs can be stored based on timestamps as a form of conversation between participants. In other implementations, chat logs can be stored based on the message initiator.

[0095] In some embodiments of this disclosure, "user" may refer to a single individual. However, other embodiments of this disclosure encompass "user" as an entity controlled by a group of users or an automated source (e.g., a creative user). For example, a group of individual users united as a community or group in a user-generated content system can be considered "users".

[0096] In some implementations, the online virtual experience server 102 may be a virtual game server. For example, the game server may provide single-player or multiplayer games to a user community that can access the “system” described herein, including the online game server 102, data storage 120, and clients, or interact with the virtual experience via network 122 using client devices 110. In some implementations, for example, the virtual experience (including virtual realms or worlds, virtual games, other computer-simulated environments) may be a two-dimensional (2D) virtual experience, a three-dimensional (3D) virtual experience (e.g., a 3D user-generated virtual experience), a virtual reality (VR) experience, or an augmented reality (AR) experience. In some implementations, users can interact with other users (e.g., playing games). In some implementations, the virtual experience can be experienced in real-time with other users of the virtual experience.

[0097] In some implementations, virtual experience participation can refer to the interaction of one or more participants in a virtual experience (e.g., 106) using a client device (e.g., 110), or the presentation of the interaction on a display or other output device (e.g., 114) of the client device 110. For example, virtual experience participation can include interaction with one or more participants in the virtual experience, or the presentation of the interaction on a display of the client device.

[0098] In some implementations, virtual experience 106 may include electronic files that can be executed or loaded using software, firmware, or hardware for presenting virtual experience content (e.g., digital media projects) to entities. In some implementations, virtual experience application 112 may be executed and combined with virtual experience engine 104 to render virtual experience 106. In some implementations, virtual experience 106 may have a set of common rules or common goals, and the environment of virtual experience 106 may share this set of common rules or common goals. In some implementations, different virtual experiences may have different rules or goals from each other.

[0099] In some implementations, a virtual experience may have one or more environments in which multiple environments can be linked (also referred to herein as a "virtual experience ring" or "virtual environment"). An example of an environment may be a three-dimensional (3D) environment. The one or more environments of virtual experience 106 may be collectively referred to herein as a "world," a "virtual experience world," a "game world," a "virtual world," or a "universe." An example of a world may be a 3D world of virtual experience 106. For example, a user can construct a virtual environment linked to another virtual environment created by another user. Characters in a virtual experience can cross virtual boundaries to enter adjacent virtual environments.

[0100] It should be noted that a 3D environment or 3D world uses graphics that represent a three-dimensional representation of the geometric data representing the virtual experience content (or, regardless of whether a 3D representation of the geometric data is used, at least the virtual experience content is rendered as 3D content). A 2D environment or 2D world uses graphics that represent a two-dimensional representation of the geometric data representing the virtual experience content.

[0101] In some implementations, the online virtual experience server 102 may host one or more virtual experiences 106 and may allow users to interact with the virtual experiences 106 using the virtual experience application 112 on the client device 110. Users of the online virtual experience server 102 can play with, create, interact with, or build virtual experiences 106, communicate with other users, and / or create and build objects of virtual experiences 106 (e.g., also referred to herein as “projects”, “virtual experience objects”, or “virtual experience projects”).

[0102] For example, when generating user-generated virtual projects, users can create characters, character accessories, one or more virtual environments for interactive virtual experiences, or structures used in building virtual experience 106. In some implementations, users can buy, sell, or trade virtual experience objects, such as platform currency (e.g., virtual currency), with other users on online virtual experience server 102. In some implementations, online virtual experience server 102 can transfer virtual experience content to virtual experience applications (e.g., 112). In some implementations, virtual experience content (also referred to herein as "content") can refer to any data or software instructions (e.g., virtual experience objects, virtual experiences, user information, videos, images, commands, media items, etc.) associated with online virtual experience server 102 or virtual experience applications. In some implementations, virtual experience objects (e.g., also referred to herein as "projects," "objects," "virtual objects," or "virtual experience projects") can refer to objects used, created, shared, or otherwise depicted in virtual experience application 106 on online virtual experience server 102 or virtual experience application 112 on client device 110. For example, virtual experience objects may include components, models, accessories, tools, weapons, clothing, buildings, vehicles, currency, flora, fauna, and components of the above (e.g., windows of buildings).

[0103] It should be noted that the online virtual experience server 102 providing the hosting virtual experience 106 is for illustrative purposes. In some embodiments, the online virtual experience server 102 may host one or more media items, which may include communication messages from one user to one or more other users. With user permission and explicit user consent, the online virtual experience server 102 may analyze chat log data to improve the virtual experience platform. Media items may include, but are not limited to, digital videos, digital movies, digital photos, digital music, audio content, melodies, website content, social media updates, e-books, e-magazines, digital newspapers, digital audiobooks, e-journals, web blogs, realsimple syndication (RSS) feeds, e-comic books, software applications, etc. In some embodiments, media items may be electronic files that can be executed or loaded using software, firmware, or hardware used to present digital media items to entities.

[0104] In some implementations, virtual experience 106 may be associated with a specific user or group of users (e.g., a private virtual experience), or it may be widely available to users with access to online virtual experience server 102 (e.g., a public virtual experience). In some implementations, where online virtual experience server 102 associates one or more virtual experiences 106 with a specific user or group of users, online virtual experience server 102 may use user account information (e.g., user account identifiers such as username and password) to associate a specific user with virtual experience 106.

[0105] In some implementations, the online virtual experience server 102 or client device 110 may include a virtual experience engine 104 or a virtual experience application 112. In some implementations, the virtual experience engine 104 may be used for the development or execution of the virtual experience 106. For example, the virtual experience engine 104 may include a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, scripting capabilities, an animation engine, an artificial intelligence engine, networking capabilities, streaming capabilities, storage management capabilities, threading capabilities, scene graph capabilities, or animation video support, and other capabilities. Components of the virtual experience engine 104 may generate commands (e.g., rendering commands, collision commands, physics commands, etc.) to aid in the calculation and rendering of the virtual experience. In some implementations, the virtual experience application 112 of the client device 110 may operate independently and / or collaborate with the virtual experience engine 104 of the online virtual experience server 102.

[0106] In some implementations, both the online virtual experience server 102 and the client device 110 can execute virtual experience engines (104 and 112, respectively). The online virtual experience server 102, using virtual experience engine 104, can execute some or all of the virtual experience engine functions (e.g., generating physics commands, rendering commands, etc.), or offload some or all of the virtual experience engine functions to the virtual experience engine 104 on the client device 110. In some implementations, the ratio between the virtual experience engine functions executed on the online virtual experience server 102 and those executed on the client device 110 for each virtual experience 106 can be different. For example, the virtual experience engine 104 of the online virtual experience server 102 can be used to generate physics commands in the event of a collision between at least two virtual experience objects, while additional virtual experience engine functions (e.g., generating rendering commands) can be offloaded to the client device 110. In some implementations, the ratio of virtual experience engine functions executed on the online virtual experience server 102 and the client device 110 can be changed based on virtual experience conditions (e.g., dynamically). For example, if the number of users participating in a particular virtual experience 106 exceeds a threshold, the online virtual experience server 102 can execute one or more virtual experience engine functions previously executed by the client device 110.

[0107] For example, a user can play a virtual experience 106 on client device 110 and send control commands (e.g., user input such as right, left, up, down, user selection, or character position, and speed information) to online virtual experience server 102. After receiving control commands from client device 110, online virtual experience server 102 can send experience commands (e.g., position and speed information of characters participating in a team experience, or commands such as rendering commands, collision commands, etc.) to client device 110 based on the control commands. For example, online virtual experience server 102 can (e.g., using virtual experience engine 104) perform one or more logical operations on the control commands to generate experience commands for client device 110. In other cases, online virtual experience server 102 can pass one or more control commands from one client device 110 to other client devices participating in virtual experience 106 (e.g., from client device 110a to client device 110b). Client device 110 can use the experience commands and render the virtual experience to be displayed on client device 110's screen.

[0108] In some implementations, control instructions may refer to instructions that direct a user character's actions within a virtual experience. For example, control instructions may include user input for controlling actions within the experience, such as right, left, up, down, user selection, gyroscope position and orientation data, force sensor data, etc. Control instructions may include character position and velocity information. In some implementations, control instructions are sent directly to the online virtual experience server 102. In other implementations, control instructions may be sent from client device 110 to another client device (e.g., from client device 110b to client device 110n), where the other client device generates the experience instructions using a local virtual experience engine 104. Control instructions may include instructions for playing voice communication messages or other sounds from another user on an audio device (e.g., a speaker, headphones, etc.), such as voice communication or other sounds generated using audio spatialization techniques as described herein.

[0109] In some implementations, experience instructions can be instructions that enable client device 110 to render a virtual experience (e.g., a multi-participant virtual experience). Experience instructions can include one or more of user input (e.g., control instructions), character position and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.).

[0110] In some implementations, a character (or typically a virtual experience object) consists of components that are automatically linked together to help the user edit them, and one or more of these components can be selected by the user.

[0111] In some implementations, the character is implemented as a 3D model and includes a surface representation (also called skin or mesh) for drawing the character and a set of interconnected layered bones (also called skeletons or rigs). This skeleton is used to animate the character and simulate its movement and actions. The 3D model can be represented as a data structure, and one or more parameters of the data structure can be modified to change various attributes of the character, such as size (height, width, perimeter, etc.); body shape; movement style; number / type of body parts; proportions (e.g., shoulder-to-hip ratio); head size, etc.

[0112] One or more characters (also referred to herein as "avatars" or "models") may be associated with a user, who can control the characters to facilitate the user's interaction with the virtual experience 106.

[0113] In some implementations, a character may include components such as body parts (e.g., hair, arms, legs, etc.) and accessories (e.g., T-shirts, glasses, decorative images, tools, etc.). In some implementations, customizable body parts of the character include head type, body part type (arms, legs, torso, and hands), face shape, hairstyle, and skin type. In some implementations, customizable accessories include clothing (e.g., shirts, trousers, hats, shoes, glasses, etc.), weapons, or other tools.

[0114] In some implementations, for certain asset types (such as shirts, trousers, etc.), online virtual experience platforms can provide users with access to simplified 3D virtual object models, which are represented by a grid with a low polygon count (e.g., between about 20 and about 30 polygons).

[0115] In some implementations, the user can also control the character's proportions (e.g., height, width, or depth) or the proportions of the character's parts. In some implementations, the user can control the character's proportions (e.g., blocky, anatomical, etc.). Note that in some implementations, the character may not include a virtual character experience object (e.g., body parts), but the user can control the character (without a virtual character experience object) to facilitate user interaction with the virtual experience (e.g., in puzzle games, no rendered character game objects are shown, but the user can still control the character to control actions in the game).

[0116] In some implementations, components (e.g., body parts) can be primitive geometries such as blocks, cylinders, spheres, etc., or other primitive shapes such as wedges, tori, tubes, channels, etc. In some implementations, the creator module can publish user roles for other users of the online virtual experience server 102 to view or use. In some implementations, users can create, modify, or customize roles, other virtual experience objects, virtual experience 106, or virtual experience environments using I / O interfaces (e.g., developer interfaces) and with or without scripts (or with or without application programming interfaces (APIs)). Note that, for illustration, the role description is humanoid. Also note that roles can be of any form, such as vehicles, animals, inanimate objects, or other creative forms.

[0117] In some implementations, the online virtual experience server 102 may store user-created characters in data storage 120. In some implementations, the online virtual experience server 102 maintains a character directory and a virtual experience directory that can be presented to users. In some implementations, the virtual experience directory includes images of virtual experiences stored on the online virtual experience server 102. Additionally, users can select characters (e.g., characters created by the user or other users) from the character directory to participate in selected virtual experiences. The character directory includes images of characters stored on the online virtual experience server 102. In some implementations, one or more characters in the character directory may have been created or customized by the user. In some implementations, the selected character may have character settings that define one or more components of that character.

[0118] In some implementations, a user's role may include component configurations, wherein the configuration and appearance of the components (more generally, the appearance of the role) can be defined by role settings. In some implementations, the role settings of a user's role may be at least partially selected by the user. In other implementations, a user may select a role with default role settings or role settings selected by other users. For example, a user may select a default role from a directory of roles with predefined role settings, and the user may further customize the default role by changing some role settings (e.g., adding a shirt with a custom logo). Role settings can be associated with specific roles via an online virtual experience server 102.

[0119] In some embodiments, client devices 110 may each include computing devices (e.g., personal computers (PCs)), mobile devices (e.g., laptops, mobile phones, smartphones, tablets, or netbooks), network-connected televisions, game consoles, etc. In some embodiments, client devices 110 may also be referred to as "user devices." In some embodiments, one or more client devices 110 may connect to the online virtual experience server 102 at any given time. It should be noted that the number of client devices 110 is provided for illustrative purposes. In some embodiments, any number of client devices 110 may be used.

[0120] In some implementations, each client device 110 may include an instance of the virtual experience application 112. In one implementation, the virtual experience application 112 may allow a user to interact with and use the online virtual experience server 102, such as controlling virtual characters in a virtual experience hosted by the online virtual experience server 102, or viewing or uploading content such as virtual experiences 106, images, video items, web pages, documents, etc. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, render, or navigate content served by the web server (e.g., virtual characters in a virtual environment). In another example, the virtual experience application may be a local application (e.g., a mobile application, app, virtual experience program, or game program) that is installed and executed locally on the client device 110 and allows the user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content (e.g., web pages, media viewers) to the user. In implementations, the virtual experience application may also include an embedded media player embedded in a web page (e.g., [missing information - likely a webpage or media viewer]). (or HTML5 player).

[0121] According to various aspects of this disclosure, a virtual experience application can be an online virtual experience server application used by users to build, create, edit, upload content to the online virtual experience server 102, and interact with the online virtual experience server 102 (e.g., participate in a virtual experience 106 hosted by the online virtual experience server 102). Thus, the virtual experience application can be provided to the client device 110 by the online virtual experience server 102. In another example, the virtual experience application can be an application downloaded from a server.

[0122] In some implementations, each developer device 130 may include an instance of a virtual experience application 132. In one implementation, the virtual experience application 132 may allow the developer user to use and interact with an online virtual experience server 102, such as controlling virtual characters in a virtual experience hosted by the online virtual experience server 102, or viewing or uploading content, such as virtual experiences 106, images, video items, web pages, documents, etc. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, render, or navigate content served by a web server (e.g., virtual characters in a virtual environment). In another example, the virtual experience application may be a native application (e.g., a mobile application, app, virtual experience program, or game program) that is installed and executed locally on the developer device 130 and allows the user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content (e.g., web pages, media viewers) to the user. In implementations, the virtual experience application may also include an embedded media player embedded in a web page (e.g., [missing information - likely a webpage or media viewer]). (or HTML5 player).

[0123] According to various aspects of this disclosure, virtual experience application 132 can be an online virtual experience server application for users to build, create, edit, upload content to online virtual experience server 102, and interact with online virtual experience server 102 (e.g., providing and / or participating in virtual experiences 106 hosted by online virtual experience server 102). Thus, the virtual experience application can be provided to client device 130 by online virtual experience server 102. In another example, game application 132 can be an application downloaded from the server. Virtual experience application 132 can be used to interact with online virtual experience server 102 and gain access to user credentials, user currency, etc., of one or more virtual experiences 106 developed, hosted, or provided by virtual experience developers.

[0124] In some implementations, users can log in to the online virtual experience server 102 through a virtual experience application. Users can access their accounts by providing user account information (e.g., username and password), whereby the user account is associated with one or more roles that can be used to participate in one or more virtual experiences 106 on the online virtual experience server 102. In some implementations, using appropriate credentials, virtual experience developers can gain access to virtual objects within the virtual experience, such as in-platform currencies (e.g., virtual currency) owned or associated with other users, avatars, special abilities, and accessories.

[0125] Typically, if appropriate, functions described in one embodiment as being performed by the online virtual experience server 102 can also be performed by the client device 110 or the server in other embodiments. Furthermore, functions belonging to a particular component can be performed by different components or multiple components operating together. The online virtual experience server 102 can also be accessed as a service provided to other systems or devices through a suitable application programming interface (API), and is therefore not limited to use on a website. Figure 2 -Creating and animing variations of template avatars

[0126] Figure 2 This is a flowchart of an example method for creating and animateing a variant of a template avatar 200, according to some implementation methods. Method 200 may begin at box 210.

[0127] In box 210, a template avatar including the template geometry is obtained. For example, the template geometry can be obtained from the mesh of the template avatar. Information about the template avatar can be obtained from the virtual experience within the virtual environment. Box 210 may be followed by box 220.

[0128] In box 220, a template cage associated with the template avatar is generated. The template cage can be generated by wrapping the cage around the template geometry. The template cage can be a low-resolution cage (where the cage's resolution is lower than that of the template geometry). The template cage can provide a way to infer surface correspondences and establish a deformation field between the existing template skeleton and the new target variant. Box 220 can be followed by box 230.

[0129] In box 230, a target cage can be created from a template cage based on user input. For example, the user sculpts the template cage (e.g., via some Digital Content Creation (DCC) tool or programmatically via a script). This cage creation is the only manual part of the process. Box 230 can be followed by box 240.

[0130] Within box 240, the geometry of the template avatar can be morphed using a target cage to generate the target avatar. Automation techniques morph the underlying geometry and transfer facial expressions. The morphing works by deforming the underlying avatar components, making their rough shapes match the shape changes of the cage. Several methods exist, including spatial deformation via radial basis functions (RBFs) or wrapping deformation.

[0131] However, another technique can include surface deformation, which may involve an optimization framework involving variational optimization, implicit surface tracing, and Laplacian fitting. Variational optimization imposes physical constraints to ensure reasonable results. Implicit surface tracing is a technique that extends implicit skinning to surface-based deformation. Laplacian fitting is a variant of Laplacian surface editing techniques for reproducing the properties of regular geometric primitives. More details about deformation are discussed in this paper. Box 240 may be followed by box 250.

[0132] Within box 250, the rigging and skinning of the target avatar can be adjusted. This operation can include pose / expression transfer. Adjusting the rigging and skinning modifies the joints and skinning to adapt the facial expressions and poses of the initial geometry to the reasonable expressions / poses of the cage deformable geometry.

[0133] Features may include using spatial deformations (e.g., RBF) calculated from the cage deformation step to compute the target geometry vertices for each facial expression and body pose. For linear hybrid skinned skeletons, target vertices computed from pose / expression transfer can be used as constraints for known solvers to adjust joint transformations and skinning. For hybrid shape skeletons, the implementation may compute the vertex delta for each pose / expression. Box 250 may be followed by box 260.

[0134] In box 260, the target avatar can be provided to a three-dimensional (3D) environment (e.g., a virtual game environment). For example, information about the generated mesh can be provided to the 3D environment for avatar rendering, avatar animation, or other applications. Following box 260, the 3D environment can utilize the information about the target avatar to display or animate the target avatar. Figure 3 - Adjust the binding and skinning of the target avatar.

[0135] Figure 3 This is a flowchart illustrating an example method for adjusting the binding and skinning of a target avatar 300 according to some implementations. Method 300 may begin at box 310. Method 300 may correspond to Figure 2 Box 250 provides further details on how to perform target avatar binding and skinning.

[0136] In box 310, the mesh can be converted to a flat panel mesh. This conversion may include stitching the animable face and its skeleton to the head-shape mesh using an automated processing pipeline. The creation of the animable face mesh and skeleton can be simplified to primarily flat, rectangular face surfaces that may include multiple views. Using this technique, no curvature specific to any head shape needs to be addressed during rigging and animation. These flat panel face skeletons can be pseudo-3D (2.5D) because the mouth pouch and internal teeth / tongue components have some depth, and the eye components and other facial features have detached surface components. Box 310 may be followed by box 320.

[0137] In box 320, a flat panel mesh can be deformed into a deformable neutral body. This operation takes a head shape and a flat facial panel skeleton in a neutral pose, and performs UV mapping to produce the deformable neutral body. Therefore, in box 320, the first part of the automated processing pipeline deforms the flat facial panel in a neutral pose into a head shape, and then redirects the skeleton. Such a deformable neutral body can be a generic model of a head, without a skeleton, skin, or pose. Box 320 can be followed by box 330.

[0138] In box 330, retargeting can be performed on the deformable neutral body to obtain a deformable skeleton. For example, retargeting may include performing shape solving on the deformable neutral skeleton. Such shape solving may involve using a Facial Action Coding System (FACS) pose via UV mapping, where UV mapping is a 3D modeling process that projects the surface of a 3D model onto a 2D image to perform texture mapping. The result of shape solving can be a deformable pose shape without a skeleton and without skinning. A flat facial panel skeleton can also provide a joint hierarchy and initial skinning weights.

[0139] After shape solving, joint solving can follow. Joint solving uses deformed pose shapes (without skeletons and skinning) along with joint hierarchies and initial skinning weights. Joint solving solves for joint transformations and skinning weights through SmoothSkinning Decomposition with Rigid Bones (SSDR). Deformed neutral bodies and each pose shape are used as the real shape to optimize joint transformations and vertex skinning weights for each pose, thus constructing a linearly blended skinned skeleton. The result of joint solving can be a deformed facial skeleton. Box 330 can be followed by box 340.

[0140] In box 340, the deformable skeleton can be stitched together to generate a stitched skeleton. The head shape and the deformable facial skeleton are stitched together to form a stitched head with binding and skinning. The skinning weight of the non-panel parts of the head can be zero. Box 340 can be followed by box 350.

[0141] In box 350, skin diffusion can be performed on the sutured skeleton. This skin diffusion can produce the final head. The final head can be a sutured head with binding and skinning. An automated processing pipeline can apply some skin diffusion around the edges of the deformable facial skeleton sutures to achieve smooth deformation decay. After box 350, the target avatar is ready and can be provided for use in box 260. Figure 4 -Workflow for creating head cages and the resulting animable head variants

[0142] Figure 4 An example of a workflow for creating a head cage and the resulting animable head variant 400, according to some implementation methods, is shown. Although Figure 4 The workflow applies to the avatar head, but it can be recognized that some implementations use similar techniques for other parts of the avatar. For example, an existing template head 402 and an existing template cage 404 may exist as templates for new avatars (especially the head of new avatars).

[0143] Figure 4 Several examples of user-sculpted target cages 406 are shown. For example, the sculpted target cages 406 show multiple cages corresponding to the avatar's head, where the facial shapes and facial features (such as eyes, mouth, nose, and ears) have different sizes and shapes. This sculpting can be performed using DCC tools or a language. Figure 4 The diagram shows that each user-carved target cage 406 is provided as input to the automation algorithm 408.

[0144] An automated algorithm 408 generates a corresponding animable head variant 410. This animable head variant 410 has a shape similar to the target cage 406 sculpted by the user. By identifying relevant information from the existing template head 402 and incorporating that information into the animable head variant 410, the animable head variant 410 thus has more detail.

[0145] As a relevant example, Figure 4 Sequences that help illustrate how some implementation methods operate are also shown. For example, Figure 4 The diagram shows that a template 420, user input 422, and cage deformation output 424 may exist. Template 420 may include, for example, a template head 430 (corresponding to an existing template head 402) and a template cage 432 (corresponding to an existing template cage 404). User input 422 may include a target cage 434. Cage deformation output 424 may include a target head 436.

[0146] Specifically, the existing stencil head 402 that can be animated can be graphically represented by a mesh, and a low-resolution existing stencil cage 404 can be created / sculpted or otherwise provided for the existing stencil head 402, for example by wrapping it around a mesh.

[0147] The implementation of workflow 400 may include at least two steps. As a first step, a user may sculpt a first existing template cage 404 into one or more target cages 406. In this first step, the user may sculpt (e.g., create) these target cages 406 using any suitable graphics tool, programmatically via script, or using other methods to define a new rough shape for the part (e.g., one of the low-resolution target cages 406 of the existing template head 402). This first step may involve significantly less specialized knowledge for the user / creator and can therefore be user-friendly and easy to operate.

[0148] As a second step, one or more automated algorithms 408 deform the original geometry of the existing template head 402 into a deformation target, thereby generating one or more head variants 410 while adjusting the joints and skinning. The second step produces an animable head with parts (e.g., mouth, lips, eyes, etc.) that can move correctly during animation.

[0149] For example, in the second step, the automated algorithm 408 deforms the underlying geometry of the existing template head 402 and transfers the facial expressions of the existing template head 402. The automated algorithm 408 also adjusts the joints and skinning to create a target facial skeleton optimized for mobile device runtime performance.

[0150] Using automation algorithm 408 in this way can reduce the time required to create character skeletons and poses from approximately one month to just a few seconds. Figure 4 In the process 400, the existing template head 402 is deformed by carving the existing template cage 404, thereby generating visually different head individuals (e.g., head variant 410), which can automatically present facial expressions.

[0151] Further details are provided for various implementations of the second step, including an automated algorithm 408 that converts the user-sculpted target cage 406 into an animable head variant 410.

[0152] Once the user has sculpted the target cage 406 into its desired shape (e.g., during the first step described above), the following operations of the automated algorithm 408 can be performed on the target cage 406 to create an animated avatar variant.

[0153] The automation algorithm 408 may include cage deformation or cage deformation techniques. This technique deforms the underlying avatar component (e.g., the existing template head 402) such that the rough shape of the avatar component matches the shape variation provided by the target cage 406. Various methods can be used. For example, possible methods include spatial deformation using radial basis functions (RBF), wrapping deformation, or surface deformation. According to various embodiments, the automation algorithm 408 uses surface deformation to perform cage deformation. Some embodiments may involve three aspects of cage deformation, such as variational optimization, implicit surface tracking, and Laplacian fitting.

[0154] Variational optimization is a method that imposes physical constraints to ensure reasonable results. Implicit surface tracing is a technique that extends implicit skinning to surface-based deformation. Laplacian fitting is a variant of the Laplacian surface editing technique used to reproduce the properties of regular geometric primitives. More details about these different methods and their use in cage deformation are provided below. The result of this cage deformation step is the generation of a new target head for the next step in the process (pose / expression transfer), which uses this mesh (from the cage deformation step) to fit the dynamic head skeleton.

[0155] The process begins with one of several predefined templates, which includes a template head (the rendered mesh), a template cage (a low-resolution approximation of the template head), a skin (the joints and weights of the template head), and a Facial Action Coding System (FACS) shape (representing the animated poses of the joints for different facial expressions / phonemes). The result of the entire process is to adapt this well-known template to a fully functional, dynamic head skeleton that matches user-specified modifications.

[0156] The way users specify the desired template head modifications is by providing a variant version of the template cage, known as the target cage. This way, users don't need to directly modify the more complex template head or its skin. Users also don't need to update the relevant joint poses of the FACS shape.

[0157] Similarly, a known high-quality dynamic head skeleton is used as a starting point (template head skeleton). Implementations can limit modifications to the template head skeleton to the feasible possibilities. Such constraints can help ensure that the result is a working dynamic head skeleton.

[0158] Conceptually, the result of the cage deformation step is the application of deformation to the template head mesh, similar to the deformation produced from the template cage to the target cage. One way to view this problem is through so-called spatial deformation. From this perspective, the problem is modeled by imagining functions that deform space such that points located at the vertices of the template cage now transform to points located at the corresponding vertices of the target cage. This deformation function also smoothly deforms the space between each of these points.

[0159] In this way, the deformation function similarly deforms a point located at the vertex of the template head to a new point that defines the shape of the target head. There are many ways to define such spatial deformation functions. One straightforward method is to use radial basis function (RBF) interpolation.

[0160] Another common approach to performing spatial deformation is to use generalized barycentric coordinates (most commonly mean and harmonic coordinates). This approach is not entirely identical to globally supported spatial deformation, but shares many similarities. The challenge with these methods lies in their requirement for defining the shape of the spatial domain (e.g., requiring a closed mesh, typically convex). These methods also involve computational and numerical challenges.

[0161] Another approach to addressing these issues is to "bind" the template head vertices to the surface of the template cage. As the template cage deforms into the target cage (through simple linear shape interpolation), the vertices of the template mesh deform along with it, maintaining their relative offset to the cage surface. This approach can be very effective, and variations of wrap deformation can be used as part of the surface-based deformation employed in the implementation.

[0162] The first aspect of the wrap-around deformation is that the template head must have appropriate surface points on the template cage for association. In some implementations, the cage is not a closed surface. The cage contains holes around the eyes, mouth, and neck. The template head is a closed mesh and does contain the geometry of these areas. Such areas can only be associated with points at the geometric boundaries of these holes on the template cage. This situation leads to less than ideal associations due to reasons including illegal line offset vectors and sensitivity to local coordinate system distortion. One solution is to fill the holes in the cage, but this introduces a second limitation.

[0163] The second limitation is that in some areas of the shape, the normal depth of the offset from the template head to the template cage is challenging. The closer the template head vertex is to the template cage surface, the more meaningful the association with the surface becomes. As the position of the template head vertex increases with the normal distance from the template cage, it becomes increasingly unclear which point on the template cage surface it should be associated with. Furthermore, the correlation between the deformation of the template cage surface and the associated position of the template head vertex also decreases. For example, it may be problematic to determine which point on the template cage surface the vertex at the rear of the mouth should be associated with.

[0164] Another issue with using wrap deformation without modification is that this technique involves associating the template mesh with any target cage shape provided by the user, regardless of how infeasible or unsuitable that shape may be for the skeleton migration stage. Implementations benefit from the ability to limit the degree of deformation of the template mesh, allowing for the possibility of finding a solution within the skeleton migration operation. There is no inherent mechanism in standard wrap deformation that provides this deformation constraint functionality. Typically, the techniques described herein are used when the template and target meshes have similar resolution.

[0165] Surface-based deformation, also known as "shape-aware" deformation, considers the intrinsic differential geometry of the mesh to generate deformations that minimize local shape distortion. Some implementations utilize this type of deformation. This technique enables implementations to achieve physically plausible results where the shape is constrained within feasible limits, which can be solved during pose / expression transfer.

[0166] The initial steps of surface-based deformation begin with a variant of the wrapping deformation. Instead of averaging a local coordinate system to be associated with the template cage, some implementations are directly associated with a coordinate system obtained from the closest triangle in the cyclic subdivision of the template cage.

[0167] To ensure the subdivision surface is free of geometric artifacts such as creases, some implementations first optimize the cage topology by performing greedy edge-flip optimization, which maximizes the minimum angular angle of the triangles while preserving characteristic edges. A similar process is performed on the target cage, transferring the associations to the template cage subdivision to the target cage subdivision. These associations on the target cage subdivision are then evaluated to obtain a wrapper deformed version of the template head.

[0168] From the template head of this wrap-around deformation, some implementations select a sparse subset of delta (the displacement vector between corresponding vertex positions) based on certain criteria. These criteria aim to uniformly distribute and emphasize mesh portions characterizing shape features. For example, delta selection could be based on cage points, curvature points, and UV boundary points. In this way, some implementations use wrap-around deformation as a guide for surface-based deformation. Some implementations take steps to match the deformation defined by the wrap-around deformation, but only to the extent that the implementation can preserve the intrinsic geometric properties of the original template head shape.

[0169] Based on the above techniques, the selection of delta excludes any vertices considered to have low-quality associations in the wrap deformer. These vertices are those that coincide with holes in the template cage or are unreliably offset too much in the normal direction (e.g., inside the mouth or eye socket).

[0170] The Euler-Lagrange equation for minimizing the energy of a thin shell is -ks Δd+k b Δ 2 d = 0 (Equation 1), which can be expressed as a linear system A = -k s L+k b (LM -1 L)(Equation 2). Add weighted delta constraints, which are soft constraints on the partial differential equation (PDE), given... (Equation 3) and Equation 4 is used to solve for the displacement d in the least-squares sense to obtain the solution displacement field, which produces a smooth displacement field defined over the entire manifold of the template head mesh. This displacement field minimizes the area and curvature variations. These curvatures correspond to the Dirichlet energy (also known as stretching) and the Laplace energy (also known as thin plate or bending), respectively. The Dirichlet energy penalizes stretching and is characterized by the Laplace equation. The Laplace energy penalizes bending and is characterized by the biLaplace equation. This surface-based deformation method is a variational surface deformation technique known as thin-shell optimization (as a type of radial basis function optimization).

[0171] In these equations, the tensile stiffness coefficient and the bending stiffness coefficient are respectively determined by k s and k b The Laplace operator is represented by Δ, and the biLaplace operator is represented by Δ. 2 (representing L and LM in Equation 2) -1 Therefore, the implementation minimizes the stretching / bending relative to the displacement (the difference vector between two locations), rather than minimizing these energies at the point locations. The former involves “smoothing” the deformation, while the latter involves “smoothing” the surface itself (thus managing the details). The displacement to be solved is denoted by d, and the 0 on the right-hand side forces the minimization of Equation 1. The system of equations is solved in the least-squares sense under the weighted Dirichlet boundary conditions provided by the delta constraint (Equation 4).

[0172] As before, the constraints do not include delta from regions such as inside the mouth. The solution to Equation 4 can include displacements from these regions that are consistent with the surface properties of the template head under the given constraints.

[0173] The solved displacement field is applied to the template head to generate the target head mesh, thus generating the target head mesh using the template mesh with the solved displacement. The target head can then possess several properties. First, the target head deforms in a manner consistent with the example deformation provided by the mapping from the template cage to the target cage. Second, the target head minimizes the deformation energy, making it locally similar to the template head everywhere (i.e., the target head retains the characteristics of the template head). Third, the target head does not contain any extensive deformations that might exist in the target cage example. Fourth, the target head represents a shape from which a solution that fits the template skin / animation can be found.

[0174] At this point, if the deformation described by the target cage is intended to define a re-shaping of the template mesh while preserving the surface features present in that mesh, then the process is complete. However, if the user intends for the target cage to model new surface features not present in the template mesh, the user may find the current result too restrictive. Depending on the design, the target mesh may not maintain its local offset from the target cage, whereas a target mesh deformed by wrapping it with a standard mesh can. Therefore, modeling new features with the target cage is generally not possible. Further deformation steps may be required to capture some of these new modeled features.

[0175] To recover some modeling features of the target cage lost in surface-based deformation, the technique can extend implicit skinning by tracing the template head mesh embedded in the implicit surface function defined in the template cage.

[0176] An implicit surface is a shape defined by a scalar field. An implicit surface exists at all points where the function (equivalence) is a constant value (most commonly zero). This set is called the zero-level set of the implicit function. All other points in space have non-zero equivalence. The sign of the equivalence indicates whether a point with a non-zero equivalence is inside or outside the surface. Typically, meshes representing the zero-level set are generated using mesh reconstruction techniques such as marching cubes or dual-contouring.

[0177] This implementation does not use the implicit surface alone. Instead, it focuses on the position of the vertices of the template head relative to the level set. By recording the isomorphisms of the template head vertices in the implicit surface field of the template cage, the implementation effectively embeds the template head into the field. Similarly, the implementation can generate an implicit surface field from the target cage and use the target implicit surface to project the vertices of the target head back to their embedded isomorphisms.

[0178] Implicit surfaces possess many useful properties, such as modeling self-contact, defining continuous gradients at every point in space (not just on the surface of the implicit surface), and defining smooth interfaces. However, there is no known direct method to make points embedded in implicit surface functions deform with the deformation of the implicit field.

[0179] Other methods can be used to assist point movement with the field. In some techniques for performing implicit skinning, linear blending skin (LBS) is used to provide this assistance, or "tracking." The tracked point is then projected back to its embedding isomorphism along the gradient of the implicit field. Some implementations employ variations of this technique. However, these implementations use surface-based deformation, as described herein, to provide this tracking.

[0180] By using a surface-based tracking method, the implementation does not simply replace one pre-deformer with another. Instead, the implementation gains additional deformation properties that are not available in implementations using relatively arbitrary deformation techniques such as linear hybrid skin (LBS).

[0181] Surface-based deformation minimizes deformation energy, especially for stretching. Implicit surface projections are constrained to the gradient direction. Therefore, the implementation achieves tangential skin-slip deformation, which is both desirable and difficult to model using techniques such as LBS. In short, surface-based mesh tracing provides a very high-quality and efficient starting point from which to proceed back to the embedded isomorphic projection.

[0182] Furthermore, the delta constraint used in surface-based deformation is directly related to the deformation from the template cage to the target cage, and the implicit surface function is directly related to both the template cage and the target cage. Therefore, the tracking pre-deformation is directly related to the deformation of the implicit surface. This method provides fewer arbitrary tracking sources within the field compared to alternative methods.

[0183] The implementation method can obtain an implicit scalar field defining the implicit surface by sampling the template and the target cage mesh (position and normal), and then applying Hermite radial basis functions. Equation 5 is fitted to these samples such that the function value is 0 at the sample points (i.e. on the surface), and the gradient of the function is aligned with the surface normal at these samples according to Equation 5.

[0184] according to (Equation 6) achieves a final return to the projection onto the embedded equivalent by using Newton's iterations to step along the gradient of the implicit surface field of the target cage. Since this projection is a deformation away from the surface that minimizes deformation energy, the implementation may end up with a surface that is not feasible in the pose fitting stage. However, this is a post-processing operation and can be weighted by the user, and the implementation can enforce truncation weights according to Equation 6 if the pose fitting error exceeds some acceptable threshold.

[0185] A natural extension of this method is to combine the original tracking techniques of LBS tracking with the surface-based approach used in the implementation. Instead of obtaining delta constraints for shell optimization (as a form of radial basis function optimization) from a wrapper-like deformer, some implementations obtain delta constraints from the LBS deformation of the skeletal deformable body. This approach offers the advantages of skin-sliding implementations obtained from surface-based methods, within the context of a more suitable skeletal skin deformation typically used for deformable character bodies.

[0186] Therefore, implicit surface tracing can include generating a first implicit surface based on a template cage, adding the embedding isometry of the vertices of the template avatar to the first implicit surface, generating a second implicit surface based on a target cage, and projecting the vertices of the target avatar toward the corresponding isometry in the second implicit surface based on the first implicit surface and the embedding isometry. In other words, adjusting the vertices so that their isometry is consistent in both the source and target implicit functions. Some implementations can use implicit functions because some implementations can take values ​​from anywhere in the 3D field, not just a uniform isometry of the level set generated by contouring.

[0187] Another consequence of surface-based shell optimization (as a type of radial basis function optimization) is that, by minimizing bending energy, implementations may tend to generate overly smooth surfaces. This can occur when the target surface contains shapes with regular geometries such as cylinders, spheres, or flat panels. By minimizing bending energy, the surface may deviate from the expected shape or even exhibit fluctuations. For inherently more organic target shapes, these deformations fit well and are actually desirable. However, if regular geometry is the desired outcome, the implementation must provide additional post-processing deformations.

[0188] To achieve this, implementations include variations of the well-known Laplacian surface editing technique, known as Laplacian fitting. This technique involves solving a Poisson problem, where the implementation reconstructs the surface based on a modified Laplacian operator designed to reproduce the regular geometry of the target cage. For example, a distorted, flattened disk might exist and can be corrected using Laplacian fitting. This correction can be seen in both isometric and cross-sectional views. Some implementations can modify the Laplacian operator used to reflect the surface features to be reproduced, such as creases and flattened areas, which would otherwise be optimized away.

[0189] The implementation method uses Laplacian fitting to forcibly impose appropriate geometric properties, specifically by: setting Δx i =δ i The so-called delta coordinate constraint in (Equation 7) is a value that reflects the appropriate shape, and the coordinate function that satisfies the Poisson problem is solved (x in Equation 7).

[0190] This approach includes variations of the implementation with a target head that exhibit these appropriate geometric properties. However, if the implementation already has this version of the target head, there is no need to resolve that version of the target head in the first place. However, the implementation actually has something similar to this version of the target head, in the form of an association already created for the wrap deformer.

[0191] The implementation previously evaluated these associations to deform the template mesh into the target mesh, establishing delta constraints for thin-shell optimization (as a type of radial basis function optimization). This evaluation preserves the relative offset of the surface with respect to the cage subdivision surface, which is associated with the cage subdivision surface. However, if the implementation evaluates these associations without preserving the offset, it effectively wraps the head around the cage subdivision surface, which has the local curvature gradient that the implementation is looking for.

[0192] Solving Equation 7 using only these values ​​can only recover the surface of the contracted wrapping, which is insufficient. Instead, the implementation adds positional constraints to the system, such as... (Equation 8) Left side (eft-hand side, LHS), and The right-hand side (RHS) of Equation 9 is shown. The implementation obtains these constraints from the selection of delta constraints used in thin-shell optimization (as a type of radial basis function optimization), using the same weights as the delta constraints (ω in Equations 8 and 9).

[0193] Finally, for coordinate functions (Equation 10) By finding x in Equation 10 that satisfies these constraints, the implementation solves the system in the least-squares sense. The result is a surface that best fits the local curvature gradient constraints obtained from a contracted wrapper version of the target head mesh, and preserves the deformation established in the surface-based deformation and (optionally) weighted implicit surface tracking projection.

[0194] Depending on the user's intent, post-processing can be applied through implicit surface tracking or Laplacian fitting.

[0195] The template head mesh can contain multiple connected components. In addition to the skin, there can be any number of components representing head features, such as eyeballs, teeth, and tongue. These components are typically treated differently from the skin (and may also be treated differently from each other). While these components follow deformations applied to the skin to fit the new shape, the implementation must not distort them in an unnatural way.

[0196] For example, there might be an eyeball component in the template head. If the area around the eye socket is enlarged and moved laterally in the target head, the eyeball component needs to be enlarged and translated in a similar way. Otherwise, the eyeball component will no longer fit the eye socket.

[0197] However, if the implementation simply deforms the eyeball assembly in the same way as the surrounding skin mesh, the eyeball assembly no longer retains its spherical shape (assuming the eyeball assembly was originally spherical rather than a stylized shape). This approach is not suitable for animations where the eyeball rotates around its center as the avatar looks in different directions.

[0198] Teeth are handled slightly differently (typically, upper teeth are modeled as one unit, and lower teeth as another). Problems with teeth may not be related to animation. Instead, issues with tooth modeling are more about the expectations for how to reasonably reshape rigid materials like teeth.

[0199] The implementation addresses the reshaping of these additional components by first allowing them to determine how to deform to fit the skin shape. Then, the implementation fits a rigid affine transformation to the deformed shape and applies the rigid affine transformation to replace the non-rigid deformation (thus "un-deforming" the component's deformation).

[0200] The Procrustes method can be used to compute a rigid transformation fit, which finds the optimal rotation through singular value decomposition (SVD) of the weighted cross-covariance matrix of the vertex positions (stationary and deformed shapes). Depending on the specific specifications of the component, resizing can employ non-uniform scaling for semi-rigid deformations (e.g., teeth) or uniform scaling for fully rigid deformations (e.g., a spherical eyeball).

[0201] Contact with the skin surface is maintained by applying an RBF-based spatial deformation to the local contact interface area of ​​the skin to be maintained, the spatial deformation being defined by the retraction deformation delta of the (semi)rigid component. Figure 5 -Example flat panel mesh and skeleton

[0202] Figure 5 An example of a flat panel grid and skeleton 500 according to some implementations is shown. Figure 5 This demonstrates the use of an automated processing pipeline to stitch an animable face and its skeleton to a head-shaped mesh. (Example) Figure 5 As shown, the creation of an animable facial mesh and skeleton can be simplified to a predominantly flat, rectangular facial surface.

[0203] Figure 5 The diagram shows a front view 510, a side view 520, skeletal joints 530, and a Facial Action Coding System (FACS) pose 540, which, when combined, can collectively create facial expressions. Although Figure 5 The example shows 6 FACS poses 540, but this is just an example, and any number between 85 and 120 poses (again, as an example) can be defined for the skeleton relocation solver.

[0204] use Figure 5 The technique shown does not require handling the curvature of any specific head shape during rigging and animation. Figure 5 These flat panel facial skeletons can be 2.5D (pseudo-3D) because the mouth pouches and internal teeth / tongue components, as well as the eye components and other facial features that detach from the surface, have a certain depth. A 2.5D perspective refers to a situation where, in a space that is typically simulated and rendered in a 3D digital environment and appears three-dimensional, gameplay or movement is confined to a two-dimensional plane, with little or no access to three-dimensional space.

[0205] The animation used for facial features such as eyes, eyebrows, and nose is primarily 2D, which simplifies rigging, skinning, and animation. By design, the main surfaces of the flat panel facial mesh and skeleton can be squares, so that the xy coordinates of the vertices are exactly the corresponding uv coordinates in UV space.

[0206] After creating the flat panel face mesh and skeleton, the next step is to automate the stitching of the flat panel face skeleton onto the existing head shape. The automated pipeline stitches the flat panel face mesh and skeleton onto the head shape mesh defined by UV mapping, similar to how actual textures are applied to the mesh. Given a flat face panel mesh and a target head shape, this automated pipeline generates a fully bound and animable stitched head. Figure 6 – An automated processing pipeline for deforming the geometry of flat facial panels

[0207] Figure 6 Example 600 of an automated processing pipeline for deforming the geometry of a flat facial panel, according to some embodiments, is shown. Figure 6 An example of an automated processing pipeline for deforming the geometry of a flat face panel, according to some embodiments, is shown. For example, the flat face panel 610 is redirected onto a head shape 620 via acquired UV mapping. This redirection produces a fully bound and animated stitched head with facial expression 630.

[0208] Some of the technical challenges of this processing pipeline and technology may include the following. First, the flat panel is deformed onto an arbitrary shaped surface of a head-shaped mesh. If the flat panel is very thin, it can be directly mapped using UV coordinates and a signed normal distance from the face surface.

[0209] However, the flat face panel is 2.5D and can have an internal mouth pouch with teeth and tongue components. Determining how these components deform when the facial surface is sewn onto a curved surface is a challenge. To address this, the skeleton of the flat face panel is redirected so that the final facial expression on the sewn head looks good (i.e., conforms to the user's intention). Because the implementation is designed to be able to sew the flat face panel to any arbitrary head shape, it can use a general approach that can take into account wide curvature differences and anisotropic scaling. For example, the implementation can be designed to handle a wide variety of head shapes and poses. Figure 7 -More details on automated processing lines

[0210] Figure 7 Further details of an automated processing line 700 according to some embodiments are shown. Figure 7In the automated processing pipeline, the first part deforms the flat facial panel skeleton 720 in a neutral pose into a head shape 710, and then redirects that skeleton. For example, the automated processing pipeline 700 starts with the head shape 710 and the flat facial panel skeleton 720. The head shape 710 and the flat facial panel skeleton 720 are fed to a deformable neutralization process 730 using UV mapping. The result of the deformable neutralization process 730 is a deformable neutral body 740. This deformable neutral body 740 has no skeleton, no skin, and no pose.

[0211] The deformable neutral body 740 can be found as follows: Given a head shape mesh 710, a flat face panel mesh, and a skeleton 712, this step finds the vertices of the flat face panel mesh and skeleton 712 in the neutral pose. To the vertex of the deformable neutral body 740 The mapping. A key part of this solution is defining the domain of this mapping. Extending this to include the relevant input space coordinates, for this problem, the aforementioned relevant input space coordinates are x and y coordinates within the unit square [0,1]. 2 A subset within.

[0212] Therefore, the implementation method was designed to find the spatial deformation function f. deform :D→R 3 This function maps 3D points in world coordinates of the flat face panel 712 to 3D points in deformation space 740. The function is constrained such that for each vertex i, the implementation has f. deform ∶(b (i) ) = c (i) There are many different f deform The options that satisfy the above constraints include radial basis functions, mean coordinates, harmonic coordinates, and Green's coordinates.

[0213] However, for the problem discussed in this paper, the implementation can use UV mapping and the signed distance from the main facial surface to find this mapping. A flat facial panel is constructed such that the xy coordinates (b x ,b y This is directly mapped to UV coordinates, and for points on the main face surface, the z-coordinate is equal to zero. Specifically, let c surface =g map (u,v) represents the 3D coordinates c on the surface of the head shape given UV coordinates u,v∈[0,1]. surface =[c x ,c y ,c z ] T The function.

[0214] In addition, let normalhead (u,v) returns the normal vector on the head surface for UV coordinates. Note that when u,v∈[0,1], the function g... map It is the gold surface on the head shape 710. Then, the spatial deformation is caused by f deform (b) = g map (b x ,b y )+s·b x ·normal head (b x ,b y The expression is given, where s is a scalar factor that takes into account the global proportional difference between the flat facial panel and the facial portion of the head shape.

[0215] In other words, for a point s in the input space, the implementation method first finds a point c on the head surface with the same UV coordinates. surface Then, along the surface normal of the head, at the original symbolic distance b on the flat face panel. z The scaling factor is used for translation. Using f deform The implementation method can calculate the deformable neutral body 740 from the flat face panel neutral body.

[0216] The deformable neutral body 740 is provided as input to the redirection process 750. Additionally, joint and skin information 742 (including joint hierarchy and initial skin weights) is also provided to the redirection process 750. The redirection process 750 includes performing a shape solving operation 752 on the deformable neutral body 740. In the shape solving operation 752, a mapping f, as calculated in the previous section, may exist. deform This mapping maps each vertex in each FACS pose of the flat panel face to the deformation space. This produces a set of 754 deformation pose shapes, with one deformation pose shape corresponding to each FACS pose.

[0217] Each target pose shape in the deformable neutral body 740 and the deformable pose shape set 754 is used as the real shape to optimize the joint transformations and vertex skinning weights for each pose, thereby constructing a linear hybrid skinned skeleton. The implementation reuses the joint structure of the original flat facial panel skeleton 720 and uses iterative techniques to find the corresponding joint transformations and vertex skinning weights.

[0218] The implementation first solves the shape of each FACS pose in the shape solving step, and then uses these shapes as constraints to optimize joint transformations and skinning in the joint solving operation 756. This method effectively decouples the appearance of the FACS pose from the underlying deformer type and parameters that deform the neutral body into the pose. Specifically, for a linear hybrid skinned skeleton, joint solving can efficiently find the optimal combination of rotations and translations to deform the neutral body into the pose shape.

[0219] The joint solving operation 756 manages joint transformations and skinning weights through smooth skinning decomposition with rigid bones (SSDR). The joint solving operation may include using a deformable neutral body and a set of deformable pose shapes as the real shape to construct a linearly blended skinned skeleton for the target avatar. Therefore, the joint solving operation 756 generates a deformable facial skeleton 760 as its output. This deformable facial skeleton 760 can be provided as input to... Figure 8 The following section of the automated processing pipeline is shown. Figure 8 -More details on automated processing lines

[0220] Figure 8 Further details of an automated processing pipeline 800 according to some embodiments are shown. Figure 8 In the second part of the automated processing line, a stitching operation 830 is performed, which stitches the deformable facial skeleton 820 to the head shape 810 to generate a stitched head skeleton 840. The deformable facial skeleton 820 can correspond to... Figure 7 The modified facial skeleton 760 generated by the processing pipeline 700 is discussed. The automated processing pipeline applies some skin diffusion 850 around the edges of the sutures of the modified facial skeleton to achieve smooth deformation attenuation in the final result 860.

[0221] To complete this pipeline, the target facial skeleton is then stitched to the UV-mapped area of ​​the head shape to obtain a complete head. The diffusion of skin weights at the boundaries and even within the deformable facial panels may be relevant to achieving a good-looking deformation on the head shape. This diffusion is particularly relevant to head shapes with a chin. For example, if the skin weights are not properly adjusted, lower teeth may pierce the chin.

[0222] Additional details are now provided regarding specific aspects of suturing and skin diffusion according to various implementation methods. To automate the processing pipeline, the target facial skeleton is sutured to the UV-mapped area of ​​the head shape to obtain a complete head. Skin weights are diffused at the boundaries and even inside the deformable facial panels to achieve a good-looking deformation of the head shape, especially for head shapes with a chin. Otherwise, if the skin weights are not properly adjusted, the lower teeth may penetrate the chin.

[0223] Since the skinning weights exist only on the vertices of the solved face panel, the automated processing pipeline blends these weights outwards across the stitching boundary to ensure that deformation at the original face panel boundary naturally propagates to the surrounding head mesh.

[0224] Ideally, diffusion only involves smoothing outward weights (e.g., a certain number of face neighbors or geodesic distances) while constraining the face panel to remain unchanged. However, this simple rule doesn't work globally. Instead, skinning diffusion needs to depend on the head shape. However, this approach is complex because the original weights are designed to deform the face panel individually, making it difficult for artists to design them to blend with even a single, unstitched head shape, let alone with a wide range of shapes. Therefore, automated processing pipelines use "blending masks" that depend on the head shape. Figure 9 - Achieve shape transfer by leveraging existing dynamic heads

[0225] Figure 9 An example of shape transfer 900 implemented using an existing dynamic head is shown according to some embodiments. For example, a template head 910 and a target head 920 may exist. The template head 910 and the target head 920 may have the same topology and have vertex correspondence.

[0226] Each of the template head 910 and the target head 920 can initially be rendered with a neutral expression. Multiple mappings exist that map the template neutral body to poses. These mappings provide pose 940, which is a variant of template head 910. Because a correspondence exists between template head 910 and target head 920, a pose 950 similar to pose 940 can be generated, but within the context of target head 920.

[0227] Creating the skeleton, skinning, and posing can be difficult and time-consuming. These techniques can leverage the combined characteristics of facial components. The challenge is how, given a template head 910 and a target head 920, the implementation can transfer the skeleton and obtain facial expressions on the target. To address this issue, existing dynamic heads can be utilized.

[0228] The implementation begins with a template neutral body and a target neutral body. The template neutral body head is associated with various poses. Using this information, the target neutral body head can be used as a basis to generate a target head pose, which is a similar pose to the template neutral body head, but for the target head. As explained in more detail herein, such target head poses can be identified based on the same topology and vertex correspondences. Figure 10 - A two-step method including shape transfer and linear blended skin (LBS) skeleton solution.

[0229] Figure 10 An example of a two-step method for solving a 1000-dimensional skeleton, including shape transfer and linear blended skin (LBS), is shown according to some implementations. For example, Figure 10A face with a neutral pose 1010 and various poses 1012 of that face are shown. The neutral pose 1010 and the various poses 1012 are provided to a shape transfer operation 1020. The shape transfer operation 1020 produces a transfer shape 1030 and a transfer pose 1032. For example, these may correspond to a template head and a target head.

[0230] A destination pose 1040 may also exist. The migration shape 1030, migration pose 1032, and destination pose 1040 are provided to the LBS skeleton solving operation 1050. The LBS skeleton solving operation 1050 generates a pose 1060 with a migration skeleton.

[0231] Shape transfer 1020 may include capturing the mapping that maps template neutral volume vertices to target neutral volume vertices. Therefore, there can be n template vertices and n corresponding target vertices. Shape transfer 1020 can use the corresponding spatial deformation method.

[0232] Spatial deformation method to find function f:R 3 →R 3 This function will convert each template vertex x i Mapped to target vertex y i , making y i =f(x) i Radial basis functions (RBFs) are one type of function that can be used (by w). j ,c i ,A,b) parameterization), where y=f(x)=SUM j w j ||xc j ||+Ax+b. Note that f(x) is R. 3 Each x is defined in the template vertex x, not just in the template vertex x. i The definition is as follows. Having vertex-to-vertex correspondences also means that the implementation has triangle-to-triangle correspondences between the template face and the target face. For example, gradient transfer methods may exist to compute the FACS shape. Techniques including Neural Jacobian Fields or as-rigid-as-possible (ARAP++) methods may also exist.

[0233] The Linear Hybrid Skinning (LBS) skeleton solver 1050 takes the skin weights from the original (source) template, the joint hierarchy from the original (source) template, and the template / target vertex position pairs for each pose as a given. The LBS skeleton solver 1050 can use the optimization results to obtain the transformation and updated skin weights for each joint in each pose. Figure 11 - Functions that achieve shape transfer through spatial deformation

[0234] Figure 11 Example functions for performing shape transfer 1100 by spatial deformation according to some embodiments are shown. For example, there may be a function f 1110 (as discussed herein) that causes spatial deformation to map a first face 1112 to a target face 1114. At 1120, f is applied to each vertex of the template geometry in each pose to obtain the corresponding target shape.

[0235] For example, template pose 1122 is mapped to target pose 1124, template pose 1126 is mapped to target pose 1128, template pose 1130 is mapped to target pose 1132, template pose 1134 is mapped to target pose 1136, and template pose 1138 is mapped to target pose 1140. This mapping provides similar poses for the template face, but is for the scene of the target face. Figure 12 -Skeleton transfer technology

[0236] Figure 12 A skeleton transfer technique according to some embodiments is illustrated, which automatically transfers the skeleton, skin, and pose to a target 1200 given a correspondence between neutral facial expressions. For example, Figure 12 The template face 1210, template pose 1212, and template cage 1214 are shown. A correspondence 1216 may exist between the template cage 1214 and the target cage 1218. This correspondence provides a basis for generating a target face 1220 using the target cage 1218.

[0237] Figure 12 Additional aspects of using the correspondence between templates and targets are illustrated. For example, there is a template face 1230 with a neutral pose. Template face 1230 is associated with multiple template poses 1232, having various facial expressions. A vertex correspondence 1234 exists between the template face 1230 in the neutral pose and the target face 1236 in the neutral pose. Therefore, a skeleton transfer operation 1238 can be performed to generate multiple target poses 1240. Template poses 1232 and target poses 1240 have similar facial expressions, but these poses are respectively similar to template face 1230 and target face 1236.

[0238] For example, skeleton migration begins with a template (as the source) associated with a neutral pose, which has been bound and posed in multiple ways. Cage deformation, or UV mapping, provides a deformed version of the template neutral body to the target neutral body. An identification function f precisely maps each vertex of the source neutral body to the corresponding vertex of the target neutral body. Therefore, function f is derived from R... 3 →R 3 The mapping. If f is over the entire R 3As defined above, implementations can transform vertices of any pose of the source to vertices of the target. Some implementations may use radial basis functions (RBFs) with linear kernels as the function f.

[0239] Joint solving can be performed so that the RBF function maps each vertex of the template neutral face to each corresponding vertex of the target neutral face. Therefore, once the recognition function f is established, the function can be appropriately applied by performing skeleton transfer 1238 to generate the target pose 1240 based on the template face 1230 and the target face 1236, based on the vertex correspondence 1234 and the template pose 1232. Figure 13 -Example computing device

[0240] Figure 13 This is a block diagram of an example computing device 1300 according to some implementation methods.

[0241] Figure 13 This is a block diagram of an example computing device 1300 that can be used to implement one or more features described herein. In one example, device 1300 can be used to implement a computer device (e.g., Figure 1 (102 and / or 110), and performs the appropriate method described herein. The computing device 1300 can be any suitable computer system, server, or other electronic or hardware device. For example, the computing device 1300 can be a mainframe computer, desktop computer, workstation, portable computer, or electronic device (portable device, mobile device, cellular phone, smartphone, tablet computer, television, set-top box, personal digital assistant (PDA), media player, gaming device, wearable device, etc.). In some embodiments, the device 1300 includes a processor 1302, a memory 1304, an input / output (I / O) interface 1306, and an audio / video input / output device 1314.

[0242] Processor 1302 may be one or more processors and / or processing circuits to execute program code and control the basic operations of device 1300. "Processor" includes any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. A processor may include a system with a general-purpose central processing unit (CPU), multiple processing units, dedicated circuitry for implementing functions, or other systems. Processing is not limited to a specific geographical location or time. For example, a processor may perform its functions in a "real-time," "offline," or "batch processing mode." Parts of the processing may be executed by different (or the same) processing systems at different times and locations. A computer may be any processor that communicates with memory.

[0243] Memory 1304 is typically disposed in device 1300 for access by processor 1302 and can be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. Memory 1304 is suitable for storing instructions for execution by the processor and is decoupled from and / or integrated with processor 1302. Memory 1304 may store software operated by processor 1302 on server device 1300, including operating system 1308 and one or more applications 1310, such as avatar generation application 1312. In some embodiments, application 1310 may include enabling processor 1302 to perform (or control) the functions described herein (e.g., regarding...). Figure 2 and Figure 3 Instructions (described in part or all of the methods).

[0244] For example, application 1310 may include avatar generation application 1312, as described herein, which can generate avatars on an online virtual experience server (e.g., 102). Software elements in memory 1304 may optionally be stored on any other suitable storage location or computer-readable medium. Furthermore, memory 1304 (and / or other connected storage devices) may store instructions and data used in the features described herein. Memory 1304 and any other type of memory (disk, optical disk, magnetic tape, or other tangible media) may be considered as "memory" or "storage device".

[0245] I / O interface 1306 can provide functionality that enables server device 1300 to interface with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data storage 120), and input / output devices can communicate through interface 1306. In some embodiments, the I / O interface can connect to interface devices including input devices (keyboard, pointing device, touch screen, microphone, camera, scanner, etc.) and / or output devices (display device, speaker device, printer, motor, etc.).

[0246] Audio / video input / output device 1314 may be an input and display device that includes a user input device (e.g., a mouse, etc.) that can be used to receive user input, a display device (e.g., a screen, monitor, etc.) that can be used to provide graphical and / or visual output, and / or a combination thereof.

[0247] For ease of explanation, Figure 13A box is shown for each of the software blocks of processor 1302, memory 1304, I / O interface 1306, and operating system 1308 and virtual experience application 1310. These boxes may represent one or more processors or processing circuits, operating systems, memory, I / O interfaces, applications, and / or software engines. In other embodiments, device 1300 may not have all the components shown and / or may have other elements, including elements that replace those shown herein or other types of elements besides those shown herein. Although online virtual experience server 102 is described as performing the operations described in some embodiments herein, online virtual experience server 102, or any suitable component or combination of components of a similar system, or any suitable one or more processors associated with such a system, may perform the described operations.

[0248] User equipment may also implement and / or be used with the functions described herein. Example user equipment may be a computer device including components similar to device 1300, such as processor 1302, memory 1304, and I / O interface 1306. Operating systems, software, and applications suitable for the client device may be provided in memory and used by the processor. The I / O interface for the client device may be connected to network communication devices and input and output devices, such as a microphone for capturing sound, a camera for capturing images or video, a mouse for capturing user input, a gesture device for recognizing user gestures, a touchscreen for detecting user input, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices. For example, a display device within audio / video input / output device 1314 may be connected to (or included in) device 1300 to display pre-processed and post-processed images as described herein, wherein such a display device may include any suitable display device, such as an LCD, LED, or plasma display screen, CRT, television, monitor, touchscreen, 3D display screen, projector, or other visual display device. Some implementations may provide an audio output device, such as a synthesized voice output or text reading.

[0249] One or more methods described herein (e.g., method 200 and / or method 300) can be implemented by computer program instructions or code executable on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry) and can be stored on a computer program product including a non-transitory computer-readable medium (e.g., a storage medium), such as magnetic, optical, electromagnetic, or semiconductor storage media, including semiconductor or solid-state memories, magnetic tape, removable computer floppy disks, random access memory (RAM), read-only memory (ROM), flash memory, rigid disks, optical disks, solid-state storage drives, etc. Program instructions can also be contained in and provided as electronic signals, for example, in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.) or a combination of hardware and software. Example hardware can be a programmable processor (e.g., a field-programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), etc. One or more methods may be executed as part of or a component of an application running on the system, or as an application or software that runs with other applications and the operating system.

[0250] One or more methods described herein can run in standalone programs that can run on any type of computing device, programs that run on a web browser, or mobile applications (“apps”) that run on mobile computing devices (e.g., mobile phones, smartphones, tablets, wearable devices (watches, armbands, jewelry, headwear, goggles, glasses, etc.), laptops, etc.). In one example, a client / server architecture can be used, whereby the mobile computing device (as a client device) sends user input data to a server device and receives final output data from the server for output (e.g., for display). In another example, all computations can be performed within a mobile application (and / or other applications) on the mobile computing device. In yet another example, computations can be split between the mobile computing device and one or more server devices.

[0251] While specific embodiments have been described herein, these embodiments are for illustrative purposes only and not for limitation. The concepts illustrated in the examples can be applied to other examples and embodiments.

[0252] The functional blocks, operations, features, methods, devices, and systems described in this disclosure can be integrated or divided into different combinations of systems, devices, and functional blocks known to those skilled in the art. Routines for a particular implementation can be implemented using any suitable programming language and programming techniques. Different programming techniques can be employed, such as procedural or object-oriented. The routine can be executed on a single processing device or multiple processors. Although steps, operations, or calculations are presented in a specific order, this order can be varied in different specific implementations. In some implementations, multiple steps or operations shown as being performed sequentially in this specification can be performed simultaneously.

Claims

1. A computer-implemented method for creating variants of a template avatar, the method comprising: Obtain a template avatar, which includes template geometry obtained from the mesh of the template avatar; Generate a template cage associated with the template avatar as a low-resolution approximation wrapped around the template geometry; Create a target cage from a template cage by modifying the template cage based on user input; as well as The template geometry is deformed using a target cage to generate a target avatar that is a variant of the template avatar.

2. The computer implementation method according to claim 1, further comprising: Adjust the rigging and skinning of the target avatar to make it animateable.

3. The computer implementation method according to claim 2, wherein, The template avatar also includes the template head of the template avatar, wherein the target avatar includes the target head of the target avatar, and wherein the adjustment binding and skinning include one or more of the following: Based on the specific pose of the template head, determine the pose of the target head; and Based on the specific facial expressions of the template head, the facial expressions of the target head are determined.

4. The computer implementation method according to claim 2, wherein, Adjustments to the target avatar's bindings and skinning include: Convert the template-based mesh into a flat panel mesh; Based on the neutral pose of the target avatar, the flat panel mesh is deformed into a deformable neutral body; Redirect the deformable neutral body to obtain the deformable skeleton of the template avatar; The deformable skeleton is stitched to the shape of the target avatar to generate a stitched skeleton with binding and skinning; and After suturing, skin diffusion is performed on the sutured skeleton to obtain the target avatar.

5. The computer implementation method according to claim 4, further comprising: Define a spatial deformation function that maps points in the 3D world coordinates of the flat panel mesh to 3D points in the deformable neutral body; as well as Use the spatial deformation function to deform the flat panel mesh into a deformation neutral body.

6. The computer implementation method according to claim 5, wherein, The template avatar is associated with multiple poses encoded via a Facial Action Coding System (FACS), and the retargeting process includes: Shape solving operations are performed using spatial deformation functions to map corresponding poses from multiple poses, thereby generating a set of deformable pose shapes; and Perform joint solving operations, which involve using a set of deformable neutral bodies and deformable pose shapes as the real shape to construct a linear hybrid skinned skeleton for the target avatar.

7. The computer implementation method according to claim 1, wherein, Deforming the template geometry of the template avatar using the target cage to generate the target avatar includes: using at least one surface-based deformation technique.

8. The computer implementation method according to claim 7, wherein, Using at least one surface-based deformation technique includes: performing a wrap deformation to provide a wrap deformation version of the template avatar, and selecting a sparse subset of delta based on the wrap deformation version of the template avatar.

9. The computer implementation method according to claim 7, wherein, At least one surface-based deformation technique includes variational optimization.

10. The computer implementation method according to claim 9, wherein, Variational optimization includes radial basis function optimization to find the displacement field, and applying the displacement field to the template avatar to generate the target avatar.

11. The computer implementation method of claim 10 further includes performing at least one of implicit surface tracking or Laplacian fitting.

12. The computer implementation method according to claim 11, wherein, Performing implicit surface tracing includes: The first implicit surface is generated based on the template cage; Add the embedding equivalent of the vertices of the template to the first implicit surface; Generate a second implicit surface based on the target cage; and Based on the first implicit surface and the embedding isometry of the first implicit surface, the vertices of the target avatar are projected toward the corresponding isometry in the second implicit surface.

13. The computer implementation method according to claim 11, wherein, Laplace fitting involves solving the Poisson problem based on a modified Laplace operator to reconstruct the target avatar. The modified Laplace operator is designed to reproduce the regular geometry of the target cage by generating a surface of the target avatar that satisfies the delta fitting constraint.

14. A non-transitory computer-readable medium storing instructions thereon, the instructions being responsive to execution by a processing device to cause the processing device to perform operations, the operations including: Obtain a template avatar, which includes template geometry obtained from the mesh of the template avatar; Generate a template cage associated with the template avatar as a low-resolution approximation wrapped around the template geometry; Create a target cage from a template cage by modifying the template cage based on user input; as well as The template geometry is deformed using a target cage to generate a target avatar that is a variant of the template avatar.

15. The non-transitory computer-readable medium of claim 14, wherein, The operation also includes adjusting the target avatar's bindings and skinning to make the target avatar animateable.

16. The non-transitory computer-readable medium of claim 15, adjusting the binding and skinning of the target avatar includes: Convert the template-based mesh into a flat panel mesh; Based on the neutral pose of the target avatar, the flat panel mesh is deformed into a deformable neutral body; Redirect the deformable neutral body to obtain the deformable skeleton of the template avatar; The deformable skeleton is stitched onto the shape of the target avatar to generate a stitched skeleton with binding and skinning. as well as After suturing, skin diffusion is performed on the sutured skeleton to obtain the target avatar.

17. The non-transitory computer-readable medium of claim 14, wherein, Deforming the template geometry of the template avatar using the target cage to generate the target avatar includes: using at least one surface-based deformation technique.

18. A system comprising: A memory that stores instructions; as well as A processing device, coupled to memory, is configured to access memory and execute instructions, wherein the instructions cause the processing device to perform operations, including: Obtain a template avatar, which includes template geometry obtained from the mesh of the template avatar; Generate a template cage associated with the template avatar as a low-resolution approximation wrapped around the template geometry; Create a target cage from a template cage by modifying the template cage based on user input; and The template geometry is deformed using a target cage to generate a target avatar that is a variant of the template avatar.

19. The system according to claim 18, wherein, The operation also includes adjusting the target avatar's bindings and skinning to make the target avatar animateable.

20. The system according to claim 19, wherein, Adjustments to the target avatar's bindings and skinning include: Convert the template-based mesh into a flat panel mesh; Based on the neutral pose of the target avatar, the flat panel mesh is deformed into a deformable neutral body; Redirect the deformable neutral body to obtain the deformable skeleton of the template avatar; The deformable skeleton is stitched to the shape of the target avatar to generate a stitched skeleton with binding and skinning; and After suturing, skin diffusion is performed on the sutured skeleton to obtain the target avatar.

Images