Automatically generating an avatar body model

Abstract

Implementations relate to methods, systems, and computer-readable media for automatically generating avatar body models based on three-dimensional (3D) meshes. In some implementations, the method may include: obtaining a 3D mesh of a virtual character; generating a skeleton for the virtual character based on the 3D mesh, wherein each of a plurality of joints is a predefined joint based on a body type of the virtual character; associating each of the plurality of joints of the skeleton with at least one corresponding vertex of a plurality of vertices of the 3D mesh to obtain a skinned 3D mesh; segmenting the skinned 3D mesh into a plurality of body part meshes corresponding to corresponding body parts of the virtual character based on skinning weights associated with the plurality of vertices; and determining a cage mesh for each body part mesh.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 548,291, filed November 13, 2023, entitled “Automatic Setup for Avatar Body,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments generally relate to computer-based virtual experiences and computer graphics, and more specifically, to methods, systems, and computer-readable media for automatically generating avatar body models based on three-dimensional (3D) meshes of virtual avatars. Background Technology

[0003] Some online virtual experience platforms allow users to connect with each other, interact with each other (e.g., within a virtual experience), create virtual experiences, and share information with each other via the internet. Users of online virtual experience platforms can participate in multiplayer environments (e.g., in a virtual 3D environment), design custom environments, design characters, 3D objects and avatars, decorate avatars, and exchange virtual items / objects with other users.

[0004] One of the challenges facing computer graphics is the rigging and animation of avatars rendered in virtual environments. Content creators (developers) can begin with a 3D mesh of an avatar created or designed according to a specific intent, and then must properly rigage, segment, and make the 3D mesh into body part meshes to support accessory attachments. However, generating an animated and accessory-ready avatar body model based on the avatar's 3D mesh remains a technical challenge.

[0005] The background description provided herein is intended to present the context of this disclosure. The work of the currently named inventors described in the background section, and other aspects of the description that may not constitute prior art at the time of filing, are not, by way of express or implied, an admission that they are prior art to this disclosure. Summary of the Invention

[0006] A system of one or more computers can be configured to perform specific operations or actions by having software, firmware, hardware, or a combination thereof installed on the system, which, when operated, causes the system to perform these actions. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform a computer-implemented method comprising: obtaining a three-dimensional (3D) mesh of a virtual character, wherein the 3D mesh includes a plurality of vertices having corresponding positions in 3D space, wherein one or more connections exist between the vertices; generating an avatar body model for the virtual character based on the 3D mesh, wherein the avatar body model includes a skinned 3D mesh, a plurality of body part meshes, and cage meshes corresponding to corresponding body part meshes in the plurality of body part meshes, wherein the generation includes: generating a avatar body model for the virtual character based on the 3D mesh. The process involves: constructing a colored skeleton, wherein the skeleton comprises a plurality of joints connected by corresponding bones, wherein each of the plurality of joints is a predefined joint based on the body type of the virtual character; associating each of the plurality of joints of the skeleton with at least one corresponding vertex of the plurality of vertices of the 3D mesh to obtain the skinned 3D mesh, wherein a corresponding skinning weight is assigned to each of the plurality of vertices; dividing the skinned 3D mesh into a plurality of body part meshes corresponding to the corresponding body parts of the virtual character based on the skinning weights associated with the plurality of vertices; determining the cage mesh for each body part mesh; and animate the avatar body model to depict the movement of the virtual character in a virtual environment.

[0007] In some implementations, the computer-implemented method may further include: receiving the 3D mesh of the virtual character; and simplifying the 3D mesh to reduce the vertex count of the 3D mesh before generating the avatar body model, wherein the simplification includes one or more of filtering, cleaning, or mirroring.

[0008] In some implementations, generating the skeleton may include: applying a trained machine learning model to the 3D mesh to obtain predicted joint positions of the plurality of joints of the skeleton, wherein vertex localization, topological edges, and geodesic edges of the 3D mesh are provided as input to the trained machine learning model; and constructing a skeleton hierarchy for the virtual character based on the predicted joint positions.

[0009] In some implementations, prior to constructing the bone hierarchy, the computer-implemented method may further include performing at least one operation from a group comprising: removing one or more predicted joint locations located outside the bounding box of the 3D mesh; removing one or more isolated predicted joint locations; mirroring the predicted joint locations across a plane of symmetry; and any combination thereof.

[0010] In some implementations, the computer-implemented method may further include: clustering the predicted joint positions to generate the skeleton. In some implementations, constructing the skeleton hierarchy may include: determining whether the predicted joint positions of the plurality of joints are valid; if the predicted joint positions are determined to be valid, connecting the plurality of joints to bones to determine the skeleton hierarchy; and if the predicted joint positions are determined to be invalid, determining the skeleton hierarchy based on either a procedurally generated geometric skeleton or a default skeleton.

[0011] In some implementations, the skin weights can be determined by: mapping the 3D mesh to a reduced 3D mesh, wherein two or more vertices of the 3D mesh correspond to a single vertex of the reduced 3D mesh; applying a trained second machine learning model to the reduced 3D mesh to determine intermediate skin weights for the reduced 3D mesh, wherein the corresponding volumetric geodesic distance between each vertex and a bone pair is provided as input to the trained second machine learning model; and calculating the skin weight for each vertex of the 3D mesh based on the intermediate skin weights and the mapping.

[0012] In some implementations, dividing the skinned 3D mesh into the multiple body part meshes may include determining the vertices of the skinned 3D mesh associated with each body part based on the skinning weight.

[0013] In some implementations, determining the cage mesh may include, for each body part: applying a trained third machine learning model to the body part mesh of that body part to determine a point cloud, the point cloud including a plurality of cage vertices and canonical connectivity between the plurality of cage vertices; forming a plurality of predicted cage mesh parts based on the plurality of vertices and the canonical connectivity of each vertex; and determining the cage mesh by deforming vertices in the predicted cage mesh parts located inside the surface of the body mesh.

[0014] One general aspect includes a non-transitory computer-readable medium having instructions stored thereon, the instructions performing operations when executed, the operations including: obtaining a three-dimensional (3D) mesh of a virtual character, wherein the 3D mesh includes a plurality of vertices having corresponding positions in 3D space, wherein one or more connections exist between the vertices; generating an avatar body model for the virtual character based on the 3D mesh, wherein the avatar body model includes a skinned 3D mesh, a plurality of body part meshes, and cage meshes corresponding to corresponding body part meshes among the plurality of body part meshes, wherein the generation includes: generating a skeleton for the virtual character based on the 3D mesh, wherein the skeleton includes components made of… The system comprises: a plurality of joints connected to a corresponding bone, wherein each of the plurality of joints is a predefined joint based on the body type of the virtual character; associating each of the plurality of joints of the bone with at least one corresponding vertex of the plurality of vertices of the 3D mesh to obtain the skinned 3D mesh, wherein a corresponding skinning weight is assigned to each of the plurality of vertices; dividing the skinned 3D mesh into a plurality of body part meshes corresponding to the corresponding body parts of the virtual character based on the skinning weights associated with the plurality of vertices; determining the cage mesh for each body part mesh; and animate the avatar body model to depict the movement of the virtual character in a virtual environment.

[0015] One general aspect includes a system comprising: a memory storing instructions thereon; and a processing device coupled to the memory, the processing device configured to access the memory and execute the instructions, wherein executing the instructions causes the processing device to perform an operation, the operation including: obtaining a three-dimensional (3D) mesh of a virtual character, wherein the 3D mesh includes a plurality of vertices having corresponding locations in 3D space, wherein one or more connections exist between the vertices; generating an avatar body model for the virtual character based on the 3D mesh, wherein the avatar body model includes a skinned 3D mesh, a plurality of body part meshes, and cage meshes corresponding to corresponding body part meshes in the plurality of body part meshes, wherein the generation includes: based on The 3D mesh is used to generate a skeleton for the virtual character, wherein the skeleton includes multiple joints connected by corresponding bones, wherein each of the multiple joints is a predefined joint based on the body type of the virtual character; each of the multiple joints of the skeleton is associated with at least one corresponding vertex of the multiple vertices of the 3D mesh to obtain the skinned 3D mesh, wherein a corresponding skinning weight is assigned to each of the multiple vertices; based on the skinning weight associated with the multiple vertices, the skinned 3D mesh is divided into multiple body part meshes corresponding to the corresponding body parts of the virtual character; and the cage mesh for each body part mesh is determined; and the avatar body model is animated to depict the movement of the virtual character in a virtual environment. Attached Figure Description

[0016] Figure 1 This is an illustration of an example environment for generating an avatar body model based on a three-dimensional (3D) mesh of the avatar to be rendered on a computing device, according to some implementation methods.

[0017] Figure 2A Examples of an end-to-end workflow for generating avatar body models from avatar-based 3D meshes, according to some implementations, are described.

[0018] Figure 2B An example system architecture for generating avatar body models based on avatar-based 3D meshes is described according to some implementations.

[0019] Figure 3 An example method for generating an avatar body model based on an avatar-based 3D mesh is illustrated according to some implementations.

[0020] Figure 4 This is a schematic diagram illustrating an example of skeleton generation based on a 3D mesh of a virtual character according to some implementation methods.

[0021] Figure 5An example stage is illustrated during the generation of a skeletal hierarchy for an avatar based on an avatar's 3D mesh, according to some implementation methods.

[0022] Figure 6 This is a schematic diagram illustrating the generation of skin weights for an avatar according to some implementation methods.

[0023] Figure 7 This is a schematic diagram illustrating the generation of a body part mesh for an avatar based on an avatar-based 3D mesh according to some implementation methods.

[0024] Figure 8 An example phase during the generation of a body part mesh for an avatar, according to some implementations, is illustrated.

[0025] Figure 9 This is a schematic diagram illustrating the generation of cage meshes according to some implementation methods.

[0026] Figure 10 Example computing devices according to some implementations are illustrated. Detailed Implementation

[0027] In the following detailed description, reference is made to the accompanying drawings, which form part of this document. In the drawings, similar symbols generally identify similar components unless the context otherwise requires. The exemplary embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. As generally described herein and illustrated in the drawings, aspects of this disclosure can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are contemplated herein.

[0028] The embodiments described in the specification using references such as "some embodiments," "embodiments," "example embodiments," etc., may include specific features, structures, or characteristics, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Additionally, when a specific feature, structure, or characteristic is described in connection with an embodiment, whether explicitly described or not, such feature, structure, or characteristic may be implemented in combination with other embodiments.

[0029] Online virtual experience platforms (also known as "user-generated content platforms" or "user-generated content systems") offer users a variety of ways to interact with each other. For example, users on an online virtual experience platform can work together towards a common goal, share various virtual experience projects, and send each other electronic messages. Users on an online virtual experience platform can join virtual experiences, such as games or other experiences, as virtual characters playing specific roles. For example, a virtual character can be part of a team or multiplayer environment, where each character is assigned a specific role and has associated parameters corresponding to that role, such as clothing, armor, weapons, skills, etc. In another example, when a single player is part of a game, the virtual character can be a computer-generated character.

[0030] Virtual experience platforms enable users (developers) to create objects, new games, and / or characters. For example, online game platforms can enable users to create, design, and / or customize new characters (avatars), new animation packs, new 3D objects, etc., and make them available to other users.

[0031] On some virtual platforms, developer users can generate and / or upload three-dimensional (3D) models of virtual characters or avatars (such as meshes and / or textures of the avatar) for use in virtual experiences and for trading, exchanging, or selling on online marketplaces. Other users can utilize and / or modify the models. The model may include a 3D mesh that represents the geometry of an object and includes vertices, and defines edges and faces. The model may additionally include UV maps and / or textures that define surface properties of the virtual character, such as skin color, tattoos, etc.

[0032] Virtual experience platforms can also allow users to create new characters and avatars and animate them. For example, users of a virtual experience platform can create, design, customize new characters, and animate them.

[0033] In some implementations, animation may include a character that moves one or more body parts to simulate movement, such as walking, running, jumping, dancing, fighting, wielding weapons (such as swords), etc. In some implementations, the avatar may generate facial expressions, where body parts (such as the face) are depicted as being in motion. In some scenes, the movement of the character's entire body may be depicted. Animation can correspond to various movements, such as graceful, warrior-like, ballet-like, etc.

[0034] In some implementations, animation of a virtual character can be performed by rendering a sequence of images to create the illusion of movement of one or more parts of the virtual character. In some implementations, specific frames can be used to define the start and end points of movement of an object or virtual character in the animation. Interpolation can be performed between specific frames to determine the sequence of frames (images) that can subsequently be rendered (e.g., on a display device). In some implementations, each specific frame within a specific frame may include the vertices of a 3D mesh of the virtual character.

[0035] In some implementations, the virtual experience platform may include support for animable 3D models. Binding in 3D objects (such as avatars, e.g., humanoid creatures, animals, birds, etc.) is based on an underlying skeletal structure. The animable model may include an internal skeleton or bone structure that drives deformation of the visual geometry associated with the virtual character. The bone structure may include multiple bones that can move or deform to enable the avatar to make various types of expressions and / or movements. For example, animating a virtual character depicting a smile can be performed by translating the vertices of a 3D mesh corresponding to the endpoints of the mouth by a specified amount in the opposite direction. Similarly, animating an avatar walking or running can be performed by moving the vertices of the avatar's body mesh in the direction of movement while deforming the vertices of the avatar's limbs to mimic walking / running movement.

[0036] In some implementations, developers can store various bone deformities as individual poses. In other implementations, animation can be performed by combining individual poses to create expressions and animations.

[0037] In some implementations, avatar binding can be performed, which involves associating the geometry of a polygonal mesh or implicit surface to be driven by the underlying skeleton structure. This conveys the effects of deformation and / or motion of the avatar based on the movement of the underlying skeleton structure.

[0038] Rigging 3D objects (such as characters (avatars)) can utilize standardized skeletons, such as R6-based avatars (with 6 joints for the avatar body) and R15-based avatars (with 15 joints for the avatar body), etc. These standardized skeletons are adapted to the avatar's surface geometry to provide the underlying structure. In some implementations, avatar rigging can be performed automatically. The goal of avatar rigging is to generate aesthetically pleasing geometric deformations of the avatar based on the motion of the skeletons.

[0039] In some implementations, animation routines from one virtual character can be used for other virtual characters (e.g., when they share a common form). In some implementations, the virtual experience platform can provide a set of standardized animation routines for the platform to use.

[0040] In some implementations, the virtual experience platform can provide support for avatars to use accessories. For example, an avatar can be equipped with accessories such as shirts, trousers, and hats. Therefore, support for cage meshes can be provided, which delineate (bound) the inner and outer surfaces of layered accessories. In some implementations, each body part of the avatar can be associated with a corresponding cage mesh component.

[0041] In some implementations, the cage mesh may include an inner cage and an outer cage that enable accessory assets to stretch, fit, and cover the target avatar and / or any existing clothing items already worn by the avatar. For example, the inner cage of a T-shirt may define how the T-shirt stretches and fits the avatar's body, while the outer cage of the T-shirt defines how any additional layered clothing items can fit onto the T-shirt.

[0042] The avatar body model comprises a 3D mesh, skeletal structure, and / or hierarchy, which can support avatar animation through deformation of the mesh vertices and support accessories through a cage mesh surrounding the avatar body.

[0043] The technical goal of automatically generating avatar body models is to associate the avatar's 3D geometry with the underlying skeleton to automatically segment the 3D model into body components and generate cage meshes compatible with the avatar body. Once generated, the avatar body model can be used to provide immersive experiences on virtual experience platforms, including the avatar's animation and accessory attachments.

[0044] One challenge in computer graphics and virtual experience (e.g., game) design is the process of automatically generating an avatar's body model based on its 3D mesh. In many scenarios, content creators (developers) can start with a mesh of the virtual avatar that accurately represents the avatar's surface features, such as its external shape / geometry, texture, etc. However, the avatar's own 3D mesh is insufficient to provide a complete immersive experience because 3D meshes cannot be automatically animated or properly fitted with accessories.

[0045] To achieve a superior user experience, the 3D mesh must be properly bound, skinned, and divided into corresponding body parts, with cage meshes generated for each body part. Dividing the avatar's 3D mesh into different body parts (e.g., torso, hands, legs, head, etc.) and determining a corresponding cage mesh for each body part allows for better customization and more realistic depiction of the avatar, especially for humanoid avatars.

[0046] Avatar rigging involves determining the skeletal hierarchy, which is a set of joints connected by bones to simulate the movement of the avatar's limbs. Avatar skinning refers to determining the skinning weights associated with the corresponding vertices of the avatar's 3D mesh, and these skinning weights are used to determine the amount of deformation of the vertices caused by deformation of one or more joints of the avatar. Accurate determination of bone and skinning weights is necessary to accurately simulate an avatar under external and / or internal forces. However, the rigging and skinning processes can be time-consuming and laborious, requiring manual work, thus hindering developers' ability to efficiently rigate and skin 3D meshes. Rigging and skinning via manual intervention requires expertise in geometry handling, 3D modeling, and body mechanics simulation. This can be challenging for novice users, and even experienced users may face challenges in achieving proper and accurate avatar customization.

[0047] The automated pipeline described in this article enables a larger group of developer users, including those with limited or no skills in avatar body model generation, to create custom and complete avatar body models.

[0048] The goal of virtual experience platform providers is to offer realistic depictions of virtual characters (avatars), and particularly of their physical behavior. An additional goal is to provide content creators with tools that enable them to generate custom virtual characters (avatars).

[0049] A technical challenge lies in providing automated, accurate, scalable, cost-effective, and reliable tools for creating (generating) and editing avatars. The techniques described herein can be used to provide scalable and adaptive technical solutions for generating avatar body models. In some implementations, the automated pipeline may include a first stage that accepts a 3D mesh or geometry representing the avatar as input. The first stage may include bone generation, followed by a second stage in which skinning weights associated with the vertices of the 3D mesh are determined. Subsequently, the 3D mesh can be segmented into individual body part meshes based on the skinning weights. Finally, cage meshes corresponding to the individual body parts of the avatar can be determined. Additionally, in some implementations, a preprocessing stage may be performed to combine multiple texture images into a single texture image.

[0050] According to the technology disclosed herein, an avatar body model is automatically generated based on the avatar's 3D mesh. The generation process includes multiple stages, whereby the avatar's avatar body model (e.g., the avatar's fully rigged skeleton, which includes skinning weights, segmented body part meshes, and corresponding cage meshes) can be generated from the avatar's 3D mesh, which contains only the geometric representation of the avatar.

[0051] In some implementations, skeletal generation rigging is performed based on a simplified 3D mesh of the avatar by applying a trained machine learning model. The trained machine learning model is applied by feeding vertex localization, topological edges, and geodesic edges as input features to obtain predicted vertex offsets (perturbed vertices) relative to potential joint positions (localizations). Based on the obtained vertex offsets, a set of joint positions for the avatar is obtained by performing one or more post-processing stages. Post-processing stages may include: filtering vertices localized outside the 3D mesh; filtering vertices isolated from other vertices; mirroring perturbed vertices across a symmetry plane to simulate joint symmetry; and clustering the vertices.

[0052] In some implementations, the set of joint positions can be validated, for example, using heuristic techniques. Once the set of joint positions is successfully validated, the joints can be connected to the corresponding bones to generate a bone hierarchy. If the joint positions are determined to be invalid, applied geometric bone generation techniques can be used. In some implementations, a default bone can be created based on the bounding box of the input 3D mesh, and this default bone can be used to determine the bone hierarchy.

[0053] A skinning module can be used to generate skinning weights for each vertex based on the input 3D mesh and bone hierarchy. In some implementations, as described herein, the bone hierarchy can be generated automatically. The input 3D meshes can be merged to ensure that there are no broken components. The merged 3D mesh is then reduced to a smaller number of vertices. During reduction, the mapping between the reduced vertices of the merged 3D mesh and the original vertices is stored for later retrieval.

[0054] The skinning model can utilize a second trained ML model, where volumetric geodesic distance (e.g., the shortest path between two vertices in a 3D mesh) is provided as an input feature to the second trained ML model. Predicted skinning weights are obtained from the second trained ML model and transferred to the original merged 3D mesh based on the mapping generated during simplification.

[0055] One or more post-processing techniques can be applied to improve the smoothness of the predicted skin weights. Post-processing techniques may include: neighborhood filtering, which filters out incorrectly predicted skin weights that bind vertices to unconnected joints; outlier removal, which applies flooding to filter out any incorrectly predicted weights that create islands; and average smoothing, which smooths the skin weights for each vertex by averaging the weights of each vertex's neighboring vertices.

[0056] Based on the skinning weights associated with each vertex of the avatar's 3D mesh, the 3D mesh can be segmented into distinct body part meshes associated with the respective body parts. In some implementations, the skinning weights may include skinning weights automatically determined based on the techniques described herein.

[0057] In some implementations, the interpolation boundary (“ideal boundary”) between corresponding parts can be identified based on the skinning weights assigned to vertices. To avoid adding new geometry and edges, the mesh edges closest to the interpolation boundary are identified and discarded to obtain a list of edges that include only significant continuous loops (“neighbor loops”) of connected vertices.

[0058] Guided by neighboring loops, smooth loops for vertices can be constructed by applying shortest path techniques. Therefore, the 3D mesh is treated as a graph, where each mesh edge is a pair of directed graph edges. Smooth loops are determined based on optimizations of the distance from each vertex to its neighboring loop and visual smoothness, where visual smoothness is calculated as a factor of the surface angle between an edge and its predecessor edge. To ensure that smooth loops travel around the volume of the 3D mesh, the chirality of each directed edge is matched as the loop is extended. Then, the 3D mesh of the avatar is divided at the smooth edge loops to create corresponding body part meshes, and hemispheres are added at the holes to create endcaps for each body part mesh.

[0059] In some implementations, the cage envelope can be automatically generated based on a segmented body mesh (e.g., a segmented body mesh generated at a previous stage of the processing pipeline). A point cloud is generated based on vertices in the input 3D mesh. A trained third ML model is utilized, where vertex localization, normals, and part identifiers (e.g., the body part to which a particular vertex belongs) are provided as input features. The third ML model includes an encoder-decoder model, where the encoder operates directly on the point cloud, and the decoder includes stacked convolutional layers to regress the global features generated from the encoder. The output of the trained third ML model is a point cloud corresponding to the cage vertices in canonical order.

[0060] The cage mesh components are generated based on their respective vertices and their canonical connectivity. In some implementations, a post-processing stage can be used to deform the predicted cage by adjusting the positioning of the cage vertices located inside the mesh surface.

[0061] In some implementations, these techniques can be utilized within tools (such as studio tools) that developers can use to manipulate 3D mesh assets of avatars generated based on descriptions (e.g., text prompts, voice prompts, sketches, etc.). In some implementations, the tool can support creators in creating asset body models for, for example, avatars with 3D models (e.g., 3D meshes) already created by users; 3D models (3D meshes) provided via a virtual experience platform; and 3D meshes obtained or purchased from other users.

[0062] In some implementations, the techniques described herein can be used by virtual experience platforms to enable users to modify the properties of 3D objects during their participation in a virtual experience, thereby allowing creators and players to customize 3D objects based on their preferences. This enables in-experience creation, where users (e.g., non-developer users) can leverage these techniques to customize 3D objects for their virtual experiences.

[0063] In some implementations, iterative refinement of the avatar body model can be performed based on modified descriptors (e.g., text prompts) received from the user. For example, in scenarios where the initial generation of the avatar body model does not meet the user's expectations, iterative refinement can be used to provide additional mesh customization via an interactive method. During this iterative process, the user can provide text prompts, for example, to introduce additional descriptions of the avatar, avatar body parts, etc. This allows the user to be guided in the creation direction and achieve satisfactory results for avatar body model customization.

[0064] In some implementations, support for multiple types of input modalities from the user can be provided. For example, there may be scenarios where some parts of the avatar body model can be generated (designed) by the user, while the rest of the avatar body model can be automatically generated using the techniques described herein. For example, the skeletal hierarchy can be specified by the user, while the skinning weights for the vertices can be determined automatically.

[0065] The technology described in this article can be used to enable a wide range of users to create user-generated avatars. It allows even non-professional creators with limited experience in 3D animation and rigging to successfully create and upload their own fully functional and aesthetically pleasing avatars.

[0066] The technology described in this article introduces a new method for generating animated, ready-to-use avatars, enabling users to create a variety of virtual characters that can be animated and support accessories such as layered clothing. The automated process facilitates more efficient and convenient 3D mesh customization of avatars, thereby fostering creativity and enabling a wider range of users to easily create virtual characters.

[0067] Figure 1 This is an illustration of an example environment for generating an avatar body model based on a three-dimensional (3D) mesh of the avatar to be rendered on a computing device, according to some implementation methods. Figure 1 The same reference numerals are used to identify the same elements as in other figures. The characters following the reference numeral (such as "110") indicate in the text that the element with that particular reference numeral is specifically referred to. Reference numerals without subsequent characters in the text (such as "110") refer to any or all elements in the figure that have that reference numeral (e.g., "110" in the text refers to reference numerals "110a", "110b", and / or "110n" in the figure).

[0068] System architecture 100 (also referred to herein as the "system") includes an online virtual experience server 102, a data storage device 120, user devices 110a, 110b, and 110n (generally referred to herein as "user device 110"), and developer devices 130a and 130n (generally referred to herein as "developer device 130"). The virtual experience server 102, content management server 140, data storage device 120, user device 110, and developer device 130 are coupled via network 122. In some embodiments, user device 110 and developer device 130 may refer to the same device or devices of the same type.

[0069] The online virtual experience server 102 may include a virtual experience engine 104, one or more virtual experiences 106, and a graphics engine 108. The user device 110 may include a virtual experience application 112 and an input / output (I / O) interface 114 (e.g., input / output devices). Input / output devices may include one or more of the following: microphone, speaker, headphones, display device, mouse, keyboard, game controller, touchscreen, virtual reality console, etc. Input / output devices may also include accessory devices connected to the user device via a (wired) cable or a wireless connection.

[0070] Content management server 140 may include graphics engine 144 and category controller 146. In some embodiments, the content management server may include multiple servers. In some embodiments, the multiple servers may be arranged hierarchically, for example, based on corresponding priority values ​​assigned to content sources.

[0071] The graphics engine 144 can be used to render one or more objects, such as 3D objects associated with a virtual environment. The classification controller 146 can be used to classify assets such as 3D objects and to detect counterfeit digital assets. The data storage device 148 can be used to store search indexes, model information, etc.

[0072] Developer device 130 may include virtual experience application 132 and input / output (I / O) interface 134 (e.g., input / output device). Input / output device may include one or more of the following: microphone, speaker, headphones, display device, mouse, keyboard, game controller, touch screen, virtual reality console, etc.

[0073] A system architecture 100 is provided for illustration. In different implementations, the system architecture 100 may include configurations compatible with... Figure 1 The same, fewer, more, or different components are configured in the same or different ways as shown.

[0074] In some implementations, network 122 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., Ethernet), a wireless network (e.g., an 802.11 network, a Wi-Fi® network, or a wireless LAN (WLAN)), a cellular network (e.g., a 5G network, a long term evolution (LTE) network, etc.), a router, a hub, a switch, a server computer, or a combination thereof.

[0075] In some implementations, the data storage device 120 may be a non-transitory computer-readable storage device (e.g., random access memory), a cache, a drive (e.g., a hard disk drive), a flash drive, a database system, a cloud storage system, or another type of component or device capable of storing data. The data storage device 120 may also include multiple storage components (e.g., multiple drives or multiple databases) that may span multiple computing devices (e.g., multiple server computers).

[0076] In some implementations, the online virtual experience server 102 may include a server with one or more computing devices (e.g., a cloud computing system, rack server, server computer, physical server cluster, etc.). In some implementations, the online virtual experience server 102 may be a standalone system, may include multiple servers, or may be part of another system or server.

[0077] In some implementations, the online virtual experience server 102 may include one or more computing devices (such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, distributed computing systems, cloud computing systems, etc.), data storage devices (e.g., hard disks, storage devices, databases), networks, software components, and / or hardware components that can be used to perform operations on the online virtual experience server 102 and provide users with access to the online virtual experience server 102. The online virtual experience server 102 may also include websites (e.g., web pages) or application backend software that can be used to provide users with access to the content offered by the online virtual experience server 102. For example, a user can access the online virtual experience server 102 using a virtual experience application 112 on user device 110.

[0078] In some embodiments, the online virtual experience server 102 may be a type of social network that provides connectivity between users or a type of user-generated content system that allows users (e.g., end users or consumers) to communicate with other users on the online virtual experience server 102. In this user-generated content system, communication may include voice chat (e.g., synchronous and / or asynchronous voice communication), video chat (e.g., synchronous and / or asynchronous video communication), or text chat (e.g., synchronous and / or asynchronous text-based communication). In some embodiments of this disclosure, "user" may refer to a single individual. However, other embodiments of this disclosure encompass "user" as an entity controlled by a group of users or an automated source (e.g., creative user). For example, a group of individual users united as a community or group in a user-generated content system can be considered "users".

[0079] In some implementations, the online virtual experience server 102 may be an online game server. For example, the virtual experience server may provide single-player or multiplayer games to a user community that can access or interact with the game via network 122 using user devices 110. In some implementations, for example, the game (also referred to herein as a "video game," "online game," or "virtual game") may be a two-dimensional (2D) game, a three-dimensional (3D) game (e.g., a user-generated 3D game), a virtual reality (VR) game, or an augmented reality (AR) game. In some implementations, users can participate in the game with other users. In some implementations, the game can be played in real-time with other users in the game.

[0080] In some implementations, a game may refer to the interaction of one or more players using a user device (e.g., 110) within a game (e.g., a game as part of virtual experience 106), or it may refer to the presentation of that interaction on a display or other output device (e.g., 114) of the user device 110.

[0081] In some implementations, virtual experience 106 may include electronic files that can be executed or loaded using software, firmware, or hardware configured to present game content (e.g., digital media projects) to entities. In some implementations, virtual experience application 112 may be executed, and virtual experience 106 may be executed in conjunction with virtual experience engine 104. In some implementations, virtual experience (e.g., game) 106 may have a set of common rules or common goals, and the environment of virtual experience 106 may share this set of common rules or common goals. In some implementations, different games may have different rules or goals from each other.

[0082] In some implementations, a virtual experience may have one or more environments in which multiple environments can be linked (also referred to herein as "game environments" or "virtual environments"). An example of an environment may be a three-dimensional (3D) environment. One or more environments of virtual experience application 112 may be collectively referred to herein as "worlds," "game worlds," "virtual worlds," or "universes." An example of a world may be a 3D world of virtual experience 106. For example, a user can construct virtual environments linked to another virtual environment created by another user. Characters in a virtual game can cross virtual boundaries to enter adjacent virtual environments.

[0083] It should be noted that 3D environments or 3D worlds use graphics that represent a three-dimensional representation of the game content using geometric data (or at least render the game content as 3D content, regardless of whether a 3D representation of geometric data is used). 2D environments or 2D worlds use graphics that represent a two-dimensional representation of the game content using geometric data.

[0084] In some implementations, the online virtual experience server 102 may host one or more virtual experiences 106 and allow users to interact with the virtual experiences 106 using the virtual experience application 112 on user device 110. Users of the online virtual experience server 102 can play, create, interact with, or build virtual experiences 106, communicate with other users, and / or create and build objects of the virtual experience 106 (e.g., also referred to herein as “projects”, “game objects”, or “game items”). For example, when generating user-generated virtual items, users can create characters, character accessories, one or more virtual environments for interactive games, or build structures used in the game. In some implementations, users can buy, sell, or trade virtual game objects, such as in-platform currency (e.g., virtual currency), with other users of the online virtual experience server 102. In some implementations, the online virtual experience server 102 may send game content to the virtual experience application (e.g., 112). In some implementations, game content (also referred to herein as "content") may refer to any data or software instructions (e.g., game objects, games, user information, videos, images, commands, media items, etc.) associated with the online virtual experience server 102 or the virtual experience application. In some implementations, game objects (e.g., also referred to herein as "items," "objects," "virtual objects," or "virtual game items") may refer to objects used, created, shared, or otherwise depicted in the virtual experience 106 of the online virtual experience server 102 or the virtual experience application 112 of the user device 110. For example, game objects may include parts, models, characters, accessories, tools, weapons, clothing, buildings, vehicles, currency, flora, fauna, components of the above content (e.g., windows of buildings), etc.

[0085] It should be noted that the online virtual experience server 102 providing the hosting virtual experience 106 is for illustrative purposes and not for limitation. In some embodiments, the online virtual experience server 102 may host one or more media items, which may include communication messages from one user to one or more other users. Media items may include, but are not limited to, digital videos, digital movies, digital photos, digital music, audio content, melodies, website content, social media updates, e-books, e-magazines, digital newspapers, digital audiobooks, e-journals, web blogs, realsimple syndication (RSS) feeds, e-comic books, software applications, etc. In some embodiments, media items may be electronic files that can be executed or loaded using software, firmware, or hardware configured to present digital media items to entities.

[0086] In some implementations, virtual application 106 may be associated with a specific user or a specific group of users (e.g., a private game), or it may be widely available to users with access to the online virtual experience server 102 (e.g., a public game). In some implementations, where the online virtual experience server 102 associates one or more virtual experiences 106 with a specific user or group of users, the online virtual experience server 102 may use user account information (e.g., a user account identifier such as a username and password) to associate a specific user with the virtual experience 106.

[0087] In some implementations, the online virtual experience server 102 or user device 110 may include a virtual experience engine 104 or a virtual experience application 112. In some implementations, the virtual experience engine 104 may be used for the development or execution of a virtual experience 106. For example, the virtual experience engine 104 may include a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, scripting functionality, an animation engine, an artificial intelligence engine, networking functionality, streaming functionality, memory management functionality, threading functionality, scene graph functionality, or video support for movies, and other features. Components of the virtual experience engine 104 may generate commands that aid in the calculation and rendering of the game (e.g., rendering commands, collision commands, physics commands, etc.). In some implementations, the virtual experience application 112 of user device 110 may operate independently, may work in conjunction with the virtual experience engine 104 of online virtual experience server 102, or a combination of both.

[0088] In some implementations, both the online virtual experience server 102 and the user device 110 can execute a virtual experience engine and a virtual experience application (104 and 112, respectively). The online virtual experience server 102, using the virtual experience engine 104, can execute some or all of the virtual experience engine functions (e.g., generating physics commands, rendering commands, etc.), or offload some or all of the virtual experience engine functions to the virtual experience engine 104 on the user device 110. In some implementations, the ratio between the virtual experience engine functions executed on the online virtual experience server 102 and those executed on the user device 110 for each virtual application 106 can be different. For example, the virtual experience engine 104 of the online virtual experience server 102 can be used to generate physics commands in the event of a collision between at least two virtual application objects, while additional virtual experience engine functionality (e.g., generating rendering commands) can be offloaded to the user device 110. In some implementations, the ratio of virtual experience engine functions executed on the online virtual experience server 102 and the user device 110 can be changed based on game conditions (e.g., dynamically). For example, if the number of users participating in a game of a specific virtual application 106 exceeds a threshold, the online virtual experience server 102 can execute one or more virtual experience engine functions previously executed by the user device 110.

[0089] For example, a user can play virtual application 106 on user device 110 and can send control commands (e.g., user input such as right, left, up, down, user selection, or character positioning and speed information) to online virtual experience server 102. After receiving control commands from user device 110, online virtual experience server 102 can send game commands (e.g., positioning and speed information of characters participating in team games, or commands such as rendering commands, collision commands, etc.) to user device 110 based on the control commands. For example, online virtual experience server 102 can (e.g., using virtual experience engine 104) perform one or more logical operations on the control commands to generate game commands for user device 110. In other cases, online virtual experience server 102 can pass one or more control commands from user device 110 to other user devices participating in virtual application 106 (e.g., from user device 110a to user device 110b). User device 110 can use the game commands and render the game to be displayed on the display of user device 110.

[0090] In some implementations, control commands may refer to instructions for in-game actions of a user character. For example, control commands may include user input for controlling in-game actions, such as right, left, up, down, user selection, gyroscope positioning and orientation data, force sensor data, etc. Control commands may include character position and velocity information. In some implementations, control commands are transmitted directly to the online virtual experience server 102. In other implementations, control commands may be transmitted from user device 110 to another user device (e.g., from user device 110b to user device 110n), wherein the other user device generates game commands using a local virtual experience engine 104. Control commands may include instructions for playing voice communication messages or other sounds from another user on an audio device (e.g., a speaker, headphones, etc.), such as voice communication or other sounds generated using audio spatialization techniques as described herein.

[0091] In some implementations, game instructions may refer to instructions that allow user device 110 to render gameplay of a game (such as a multiplayer game). Game instructions may include one or more of the following: user input (e.g., control instructions), character positioning and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.).

[0092] In some implementations, the online virtual experience server 102 may store user-created characters in the data storage device 120. In some implementations, the online virtual experience server 102 maintains a character directory and a game directory that can be presented to users. In some implementations, the game directory includes images of virtual experiences stored on the online virtual experience server 102. Furthermore, users can select characters (e.g., characters created by the user or another user) from the character directory to participate in selected games. The character directory includes images of characters stored on the online virtual experience server 102. In some implementations, one or more characters in the character directory may have been created or customized by the user. In some implementations, the selected character may have character settings for one or more components that define the character.

[0093] In some implementations, a user's role may include the configuration of components, wherein the configuration and appearance of the components (and more generally, the appearance of the role) may be defined by role settings. In some implementations, the role settings of a user's role may be at least partially selected by the user. In other implementations, a user may select a role with default role settings or role settings selected by other users. For example, a user may select a default role from a directory of roles with predefined role settings, and a user may further customize the default role by changing some of the role settings (e.g., adding a shirt with a custom logo). Role settings may be associated with specific roles by the online virtual experience server 102.

[0094] In some implementations, the virtual experience platform can support 3D objects represented by three-dimensional (3D) models and includes a surface representation (also called skin or mesh) for drawing characters or objects and a set of layered interconnected bones (also called skeleton or skeleton). The skeleton can be used to animate objects and simulate their movement. The 3D model can be represented as a data structure, and one or more parameters of the data structure can be modified to change various attributes of the character, such as size (height, width, perimeter, etc.); shape; movement style; number / type of parts; scale, etc.

[0095] In some implementations, the 3D model may include a 3D mesh. The 3D mesh can define the three-dimensional structure of an unidentified virtual 3D object. In some implementations, the 3D mesh may also define one or more surfaces of the 3D object. In some implementations, the 3D object may be a virtual avatar, such as a virtual character, like a humanoid character, animal character, or robot character.

[0096] In some implementations, the mesh can be received (imported) in FBX file format. The mesh file includes data providing dimension data for the polygons including the virtual 3D object and UV mapping data describing how textured portions are attached to the various polygons including the 3D object. In some implementations, the 3D object may correspond to an accessory, such as a hat, weapon, or garment, which is worn by the virtual avatar or otherwise depicted with reference to the virtual avatar.

[0097] In some implementations, the platform allows users to submit (upload) candidate 3D objects for use on the platform. According to some implementations, a virtual experience development environment (developer tools) may be provided by the platform. The virtual experience development environment can provide a user interface that enables developer users to design and / or create virtual experiences (e.g., games). The virtual experience development environment can be a client-based tool (e.g., downloaded and installed on a client device and operated from the client device), a server-based tool (e.g., installed and executed on a server remotely from the client device and accessed and operated by the client device), or a combination of client-based and service-based components.

[0098] A virtual experience development environment can be operated by the developer of the virtual experience (e.g., a game developer or any other person who wishes to create a virtual experience that can be published by an online virtual experience platform and made available to others). The user interface of the virtual experience development environment can be rendered on the client device (e.g., such as a reference). Figure 1 The described developer device 130) displays an interface that allows the creator / developer to interact with the development environment via actions such as typing, highlighting, selecting, dragging and dropping, and clicking, using a mouse, keyboard, or other input device configured to communicate with the user interface. The user interface may include a menu bar, toolbar, workspace pane, and multiple auxiliary panes. Depending on the specific implementation, the user interface may include alternative or additional elements, arrangements, operational features, etc., of the virtual experience development environment beyond those shown and described herein.

[0099] Developers (creators) can use the virtual experience development environment to create virtual experiences. As part of the development process, developers / creators can upload various types of digital content, such as object files (mesh), image files, audio files, short videos, etc., to enhance the virtual experience.

[0100] In implementations where the 3D object is an accessory, data indicating the object's use in the virtual experience can also be received. For example, a "shoe" object may include a note indicating that the object can be depicted as being worn on the feet of a virtual humanoid character, while a "shirt" object may include a note indicating that it can be depicted as being worn on the torso of a virtual humanoid character.

[0101] In some implementations, the 3D model may also include texture information associated with the 3D object. For example, texture information may indicate the color and / or pattern of the outer surface of the 3D object. Texture information can realize different levels of transparency, reflectivity, diffusion, material properties, and fire resistance behavior of textures and meshes associated with the 3D object. Examples of textures include plastic, cloth, grass, a piece of light blue glass, ice, water, concrete, brick, carpet, wood, etc.

[0102] In some embodiments, user equipment 110 may each include a computing device (such as a personal computer (PC)), a mobile device (e.g., a laptop computer, mobile phone, smartphone, tablet computer, or netbook computer), a network-connected television, a game console, etc. In some embodiments, user equipment 110 may also be referred to as a "client device." In some embodiments, one or more user devices 110 may connect to the online virtual experience server 102 at any given time. It should be noted that the number of user devices 110 is provided as an example. In some embodiments, any number of user devices 110 may be used.

[0103] In some implementations, each user device 110 may include an instance of a virtual experience application 112. In one implementation, the virtual experience application 112 may allow a user to use and interact with an online virtual experience server 102, such as controlling virtual characters in a virtual game hosted by the online virtual experience server 102, or viewing or uploading content such as virtual experiences 106, images, video items, web pages, documents, etc. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, render, or navigate content served by a web server (e.g., virtual characters in a virtual environment). In another example, the virtual experience application may be a native application (e.g., a mobile application, app, or game program) that is locally installed and executed on the user device 110 and allows the user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content (e.g., web pages, media viewers) to the user. In implementations, the virtual experience application may also include an embedded media player (e.g., a Flash® player) embedded in a web page.

[0104] In some implementations, the virtual experience application may include an audio engine 116 installed on the user device, enabling sound playback on the user device. In some implementations, the audio engine 116 may work in conjunction with an audio engine 144 installed on a sound server.

[0105] According to various aspects of this disclosure, a virtual experience application can be an online virtual experience server application for users to build, create, edit, upload, and interact with the online virtual experience server 102 (e.g., participate in a virtual experience 106 hosted by the online virtual experience server 102). Thus, the virtual experience application can be provided to user device 110 by the online virtual experience server 102. In another example, the virtual experience application can be an application downloaded from a server.

[0106] In some implementations, each developer device 130 may include an instance of a virtual experience application 132. In one implementation, the virtual experience application 132 may allow the developer user to use and interact with an online virtual experience server 102, such as controlling virtual characters in a virtual game hosted by the online virtual experience server 102, or viewing or uploading content such as virtual experiences 106, images, video projects, web pages, documents, etc. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, render, or navigate content served by the web server (e.g., virtual characters in a virtual environment). In another example, the virtual experience application may be a native application (e.g., a mobile application, app, or virtual experience program) that is installed and executed locally on the user device 130 and allows the user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content (e.g., a web page, a media viewer) to the user. In implementations, the virtual experience application may also include an embedded media player (e.g., a Flash® player) embedded in a web page.

[0107] According to various aspects of this disclosure, virtual experience application 132 may be an online virtual experience server application for users to build, create, edit, upload, and interact with online virtual experience server 102 (e.g., provide and / or play virtual experiences 106 hosted by online virtual experience server 102). Thus, the virtual experience application may be provided to user device 130 by online virtual experience server 102. In another example, virtual experience application 132 may be an application downloaded from a server. Virtual experience application 132 may be configured to interact with online virtual experience server 102 and gain access to user credentials, user currency, etc., of one or more virtual experiences 106 developed, hosted, or provided by the virtual experience application developer.

[0108] In some implementations, users can log in to the online virtual experience server 102 via a virtual experience application. Users can access their accounts by providing user account information (e.g., username and password), where the user account is associated with one or more roles that can be used to participate in one or more virtual experiences 106 on the online virtual experience server 102. In some implementations, using appropriate credentials, virtual experience application developers can gain access to virtual experience application objects, such as in-platform currency (e.g., virtual currency) owned or associated with other users, avatars, special abilities, and accessories.

[0109] Typically, functions described in one implementation as being performed by the online virtual experience server 102 may also be performed by the user device 110 or the server in other implementations, if appropriate. Furthermore, functionality belonging to a particular component may be performed by different components or multiple components operating together. The online virtual experience server 102 can also be accessed as a service provided to other systems or devices through a suitable application programming interface (API), and is therefore not limited to use on a website.

[0110] In some implementations, the online virtual experience server 102 may include a graphics engine 108. In some implementations, the graphics engine 108 may be a system, application, or module that allows the online virtual experience server 102 to provide graphics and animation capabilities. In some implementations, the graphics engine 108 and / or the content management server 140 may perform the following combined... Figure 3 The flowcharts and workflows shown depict one or more of the operations described.

[0111] Figure 2A Examples of an end-to-end workflow for generating avatar body models from avatar-based 3D meshes, according to some implementations, are described.

[0112] like Figure 2A The described workflow includes providing a 3D mesh of a virtual character (avatar) 205 to a skeleton generation module 210. The segmentation module may include a preprocessing module 212, a post-processing module 214, a trained machine learning model 216 (also referred to as a first trained machine learning model 216), and a validation module 218. The skeleton generation module 210 outputs a skeleton hierarchy 220 of the virtual character. The skeleton hierarchy 220 is provided to a skinning module 225. The constraint graph determination module 225 includes a preprocessing module 226, a trained second ML model 228, and a post-processing module 230, and generates a skinned 3D mesh 232 of the virtual character.

[0113] The input 3D mesh 232 of the virtual character is provided to the body part segmentation module 235. The body part segmentation module 235 generates a segmented body part mesh 240 of the virtual character, which is provided to the cage mesh determination module 242.

[0114] The cage mesh determination module 242 includes a preprocessing module 244, a trained third ML model 246, and a postprocessing module 248.

[0115] The output of the workflow is a virtual character avatar body model 250, which includes a skeleton, a skinned 3D mesh, body part meshes corresponding to different body parts, and corresponding cage meshes.

[0116] Figure 2B An example system architecture for generating avatar body models based on avatar-based 3D meshes is described according to some implementations.

[0117] like Figure 2B As depicted, a Content Delivery Network (CDN) 254 is used to store digital assets associated with virtual characters (avatars). A 3D mesh 256 is obtained from the CDN 254 ​​and provided to a skeleton predictor 258, which can be used to implement, for example, reference... Figure 2A The skeleton generation module 210 is described. A skeleton predictor 258 can be used to predict joint positions 260, which can be provided to a (skinning) weight predictor 262. In some implementations, the predicted joint positions can be adjusted via a skeleton and weight editor 264, which allows for manual adjustment and editing of skeleton and skinning weights.

[0118] A skinned 3D mesh 266, including bone hierarchy and skinning weights, is provided to a body part partitioner 268, which can be used to implement, for example, a reference... Figure 2A The body segmentation module 235 is described.

[0119] A body part partitioner 268 can be used to generate a body part mesh 272 based on a skinned 3D mesh 266. The body part mesh 272 can be provided to a cage predictor 274 and alternatively (optionally) to a partition editor 270, which can enable direct editing of the body part mesh 272 (e.g., by a developer user who may want to adjust / edit the body part mesh 272).

[0120] For example, the cage predictor 274 can be used to implement a reference Figure 2AThe cage mesh determination module 242 is described. The cage mesh 278 generated by the cage predictor 274 can be provided to the avatar builder 280, which is also provided with a skinned 3D mesh 266. In some embodiments, the cage mesh 278 can be provided to the cage editor 276 for manual editing of the cage mesh.

[0121] Avatar builder 280 generates avatar body model 282 that can be stored at CDN 252.

[0122] Figure 3 An example method for generating an avatar body model based on an avatar-based 3D mesh is illustrated according to some implementations.

[0123] In some implementations, method 300 may, for example, be referenced in Figure 1 The virtual experience 102 described is implemented. In some implementations, some or all of method 300 can be implemented in (e.g.) Figure 1 The method 300 may be implemented on one or more client devices 110, on one or more developer devices 130, or on one or more server devices 102, and / or on a combination of developer devices, server devices, and client devices. In the described examples, the implementation system includes one or more digital processors or processing circuitry (“processors”) and one or more storage devices (e.g., database 120 or other storage devices). In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 300. In some examples, the first device is described as the block that performs method 300. Some implementations may have one or more blocks of method 300 performed by one or more other devices (e.g., other client devices or server devices) that can transmit results or data to the first device.

[0124] In some implementations, method 300, or a portion thereof, may be initiated automatically by the system. In some implementations, the implementing system is the first device. For example, the method (or a portion thereof) may be executed periodically, or based on one or more specific events or conditions (e.g., receiving a request from a user to generate an avatar body model for the avatar; receiving a 3D mesh of the virtual character (avatar) at the virtual experience platform; a predetermined time period having elapsed since the last execution of method 300; and / or the occurrence of one or more other conditions, which may be specified in settings read by the method). Method 300 may begin at box 310.

[0125] At box 310, a three-dimensional (3D) mesh of the virtual character, comprising multiple vertices and edges, is obtained. In some embodiments, the 3D mesh comprises multiple vertices that are positioned in 3D space, wherein one or more connections exist between the vertices. In some embodiments, the virtual character may be a humanoid character and includes portions corresponding to a head, torso, arms, and legs.

[0126] In some implementations, a user can upload a 3D mesh to a virtual experience platform with the intent of generating an avatar body model. In some implementations, the virtual experience platform allows users to submit (upload) 3D meshes of 3D objects for use on the platform. In some implementations, one or more textures may be provided associated with the 3D mesh. In some implementations, the 3D mesh may include descriptors and / or tags associated with different portions of the 3D mesh, which can provide descriptions or identifiers for the different portions. In some other implementations, the 3D mesh may not include any additional descriptors or tags.

[0127] A 3D mesh is a geometric representation (e.g., a mathematical model) of a virtual character (avatar). In some implementations, a 3D mesh can define one or more surfaces of the avatar. In some implementations, a 3D object can be a humanoid avatar.

[0128] In some implementations, a 3D mesh can be generated by applying a generative machine learning (gen-ML) model to user-provided prompts (e.g., text prompts, voice prompts, sketches, etc.) that specify the elements of the avatar to be generated. Tools provided by virtual experience platforms enable users to apply gen-ML techniques to generate avatar meshes based on provided descriptions.

[0129] According to some implementations, the virtual experience development environment (developer tools) may be provided by the platform. In some implementations, the 3D mesh may be received (imported) in FBX file format. The mesh file may include data that provides size data about the polygons including the virtual 3D object and UV mapping data describing how parts of the texture are attached to the various polygons including the 3D object.

[0130] In some implementations, an avatar body model for a virtual character can be automatically generated based on a 3D mesh. In some implementations, the avatar body model includes a skinned 3D mesh, multiple body part meshes, and a cage mesh corresponding to the respective body part mesh among the multiple body part meshes.

[0131] In some implementations, the received 3D mesh can be simplified to reduce the vertex count of the 3D mesh before generating the avatar body model. Additionally, simplification may include one or more operations, such as filtering, cleaning, or mirroring the mesh vertices.

[0132] In some implementations, a preprocessing stage can be utilized before generating the avatar body model, where one or more textures (e.g., UV maps) associated with the received 3D mesh can be merged into a single texture image. The UV maps of the received 3D mesh can be modified appropriately to match the merged texture image.

[0133] Box 310 can be followed by box 320.

[0134] At box 320, the skeleton of the avatar body model of the virtual character is generated based on a 3D mesh, the skeleton including multiple joints. In some implementations, generating the skeleton for the virtual character may include generating a skeleton including multiple joints connected by corresponding bones. Each joint may be defined by a joint position, which may be specified using a common reference frame (e.g., coordinate axes) shared by the skeleton and the 3D mesh.

[0135] In some implementations, each joint in a plurality of joints can be a predefined joint, defined based on the virtual character's body type. For example, in some implementations, 15 joints can be predefined for a humanoid character. In some implementations, these 15 joints may include: one neck joint, one torso joint, two shoulder joints, two elbow joints, two wrist joints, two hip joints, two knee joints, two ankle joints, and one root joint. In different implementations, depending on the specific implementation, different numbers of joints and body parts can be predefined.

[0136] In some implementations, the received 3D mesh can be normalized by repositioning the bounding box of the received 3D mesh to a standardized (predetermined) size before generating the skeleton. In some implementations, the received 3D mesh (e.g., a 3D mesh uploaded by a user) can be simplified by remeshing to adjust its topology, and optionally streamlined to reduce the number of vertices in the received 3D mesh. Mesh simplification can reduce additional noise from local features on the 3D mesh and enable more accurate skeleton generation.

[0137] In some implementations, a skeleton can be generated by applying a trained machine learning model to a 3D mesh to obtain predicted joint positions for multiple joints of the skeleton. In some implementations, vertex localization, topological edges, and geodesic edges of the 3D mesh can be provided as input to the trained machine learning model.

[0138] In some implementations, the trained machine learning model can be a graph convolutional neural network (GCNN) trained to generate vertex offsets toward potential joint locations for each vertex in a 3D mesh. Based on the input features, the trained machine learning model can provide a corresponding predicted offset (distance) from the vertex to the nearest joint location as output.

[0139] Based on the predicted joint positions, a skeletal hierarchy for the virtual character can be determined. In some implementations, one or more post-processing operations may be performed before determining (building) the skeletal hierarchy. In some implementations, post-processing operations may include: removing one or more vertices located outside the bounding box of the 3D mesh; removing one or more isolated vertices; mirroring vertices across a plane of symmetry; and combinations thereof.

[0140] For example, in some implementations, as part of post-processing, vertices located outside the body mesh can be excluded (filtered), and outliers that are more than a threshold distance from any other vertex can also be excluded. For example, the distance between vertex pairs can be calculated, and vertices whose minimum distance to other vertices is greater than a certain threshold can be filtered out.

[0141] In some implementations, post-processing may include mirroring, thereby mirroring the vertices around an axis (e.g., the Y-axis) to double the number of vertices. After applying filtering and mirroring, the vertices can be clustered into different groups, and the center of each group of vertices can be selected as the final joint position (localization) for a particular joint.

[0142] In some implementations, post-processing may further include clustering the predicted joint locations to generate a skeleton. For example, K-means clustering and / or mean-shift clustering techniques may be used to group perturbation vertices obtained from the trained ML model into clusters, and the cluster center of each cluster may be selected as the corresponding joint location.

[0143] In some implementations, the determined joint positions can be validated before constructing the skeletal hierarchy for the virtual avatar. In some implementations, heuristics can be used to determine whether the predicted joint positions of multiple joints are valid.

[0144] In some implementations, determining the validity of a joint may include verifying the relative positioning of each joint (the predicted joint). In some implementations, verification may include verifying whether joints located on the spine (e.g., head joints, upper trunk joints, and lower trunk joints) form a relatively straight line (e.g., they may fit to a straight line when a predetermined error criterion is met). Additionally, it may be verified whether the shoulder joint is higher than the forearm and hand joints, and / or whether the hip joint (e.g., thigh joint) is higher than the knee and foot joints (e.g., lower leg and foot joints).

[0145] If the predicted joint locations are determined to be valid, multiple joints can be connected to bones to determine the bone hierarchy. If the predicted joint locations are determined to be invalid, the bone hierarchy can be determined based on either a procedurally generated geometric bone or a default bone.

[0146] For example, in some implementations, if the predicted joint positions are determined to be invalid, the bone hierarchy can be determined by applying applied geometry generation techniques to the 3D mesh. Further verification can be performed, and if the predicted joint positions are further determined to be invalid, the bone hierarchy can be determined using default bones. In some implementations, a suitable default bone can be selected using the type and size of the virtual character (e.g., from a set of default bones).

[0147] Figure 4 This is a schematic diagram illustrating an example of skeleton generation based on a 3D mesh of a virtual character according to some implementation methods.

[0148] like Figure 4 As depicted, an input 3D mesh 410 of the virtual character (avatar) is provided to a remeshing and simplification module 420. After remeshing and simplification, a simplified 3D mesh 325 is obtained and provided to a trained ML model 430. Based on the input features obtained from the simplified 3D mesh 425, the predicted joint positions 432 are determined by utilizing the ML model 430.

[0149] Post-processing 440 can be performed on the predicted joint positions 432. Post-processing 440 may include filtering 442, mirroring 444, and clustering 446.

[0150] The joint position 448 obtained after post-processing 440 is verified by the verification module 450. In some embodiments, a manual editing tool 452 may be provided to adjust the joint position.

[0151] If the predicted joint location is determined to be valid, the joint location can be connected to a bone to determine bone level 460. If the joint location is determined not to meet the predetermined validation criteria, bone level 460 can be determined by: procedurally generating 454; or selecting a suitable bone from a set of default bones 456.

[0152] Figure 5 An example stage is illustrated during the generation of a skeletal hierarchy for an avatar based on an avatar's 3D mesh, according to some implementation methods.

[0153] Figure 5 A 3D mesh of the virtual character is depicted in the following stages: a first stage 510, after joint positions are inferred based on a trained ML model, which includes a set of predicted joint positions 515; a second stage 540, after filtering and mirroring 520, which includes a set of joint positions 545; and a third stage 570, after clustering and validation 550, which includes joint positions 585 connected by bones 580.

[0154] Based on the input features provided as input to the trained ML model, a set of predicted joint positions are inferred by the trained ML model (e.g., based on predicted vertex offsets) 515.

[0155] In some embodiments, method 300 may further include: training an ML model (e.g., a graph convolutional neural network). In some embodiments, training includes: training the ML model on a training dataset that includes an avatar's 3D mesh and joint locations, bones, and / or bone hierarchies corresponding to the avatar's 3D mesh. The training dataset may be generated by humans or may be generated from an automatically constructed 3D mesh constructed from known bones and / or various combinations of bones. Box 320 may be followed by box 330.

[0156] At box 330, the skinned 3D mesh of the avatar body model is obtained by determining the corresponding skinning weights and joints associated with each vertex of the skinned 3D mesh.

[0157] In some implementations, skinning weights are determined based on bone hierarchy (e.g., the bone hierarchy determined at box 320) and can be represented as a weight vector for each mesh vertex, indicating the degree of influence of different bones in the avatar's 3D mesh on the vertex.

[0158] In some implementations, obtaining a skinned 3D mesh may include associating each of a plurality of joints with at least one corresponding vertex of a plurality of vertices of the 3D mesh to obtain a skinned 3D mesh, wherein a corresponding skinning weight is assigned to each of the plurality of vertices.

[0159] In some implementations, a 3D mesh can be mapped to a simplified 3D mesh, where two or more vertices of the 3D mesh correspond to a single vertex of the simplified 3D mesh. In some implementations, simplification can be performed by applying a quadratic mesh simplification technique. In some implementations, the mapping used to perform simplification is stored for subsequent retrieval.

[0160] The reason for merging and simplifying the grid is to improve the accuracy of calculating the volumetric geodesic distance used as an input feature for ML and to reduce the runtime of that calculation.

[0161] In some implementations, the input 3D mesh can be merged before simplification to ensure that there are no broken components in the input 3D mesh. For example, in some scenarios, the 3D mesh received by a virtual character (avatar) may include cuts or seams that may need to be repaired. Cuts and / or seams may be due to errors introduced during mesh generation, or they may be artifacts from another technique (process) that may have split the 3D mesh into multiple parts as part of the rendering optimization.

[0162] In some implementations, the seams of the #D mesh can be adjusted to ensure topological continuity before simplification. For example, in some scenarios, the input 3D mesh may include multiple disconnected components, such as multiple separate 3D meshes. For instance, an arm might be modeled as three distinct meshes: an upper arm mesh, a lower arm mesh, and a head mesh. Vertices in each mesh may not be connected to vertices in other meshes. In some implementations, to address connectivity issues, merging can be performed to combine multiple separate meshes into a single mesh, thereby ensuring paths exist between each individual vertex and other vertices.

[0163] In some implementations, a trained second machine learning model can be applied to the reduced 3D mesh to determine intermediate skinning weights for the reduced 3D mesh. In some implementations, the corresponding volumetric geodesic distance between each vertex and a bone pair is provided as input features to the trained second ML model. The output from the trained second ML model may include intermediate skinning weights for each vertex of the reduced 3D mesh.

[0164] Based on the mapping previously used during the 3D mesh reduction, the skin weights for each vertex of the (original, unreduced) 3D mesh can be calculated based on the intermediate skin weights and mapping.

[0165] In some implementations, additional post-processing can be performed based on the skin weights to determine a final set of skin weights. Post-processing may include smoothing the skin weights through neighborhood filtering, outlier removal, and average smoothing.

[0166] In some implementations, neighborhood filtering can be used to identify and filter out vertices that may be assigned incorrectly predicted weights, such as vertices predicted to be bound to unconnected joints. For example, if a vertex's predicted skinning weight is determined to be 0.7 relative to the head joint and 0.3 relative to the foot joint, it can be inferred that the skinning weight associated with the foot joint (0.3 weight) is an incorrect prediction because the head and foot joints are not directly connected. Therefore, the weight on the foot joint can be removed (e.g., set to zero), and the remaining skinning weights can be renormalized.

[0167] In some implementations, outlier removal can be used to identify and filter out outlier vertices. Vertices assigned the maximum skinning weight for a joint are expected to form a single island (e.g., a closed loop connecting vertices assigned a specific skinning weight). Therefore, each joint of a virtual character is expected to have a corresponding island of vertices assigned the maximum skinning weight for that joint. However, in some scenarios, the predicted skinning weights may result in more than one island for a particular joint. In such scenarios, it can be inferred that the smaller island is a misprediction. Therefore, the smaller island is removed, and the skinning weights for the vertices of the removed island are flooded with the skinning weights assigned to neighboring islands (e.g., neighboring vertex islands including the maximum number of included vertices).

[0168] In some implementations, each vertex is assigned to (e.g., grouped into) a set of vertices based on the joint associated with the maximum skinning weight for the vertex. Vertices that are topologically connected within the same set of vertices are considered islands. If multiple islands are identified for the same vertex, the smaller island is removed by flooding it with the skinning weight associated with the boundary vertices of the smaller island.

[0169] In some implementations, average smoothing can be performed, thereby smoothing the skin weights for each vertex by setting the skin weights for each vertex to the average of the skin weights for its neighboring vertices.

[0170] Figure 6 This is a schematic diagram illustrating the generation of skin weights for an avatar according to some implementation methods.

[0171] like Figure 6 As depicted, the input 3D mesh 610 and skeletal hierarchy 612 of the virtual character are provided to the merging module 615. The merged mesh 620 is then provided to the simplification module 622, where the merged mesh is simplified so that the number of vertices in the simplified mesh 624 is less than the number of vertices in the input 3D mesh 610. The mapping 626 used for performing the simplification can be stored (saved) for later retrieval.

[0172] The simplified mesh 624 is provided to the trained ML model 630, which generates intermediate skin weights 632, and then the intermediate skin weights are transferred to the original merged mesh 635.

[0173] One or more post-processing operations 640 (including neighborhood filtering 645, outlier removal 650, and average smoothing 655) can be performed to generate skinning weights 660 for each vertex.

[0174] In some implementations, method 300 may further include training a second ML model 630 using a training dataset. In some implementations, training of the second ML model may be performed using a training dataset that includes a 3D mesh and skinning weights associated with the 3D mesh. In some implementations, the training dataset may be created based on a 3D mesh available on the virtual experience platform (e.g., a mesh of an avatar that has been previously created and uploaded to the platform). Box 330 may be followed by box 340.

[0175] At box 340, the skinned 3D mesh of the avatar body model is divided into multiple body part meshes based on skinning weights. In some implementations, each body part mesh in the body part mesh may correspond to a specific body part of the virtual character, such as the head, torso, left leg, right leg, etc.

[0176] In some implementations, dividing a skinned 3D mesh into multiple body part meshes may include: determining the vertices of the skinned 3D mesh associated with each body part based on skinning weights; and dividing the skinned 3D mesh into multiple body part meshes, each body part mesh including a corresponding vertex associated with the body part. In some implementations, end caps may be generated for holes that may be created through the division and added to the body part meshes.

[0177] Figure 7 This is a schematic diagram illustrating the generation of a body part mesh for an avatar based on an avatar-based 3D mesh according to some implementation methods.

[0178] like Figure 7 As depicted, the interpolation boundary 730 between body parts is determined using an input 3D mesh 710 and skin weights 720. In some embodiments, the interpolation boundary may be referred to as the "ideal boundary" and may indicate the boundary between body parts determined solely by the skin weights.

[0179] In some implementations, vertices of the 3D mesh can be grouped into multiple sets of vertices based on the joints that exert the greatest influence on the vertices. For example, all vertices in which the head joint exerts the greatest influence on the vertices can be grouped into a single set of vertices. The input 3D mesh can be partitioned at the boundaries between these vertex groups.

[0180] In some implementations, cages can be generated and temporary weights can be assigned to each vertex within the cage. The boundaries between cages can be used to split the 3D mesh into corresponding parts. Using a cage-based approach allows for better alignment of the split locations with clothing; for example, the waist of an avatar can be precisely split at the belt position.

[0181] In some implementations, equiweighted lines (lines connecting vertices with the same skinning weight) on the mesh surface can be used to separate a 3D mesh into multiple parts. These equiweighted lines can indicate boundary lines where the mesh transition between bones has the greatest impact.

[0182] Based on the interpolation boundary, a neighboring boundary 740 can be determined, thereby selecting the mesh edge in the input 3D mesh that is closest to the interpolation (ideal) boundary as the neighboring boundary. In some implementations, the loop created by connecting vertices along the neighboring boundary can be called a neighboring loop.

[0183] Based on neighboring boundaries, smooth boundaries (smoothing loops) 750 can be determined (constructed) such that the boundaries between corresponding body component meshes satisfy a predetermined smoothness criterion. In some embodiments, smooth loops can be determined from neighboring boundaries by applying shortest path techniques. In some embodiments, determining smooth loops may include identifying any islands formed by interpolation boundaries and excluding islands during the determination of smooth loops.

[0184] In some implementations, determining smooth circuits may include applying graph theory techniques to a 3D mesh. In some implementations, each mesh edge is treated as a pair of directed graph edges. The distance from each vertex to its neighboring circuit and a smoothness factor (which is associated with visual smoothness) are optimized, where the smoothness factor is calculated as a factor of the surface angle between an edge and its predecessor edge.

[0185] In some implementations, a smoothness factor is calculated during the traversal between each traversal edge and each predecessor edge. In some implementations, directed edges can be used during traversal because traversing edges in one direction can produce better results compared to the opposite direction.

[0186] In some implementations, excessive computation cycles are prevented by discarding edges whose guiding paths include themselves. In some implementations, the chirality of each directed edge is matched to ensure that smooth loops travel around the volume of the mesh.

[0187] Based on smooth boundaries, body part meshes corresponding to the respective body parts can be generated. Any holes created by separating the body part meshes are sealed by the created hemispherical end caps 770. This has the effect of creating closed body part meshes.

[0188] Figure 8 An example phase during the generation of a body part mesh for an avatar, according to some implementations, is illustrated.

[0189] In this illustrative example, the boundary between the two body parts is depicted as being refined between the first stage 810, the second stage 820, the third stage 830, and the final stage 840.

[0190] like Figure 8 The depiction utilizes the skinning weights of mesh vertices to delineate portions of the mesh with different skinning weight values. For example, in the first stage 810, region 815 includes vertices that have different skinning weight values ​​compared to other vertices (different shades indicate differences in skinning weights in different portions of the 3D mesh).

[0191] At the second stage 820, an interpolation boundary 825 is depicted based on the boundary identified between parts with different skin weight levels.

[0192] In the third stage 830, the mesh edge closest to the interpolation boundary 825 is selected to create the neighbor boundary 835. In the final stage 840, the smooth boundary 845 is determined based on the neighbor boundary 835. The smooth boundary 845 can be used to partition a single 3D mesh into body part meshes associated with the corresponding parts of the 3D mesh.

[0193] The combination of bones (and bone levels) and skinning weights can be used to animate and simulate the movement of an avatar in a virtual environment. Box 340 can be followed by box 350.

[0194] At box 350, a cage mesh is determined for each of the multiple body part meshes. In some implementations, determining the cage mesh for each body part mesh may include determining a cage mesh surrounding an input 3D mesh that includes the multiple body parts that have been segmented.

[0195] In some implementations, determining the cage mesh may include applying a trained third machine learning model to the 3D mesh to determine a point cloud comprising multiple cage vertices and canonical connectivity between the multiple cage vertices. In some implementations, the vertex location, normal, and body part type of each vertex in the body part mesh are provided as input features to the third trained machine learning model.

[0196] In some implementations, multiple predicted cage mesh components can be determined based on multiple cage vertices and the corresponding canonical connectivity of each cage vertex. Based on the multiple cage mesh components, the cage mesh for the virtual character can be determined by stitching the individual cage meshes together.

[0197] However, in some scenarios, some vertices of the predicted cage mesh component may lie inside the surface of the avatar's 3D mesh (body mesh). If such a scenario occurs, vertices in the predicted cage mesh component that are inside the surface of the 3D mesh may deform from their initial predicted position to a position outside the surface of the 3D mesh.

[0198] In some implementations, applying deformation may include: applying the As Stiffest As Possible (ARAP) deformation technique, where cage vertices located inside the mesh surface are set as hard constraints and their positions are moved to outside the surface of the 3D mesh of the virtual character (avatar). The remaining vertices (e.g., cage vertices whose predicted positions are outside the 3D mesh) follow the movement of the hard constraints and deform smoothly to determine a final set of cage vertices. The final cage mesh can then be split back into cage mesh components associated with the corresponding body component meshes.

[0199] Figure 9 This is a schematic diagram illustrating the generation of cage meshes according to some implementation methods.

[0200] like Figure 9 As depicted, a segmented body part mesh 910 is provided to a sampling module 920, where the segmented body part mesh is sampled to generate a point cloud. Input features 930 are computed based on the point cloud and are provided to a trained third ML model 940. The point cloud 950 of the cage vertices is obtained as output from the trained third ML model 940.

[0201] The post-processing operation 960 can be performed using the point cloud 950 of the cage vertices and can include (e.g., based on applied ARAP deformation) deformation 965 of the first set of cage vertices located within the 3D mesh surface. The remaining cage vertices (e.g., the second set of cage vertices located outside the 3D mesh surface) can be deformed by following the movement of the first set of cage vertices.

[0202] In some implementations, a cage mesh component 980 can be formed based on a set of cage vertices obtained after performing post-processing 960. The cage envelope 985, including the cage mesh component, can be exported and stored.

[0203] In some implementations, user annotations can be used to enhance the performance of the machine learning model utilized herein. For example, user-provided descriptions, explanatory text, tags, etc., can be used to segment the 3D mesh of a virtual character into body part meshes. In some implementations, the category or type of a virtual avatar can be determined by performing extensive classification on the virtual avatar, and the category of the 3D object can be used to assist the method 300 described herein.

[0204] After box 350, you can have box 360.

[0205] At frame 360, the avatar's body model can be animated to depict the virtual character's movement within the virtual environment. In some implementations, the animation can utilize animation routines provided by the virtual experience platform. In other implementations, the animation can utilize user-provided animation routines.

[0206] In some implementations, animation production may include determining the corresponding deformation of one or more vertices of a 3D mesh based on the underlying skeleton and skinning weights associated with one or more vertices.

[0207] In some implementations, items of layered clothing represented by corresponding 3D meshes can be attached to cage vertices on cage mesh components of a 3D avatar body model. For example, a 3D mesh of a shirt can be attached to cage vertices of the avatar's torso cage mesh to depict an avatar wearing the shirt.

[0208] After box 360, you can have box 370.

[0209] At frame 370, an avatar (virtual character) can be displayed (e.g., on a screen). The avatar's body model of the 3D object can be used for animation and to define images (frames) of the virtual environment that depict the avatar's state and movement within that environment.

[0210] Method 300 or parts thereof may be repeated any number of times using additional input. Boxes 310 through 370 may be executed (or repeated) in a different order than described above and / or one or more steps may be omitted. For example, in some embodiments, boxes 360 through 370 may be omitted. In some embodiments, boxes 320, 330, 340, and / or 350 may be executed by themselves without executing other boxes. For example, only boxes corresponding to bone generation, determining a skinned 3D mesh, or cage mesh generation may be executed based on the corresponding input. Boxes 310 through 350 may be executed at different rates. For example, boxes 310 through 340 may be executed once when a 3D object mesh is received, and boxes 350 through 360 may be executed multiple times based on the 3D mesh received / obtained at box 310. Additionally, boxes 320 through 350 may be repeated if it is determined that the 3D object has changed in the virtual environment, making it necessary to redetermine one or more parts of the avatar body model.

[0211] Figure 10 Example computing devices according to some implementations are illustrated.

[0212] In one example, device 1000 can be used to implement a computer device (e.g., Figure 1 The computing device 1000 (102, 110, and / or 130) performs the suitable method implementation described herein. The computing device 1000 can be any suitable computer system, server, or other electronic or hardware device. For example, the computing device 1000 can be a mainframe computer, desktop computer, workstation, portable computer, or electronic device (portable device, mobile device, cellular phone, smartphone, tablet computer, television, set-top box, personal digital assistant (PDA), media player, gaming device, wearable device, etc.). In some embodiments, the device 1000 includes a processor 1002, a memory 1004, an input / output (I / O) interface 1006, and an audio / video input / output device 1014.

[0213] Processor 1002 may be one or more processors, processing devices, and / or processing circuitry to execute program code and control the basic operations of device 1000. "Processor" includes any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. A processor may include a system with a general-purpose central processing unit (CPU), multiple processing units, dedicated circuitry for implementing functionality, or other systems. Processing is not limited to a specific geographical location or time. For example, a processor may perform its functions in a "real-time," "offline," or "batch processing mode." Parts of the processing may be executed by different (or the same) processing systems at different times and locations. A computer may be any processor that communicates with memory.

[0214] Memory 1004 is typically disposed in device 1000 for access by processor 1002 and can be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. This processor-readable storage medium is suitable for storing instructions for execution by the processor and is separate from and / or integrated with the processor 1002. Memory 1004 may store software operated by processor 1002 on server device 1000, including operating system 1008, one or more applications 1010 (e.g., audio spatialization application, sound application, content management application), and application data 1012. In some embodiments, application 1010 may include enabling processor 1002 to perform (or control) the functions described herein (e.g., regarding...). Figure 3 Instructions for some or all of the methods described.

[0215] For example, application 1010 may include an audio spatialization module, as described herein, which can provide audio spatialization within an online virtual experience server (e.g., 102). Any software in memory 1004 may alternatively be stored in any other suitable storage location or on a computer-readable medium. Furthermore, memory 1004 (and / or other connected storage devices) may store instructions and data used in the features described herein. Memory 1004 and any other type of storage device (disk, optical disk, magnetic tape, or other tangible media) may be considered as a “storage device” or “storage apparatus”.

[0216] I / O interface 1006 provides functionality that enables server device 1000 to interface with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data storage device 120), and input / output devices can communicate via interface 1006. In some embodiments, the I / O interface can connect to interface devices including input devices (keyboard, pointing device, touch screen, microphone, camera, scanner, etc.) and / or output devices (display device, speaker device, printer, motor, etc.).

[0217] The audio / video input / output device 1014 may include a user input device (e.g., a mouse, etc.) that can be used to receive user input, a display device (e.g., a screen, a monitor, etc.) that can be used to provide graphical and / or visual output, and / or a combination of input and display devices.

[0218] For ease of illustration, Figure 10 A box is shown representing each processor 1002, memory 1004, I / O interface 1006, and software boxes 1008 and 1010. These boxes may represent one or more processors, computing instances on a distributed computing system, processing devices or processing circuitry, operating systems, memory, I / O interfaces, applications, and / or software engines. In other embodiments, device 1000 may not have all the components shown and / or may have elements that replace or are other types of elements shown herein. While the online virtual experience server 102 is described as performing the operations described in some embodiments herein, the online virtual experience server 102, or any suitable component or combination of components of a similar system, or any suitable one or more processors associated with such a system, may perform the described operations.

[0219] User equipment may also implement and / or be used with the features described herein. Example user equipment may be a computer device including components similar to device 1000 (e.g., processor 1002, memory 1004, and I / O interface 1006). Operating systems, software, and applications suitable for the user equipment may be provided in memory and used by the processor. The I / O interface for the user equipment may be connected to network communication devices and input / output devices, such as a microphone for capturing sound, a camera for capturing images or video, a mouse for capturing user input, a gesture device for recognizing user gestures, a touchscreen for detecting user input, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices. For example, a display device within audio / video input / output device 1014 may be connected to (or included in) device 1000 to display pre-processed and post-processed images as described herein, wherein such a display device may include any suitable display device, such as an LCD, LED, or plasma display screen, CRT, television, monitor, touchscreen, 3D display screen, projector, or other visual display device. Some implementations may provide an audio output device, such as a speech output of text being read aloud or a synthesized voice.

[0220] One or more methods described herein (e.g., method 300, etc.) can be implemented by computer program instructions or code executable on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry) and can be stored on a computer program product including a non-transitory computer-readable medium (e.g., a storage medium), such non-transitory computer-readable medium being, for example, magnetic, optical, electromagnetic, or semiconductor storage media, including semiconductor or solid-state memory, magnetic tape, removable computer floppy disks, random access memory (RAM), read-only memory (ROM), flash memory, hard disks, optical disks, solid-state storage drives, etc. The program instructions can also be contained in and provided as electronic signals, for example, in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.) or a combination of hardware and software. Example hardware can be a programmable processor (e.g., a field-programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), etc. One or more methods can be executed as part of or a component of an application running on the system, or as an application or software that runs with other applications and the operating system.

[0221] One or more methods described herein can operate in the following ways: a standalone program that can run on any type of computing device; a program that runs on a web browser; or a mobile application (“app”) that runs on a mobile computing device (e.g., a mobile phone, smartphone, tablet, wearable device (watch, armband, jewelry, headwear, goggles, glasses, etc.), laptop computer, etc.). In one example, a client / server architecture can be used, whereby the mobile computing device (as a user device) transmits user input data to a server device and receives final output data from the server for output (e.g., for display). In another example, all computations can be performed within a mobile application (and / or other applications) on the mobile computing device. In yet another example, computations can be split between the mobile computing device and one or more server devices.

[0222] Although the description has been given with reference to specific implementations, these specific implementations are illustrative and not limiting. The concepts illustrated in the examples can be applied to other examples and implementations.

[0223] Note that the functional blocks, operations, features, methods, devices, and systems described in this disclosure can be integrated or divided into different combinations of systems, devices, and functional blocks known to those skilled in the art. Routines for a particular implementation can be implemented using any suitable programming language and programming techniques. Different programming techniques can be employed, such as procedural or object-oriented. The routine can be executed on a single processing device or multiple processors. Although steps, operations, or calculations are presented in a specific order, this order can be varied in different specific implementations. In some implementations, multiple steps or operations shown as being performed sequentially in this specification can be performed simultaneously.

Claims

1. A computer-implemented method, the computer-implemented method comprising: Obtain a three-dimensional (3D) mesh of a virtual character, wherein the 3D mesh comprises a plurality of vertices having corresponding positions in 3D space, wherein one or more connections exist between the vertices; Based on the 3D mesh, an avatar body model for the virtual character is generated, wherein the avatar body model includes a skinned 3D mesh, multiple body part meshes, and cage meshes corresponding to the respective body part meshes among the multiple body part meshes, wherein the generation includes: The skeleton for the virtual character is generated based on the 3D mesh, wherein the skeleton includes multiple joints connected by corresponding bones, and each of the multiple joints is a predefined joint based on the body type of the virtual character. Each of the plurality of joints of the bone is associated with at least one corresponding vertex of the plurality of vertices of the 3D mesh to obtain the skinned 3D mesh, wherein a corresponding skinning weight is assigned to each of the plurality of vertices; Based on the skinning weights associated with the plurality of vertices, the skinned 3D mesh is divided into a plurality of body part meshes corresponding to the corresponding body parts of the virtual character; and Determine the cage mesh for each body part mesh; and The avatar body model is animated to depict the movement of the virtual character in the virtual environment.

2. The computer-implemented method according to claim 1, further comprising: Receive the 3D mesh of the virtual character; as well as Before generating the avatar body model, the 3D mesh is simplified to reduce the vertex count of the 3D mesh, wherein the simplification includes one or more of filtering, cleaning, or mirroring.

3. The computer-implemented method according to claim 1, wherein generating the skeleton comprises: A trained machine learning model is applied to the 3D mesh to obtain predicted joint positions of the plurality of joints of the skeleton, wherein vertex localization, topological edges, and geodesic edges of the 3D mesh are provided as input to the trained machine learning model. as well as The skeletal hierarchy for the virtual character is constructed based on the predicted joint positions.

4. The computer-implemented method of claim 3, further comprising, prior to constructing the skeletal hierarchy, performing at least one operation from the group consisting of: removing one or more vertices located outside the bounding box of the 3D mesh; removing one or more isolated vertices; mirroring the vertices across a plane of symmetry; and any combination thereof.

5. The computer-implemented method according to claim 3, further comprising: The predicted joint positions are clustered to generate the skeleton.

6. The computer-implemented method according to claim 3, wherein constructing the skeletal hierarchy comprises: Determine whether the predicted joint positions of the plurality of joints are valid; If the predicted joint positions are determined to be valid, the plurality of joints are connected to bones to determine the bone hierarchy; as well as If the predicted joint position is determined to be invalid, the bone level is determined based on either a procedurally generated geometric bone or a default bone.

7. The computer-implemented method of claim 1, wherein the skinning weights are determined by the following operations: Map the 3D mesh to a simplified 3D mesh, wherein two or more vertices of the 3D mesh correspond to a single vertex of the simplified 3D mesh; The trained second machine learning model is applied to the simplified 3D mesh to determine the intermediate skin weights for the simplified 3D mesh, wherein the corresponding volumetric geodesic distance between each vertex and bone pair is provided as input to the trained second machine learning model. as well as The skin weights for each vertex of the 3D mesh are calculated based on the intermediate skin weights and the mapping.

8. The computer-implemented method of claim 1, wherein dividing the skinned 3D mesh into the plurality of body part meshes comprises: The vertices of the skinned 3D mesh associated with each body part are determined based on the skinning weights.

9. The computer-implemented method of claim 1, wherein determining the cage mesh comprises, for each body part: A trained third machine learning model is applied to the body part mesh to determine a point cloud, the point cloud including multiple cage vertices and canonical connectivity between the multiple cage vertices; Multiple predicted cage mesh components are formed based on the plurality of vertices and the canonical connectivity of each vertex; as well as The cage mesh is determined by deforming the vertices of the predicted cage mesh component located inside the surface of the body mesh.

10. A non-transitory computer-readable medium having instructions stored thereon, the instructions being responsive to execution by a processing device to cause the processing device to perform an operation, the operation comprising: Obtain a three-dimensional (3D) mesh of a virtual character, wherein the 3D mesh comprises a plurality of vertices having corresponding positions in 3D space, wherein one or more connections exist between the vertices; Based on the 3D mesh, an avatar body model for the virtual character is generated, wherein the avatar body model includes a skinned 3D mesh, multiple body part meshes, and cage meshes corresponding to the respective body part meshes among the multiple body part meshes, wherein the generation includes: The skeleton for the virtual character is generated based on the 3D mesh, wherein the skeleton includes multiple joints connected by corresponding bones, and each of the multiple joints is a predefined joint based on the body type of the virtual character. Each of the plurality of joints of the bone is associated with at least one corresponding vertex of the plurality of vertices of the 3D mesh to obtain the skinned 3D mesh, wherein a corresponding skinning weight is assigned to each of the plurality of vertices; Based on the skinning weights associated with the plurality of vertices, the skinned 3D mesh is divided into a plurality of body part meshes corresponding to the corresponding body parts of the virtual character; and Determine the cage mesh for each body part mesh; and The avatar body model is animated to depict the movement of the virtual character in the virtual environment.

11. The non-transitory computer-readable medium of claim 10, wherein the operation further comprises: Receive the 3D mesh of the virtual character; as well as Before generating the avatar body model, the 3D mesh is simplified to reduce the vertex count of the 3D mesh, wherein the simplification includes one or more of filtering, cleaning, or mirroring.

12. The non-transitory computer-readable medium of claim 10, wherein generating the skeleton comprises: A trained machine learning model is applied to the 3D mesh to obtain predicted joint positions of the plurality of joints of the skeleton, wherein vertex localization, topological edges, and geodesic edges of the 3D mesh are provided as input to the trained machine learning model. as well as The skeletal hierarchy for the virtual character is constructed based on the predicted joint positions.

13. The non-transitory computer-readable medium of claim 12, wherein the operation further comprises, prior to constructing the skeletal hierarchy, performing at least one operation from the group consisting of: removing one or more vertices located outside the bounding box of the 3D mesh; removing one or more isolated vertices; mirroring the vertices across a line of symmetry; and any combination thereof.

14. The non-transitory computer-readable medium of claim 12, wherein constructing the skeletal hierarchy comprises: Determine whether the predicted joint positions of the plurality of joints are valid; If the predicted joint positions are determined to be valid, the plurality of joints are connected to bones to determine the bone hierarchy; as well as If the predicted joint position is determined to be invalid, the bone level is determined based on either a procedurally generated geometric bone or a default bone.

15. The non-transitory computer-readable medium of claim 10, wherein the skinning weights are determined by: Map the 3D mesh to a simplified 3D mesh, wherein two or more vertices of the 3D mesh correspond to a single vertex of the simplified 3D mesh; The trained second machine learning model is applied to the simplified 3D mesh to determine the intermediate skin weights for the simplified 3D mesh, wherein the corresponding volumetric geodesic distance between each vertex and bone pair is provided as input to the trained second machine learning model. as well as The skin weights for each vertex of the 3D mesh are calculated based on the intermediate skin weights and the mapping.

16. A system comprising: A memory, wherein instructions are stored thereon; and A processing device coupled to the memory, the processing device being configured to access the memory and execute the instructions, wherein the instructions cause the processing device to perform operations, the operations including: Obtain a three-dimensional (3D) mesh of a virtual character, wherein the 3D mesh comprises a plurality of vertices having corresponding positions in 3D space, wherein one or more connections exist between the vertices; Based on the 3D mesh, an avatar body model for the virtual character is generated, wherein the avatar body model includes a skinned 3D mesh, multiple body part meshes, and cage meshes corresponding to the respective body part meshes among the multiple body part meshes, wherein the generation includes: The skeleton for the virtual character is generated based on the 3D mesh, wherein the skeleton includes multiple joints connected by corresponding bones, and each of the multiple joints is a predefined joint based on the body type of the virtual character. Each of the plurality of joints of the bone is associated with at least one corresponding vertex of the plurality of vertices of the 3D mesh to obtain the skinned 3D mesh, wherein a corresponding skinning weight is assigned to each of the plurality of vertices; Based on the skinning weights associated with the plurality of vertices, the skinned 3D mesh is divided into a plurality of body part meshes corresponding to the corresponding body parts of the virtual character; and Determine the cage mesh for each body part mesh; and The avatar body model is animated to depict the movement of the virtual character in the virtual environment.

17. The system of claim 16, wherein generating the skeleton comprises: A trained machine learning model is applied to the 3D mesh to obtain predicted joint positions of the plurality of joints of the skeleton, wherein vertex localization, topological edges, and geodesic edges of the 3D mesh are provided as input to the trained machine learning model. as well as The skeletal hierarchy for the virtual character is constructed based on the predicted joint positions.

18. The system of claim 16, wherein the skin weights are determined by the following operation: Map the 3D mesh to a simplified 3D mesh, wherein two or more vertices of the 3D mesh correspond to a single vertex of the simplified 3D mesh; The trained second machine learning model is applied to the simplified 3D mesh to determine the intermediate skin weights for the simplified 3D mesh, wherein the corresponding volumetric geodesic distance between each vertex and bone pair is provided as input to the trained second machine learning model. as well as The skin weights for each vertex of the 3D mesh are calculated based on the intermediate skin weights and the mapping.

19. The system of claim 16, wherein dividing the skinned 3D mesh into the plurality of body part meshes comprises: The vertices of the skinned 3D mesh associated with each body part are determined based on the skinning weights.

20. The system of claim 16, wherein determining the cage mesh comprises: A trained third machine learning model is applied to the 3D mesh to determine a point cloud, which includes multiple cage vertices and canonical connectivity between the multiple cage vertices; Multiple predicted cage mesh components are formed based on the multiple vertices and the corresponding canonical connectivity of each vertex; as well as The cage mesh is determined by deforming the vertices of the predicted cage mesh component located inside the surface of the body mesh.

Images