Dynamically Changing Avatar Bodies in Virtual Experiences

Abstract

Some implementations relate to methods, systems, and computer-readable media for dynamically modifying an avatar at runtime while an avatar associated with the avatar body participates in a virtual experience. In some implementations, the method includes identifying a first avatar body having a first body cage, identifying a target avatar body having a target body cage, and performing interpolation between the first body cage and the target body cage to obtain a second body cage corresponding to the second avatar body to provide a transformation of the first avatar body to the second avatar body. The avatar body can also be modified in a setting environment. Modifying the avatar body can involve interpolating between the body cages of a pair of cages or directly manipulating the body cage of the avatar body.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 532,556, filed August 14, 2023, entitled "DYNAMICALLY CHANGING AVATAR BODIES IN A VIRTUAL EXPERIENCE," the contents of which are incorporated herein in their entirety.

[0002] The present disclosure relates generally to computer graphics, and more particularly, but not exclusively, to methods, systems, and computer-readable media for dynamically changing avatar bodies (including representations of clothing worn by the avatar bodies) in a three-dimensional (3D) virtual environment. [Background technology]

[0003] Multi-user electronic games or other types of virtual experience environments may involve the use of avatars to represent users in the virtual experience. Different three-dimensional (3D) avatars vary in geometry / shape from one avatar to another. For example, avatars may have different body shapes (e.g., tall, short, muscular, skinny, etc.), may be of different types (e.g., male, female, human, animal, alien, etc.), may have any number or types of limbs, etc. Avatars may be customizable with respect to multiple clothing and / or accessories worn by the avatar (e.g., a shirt worn over the torso, a jacket worn over the shirt, a scarf worn over the jacket, a hat worn on the head, etc.).

[0004] When users participate in a virtual experience or other type of 3D environment, if they wish to change some visual aspect of the body of their respective avatar and / or the clothing (including accessories) worn by the avatar body, it can be difficult to achieve satisfactory results in a computationally efficient manner.

[0005] Based on the above, several implementations have been devised.

[0006] The background discussion provided herein is for the purpose of providing a context for the present disclosure. To the extent described in this background section, the work of the presently named inventors, as well as aspects of the body of the specification that may not qualify as prior art at the time of filing, are not admitted expressly or impliedly as prior art to the prior disclosure. Summary of the Invention [Means for solving the problem]

[0007] Implementations of the present disclosure relate to techniques for dynamically changing the visual aspects of a user's avatar (e.g., the visual aspects of an avatar associated with a user as the user participates in a virtual experience). The entire (original) avatar body can be changed or otherwise transformed into a new (different) avatar body as a whole, or only regions / portions of the original avatar body (e.g., only the head or other body parts) can be selectively changed, leaving other regions / portions of the avatar body unchanged. Various techniques also provide specific ways to achieve aspects of dynamic change, such as skin morphing and facial action coding system (FACS) pose morphing, in near real time by using techniques that efficiently manage computational resources.

[0008] One or more computer systems may be configured to perform particular operations or actions by software, firmware, hardware, or a combination thereof installed on the systems that causes the systems to perform actions during operation. One or more computer programs are configured to perform particular operations or actions by containing instructions that, when executed by a data processing device, cause the device to perform the actions.

[0009] According to one aspect, a computer-implemented method for modifying a three-dimensional (3D) avatar body is provided, the computer-implemented method including steps of identifying a first avatar body having a first body cage, identifying a target avatar body having a target body cage, and performing interpolation between the first body cage and the target body cage to obtain a second body cage corresponding to the second avatar body to provide a transformation of the first avatar body to the second avatar body.

[0010] Various implementations of computer-implemented methods are described herein.

[0011] In some implementations, performing the interpolation includes performing the interpolation to generate a second body cage that perfectly matches the target body cage to provide a perfect transformation.

[0012] In some implementations, performing the interpolation includes transforming the first avatar body into a second avatar body that is a mixture between the first avatar body and the target avatar body to provide a partial transformation.

[0013] In some implementations, performing the interpolation includes morphing a portion of the first avatar body that is less than the entire first avatar body.

[0014] In some implementations, morphing a portion of the first avatar body that is less than the entire first avatar body includes morphing a portion of the first avatar body to perform a partial transformation of the portion of the first avatar body.

[0015] In some implementations, the first avatar body is part of a virtual experience, the step of performing interpolation is performed while the avatar is participating in the virtual experience, and the target avatar body is selected from a plurality of target avatar bodies in the virtual experience.

[0016] In some implementations, the step of performing interpolation is performed in a configured environment, and the target avatar body is selected from a plurality of target avatar bodies in a library within the configured environment.

[0017] In some implementations, the setting environment includes a transformation tool that allows a user to control the amount of transformation of a first avatar body to obtain a second avatar body, and the step of performing interpolation is based on the amount of transformation.

[0018] In some implementations, the computer-implemented method further includes identifying a rig for the first avatar body, the rig including identifying a skeleton of the first avatar body and skinning of the first avatar body; after the step of performing interpolation, updating the rig of the first avatar body to correspond to the second body cage; and animating the first avatar body by moving the skeleton of the updated rig and deforming the skinning of the updated rig.

[0019] In some implementations, moving the skeleton of the updated rig and transforming the skinning of the updated rig includes reusing skin weights from the skinning of the first avatar body based on determining areas of the skinning of the updated rig that are influenced by bones in the skeleton of the first avatar body.

[0020] According to another aspect, a computer-implemented method for modifying a three-dimensional (3D) avatar body is provided, the computer-implemented method including steps of identifying a first avatar body having a corresponding first body cage, and performing a manipulation of the first body cage to generate a second body cage corresponding to the second avatar body to provide a transformation of the first avatar body into the second avatar body, wherein the manipulating includes rearranging portions of the first body cage.

[0021] Various implementations of computer-implemented methods are described herein.

[0022] In some implementations, the step of performing the operation is performed in a configuration environment, the configuration environment includes a transformation tool that allows a user to control aspects of the operation of the first body cage to obtain the second body cage, and the step of performing the operation is based on the aspects of the operation.

[0023] In some implementations, the computer-implemented method further includes identifying a rig for the first avatar body, the rig including identifying a skeleton of the first avatar body and skinning of the first avatar body, and after performing the operation, updating the rig of the first avatar body to correspond to the second body cage, and animating the first avatar body by moving the skeleton of the updated rig and deforming the skinning of the updated rig.

[0024] In some implementations, the computer-implemented method further includes moving a skeleton of the updated rig and deforming skinning of the updated rig, which includes reusing skin weights from the skinning of the first avatar body based on determining areas of the skinning of the updated rig that are influenced by bones in the skeleton of the first avatar body.

[0025] In some implementations, transforming the first avatar body into the second avatar body includes performing an interpolation between the first body cage and the second body cage.

[0026] In some implementations, transforming the first avatar body into the second avatar body includes morphing a portion of the first avatar body that is less than the entire first avatar body.

[0027] According to another aspect, a system is disclosed comprising a memory having instructions stored thereon and a processing device coupled to the memory, the processing device configured to access the memory, the instructions, when executed by the processing device, causing the processing device to perform operations including identifying a first avatar body having a first body cage, identifying a target avatar body having a target body cage, and performing interpolation between the first body cage and the target body cage to obtain a second body cage corresponding to the second avatar body to provide a transformation of the first avatar body into the second avatar body.

[0028] Various implementations of the system are described herein.

[0029] In some implementations, performing the interpolation includes performing the interpolation to generate a second body cage that perfectly matches the target body cage to provide a perfect transformation.

[0030] In some implementations, performing the interpolation includes transforming the first avatar body into a second avatar body that is a mixture between the first avatar body and the target avatar body to provide a partial transformation.

[0031] In some implementations, performing the interpolation includes morphing a portion of the first avatar body that is less than the entire first avatar body.

[0032] According to yet other aspects, the portions, features, and implementation details of the systems, methods, and non-transitory computer-readable media may be combined to form additional aspects that omit and / or modify some or portions of the individual components or features, include additional components or features, and / or other modifications, and all such modifications are within the scope of the present disclosure. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 illustrates an exemplary system architecture including a 3D environment platform capable of supporting clothed 3D avatars, according to some implementations. [Figure 2] 1 illustrates an exemplary body cage according to some implementations. [Figure 3] FIG. 1 illustrates another exemplary body cage according to some implementations. [Figure 4] 10A-10C illustrate examples of portions of a body cage grouped into corresponding body parts, according to some implementations. [Figure 5] 10A-10C illustrate examples of clothing layers deformed on a body cage, according to some implementations. [Figure 6] 6A-6C illustrate examples of outer cages formed based on the clothing layers and body cage portions of FIG. 5, according to some implementations. [Figure 7] A diagram illustrating an example of interpolation between two body cages to obtain a new body cage, according to some implementations. [Figure 8] 10A-10C illustrate examples of generating a new body cage according to some implementations. [Figure 9] 10A-10C illustrate examples of avatar body transformations during a virtual experience, according to some implementations. [Figure 10] A diagram showing another example of avatar body transformation during a virtual experience, according to some implementations. [Figure 11] A diagram showing another example of avatar body transformation during a virtual experience, according to some implementations. [Figure 12] 1A-1C illustrate examples of transformations of an avatar in a setting, according to some implementations. [Figure 13] A diagram illustrating an example of layering clothing on an avatar body in a virtual experience, according to some implementations. [Figure 14] 1A-1C illustrate examples of avatar transformations and animations, according to some implementations. [Figure 15] 1A-1C illustrate examples of avatar transformations and animations, according to some implementations. [Figure 16] 1A-1C illustrate examples of avatar transformations and animations, according to some implementations. [Figure 17] 1A-1C illustrate examples of avatar transformations and animations, according to some implementations. [Figure 18] 1 is a flowchart illustrating a computer-implemented method for modifying an avatar body, according to some implementations. [Figure 19] 1 is a flowchart illustrating another computer-implemented method for modifying a three-dimensional (3D) avatar body, according to some implementations. [Figure 20] 1 is a flowchart illustrating a computer-implemented method for performing skin morphing, according to some implementations. [Figure 21] 1 is a flowchart illustrating a computer-implemented method for performing Facial Action Coding System (FACS) pose morphing. [Figure 22] FIG. 1 is a block diagram illustrating an exemplary computing device according to some implementations. DETAILED DESCRIPTION OF THE INVENTION

[0034] In the following detailed description, reference is made to the accompanying drawings, which form a part of this specification. In the drawings, like symbols typically identify like components unless the context dictates otherwise. The exemplary implementations described in the detailed description, drawings, and claims are not meant to be limiting. Other implementations may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. The aspects of the present disclosure, as generally described herein and illustrated in the figures, can be arranged, substituted, combined, and separated in a wide variety of different configurations, all of which are contemplated herein.

[0035] References herein to "one implementation," "implementation," "exemplary implementation," etc. indicate that the described implementation may include a particular feature, structure, or characteristic, but not all implementations necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same implementation. Furthermore, when a particular feature, structure, or characteristic is described in connection with an implementation, such feature, structure, or characteristic may also be provided in connection with other implementations, whether or not explicitly described.

[0036] This disclosure describes techniques for dynamically changing the visual aspects of a user's avatar (e.g., the visual aspects of an avatar associated with a user as the user participates in a virtual experience). For example, while participating in a virtual experience, a user may change from a humanoid avatar to an animal-like (non-humanoid) avatar body or other different avatar body. The entire (original) avatar body may be changed or otherwise transformed into a new (different) avatar body as a whole, or only regions / portions of the original avatar body (e.g., only the head or other body part) may be selectively changed, leaving other regions / portions of the avatar body unchanged.

[0037] The transformation from the original avatar body to the target avatar body can be a complete transformation, in that the original (first) avatar body is completely transformed into the target avatar body, which becomes the new (second) avatar body. Alternatively, the transformation can be a partial transformation, in that the new (second) avatar body is a composite or other type of blend intermediate between the original (first) avatar body and the target avatar body.

[0038] According to various implementations, the new avatar body is obtained by performing an interpolation between the body cage of the original (first) avatar body and the body cage of the target avatar body, such that the resulting new avatar body has its own body cage interpolated / generated from the body cages of the original avatar body and the target avatar body.

[0039] Dynamic editing / modification of the avatar body may be performed when the avatar is not wearing clothing and / or when the avatar body is wearing one or more layers of clothing. If the avatar body is wearing clothing, dynamically modifying the avatar body (e.g., changing its shape) also results in corresponding changes to the layers of clothing worn by the avatar.

[0040] For example, the original avatar may be a humanoid avatar wearing a baseball cap on its head, thereby giving the baseball cap a rounded appearance. If the user changes the humanoid avatar's head to an alien head (such as an alien avatar with a cone-shaped head), the baseball cap may also be dynamically transformed accordingly, changing from its original rounded appearance to a more pointed appearance that matches the cone-shaped head of the alien (target) avatar. Dynamically transforming the avatar's clothing (including accessories) may also be cage-based, as described below.

[0041] According to various implementations, the target avatar body and its clothing can be selected by the user at run time during the virtual experience. For example, the user can select the target avatar body by selecting (e.g., clicking) another avatar within the virtual experience, selecting the target avatar body from a library, directly manipulating the original (current) avatar body (e.g., changing its cage) without having to select a target avatar body within the virtual experience or from a library, etc. An adjustment tool, such as a slide bar, can be provided in the user interface to allow the user to control the amount of transformation between the two avatar bodies.

[0042] In some implementations, adjustment tools and / or some other transformation tools may be used to dynamically modify / transform the current avatar body in a direct manner, without necessarily interpolating between two avatar bodies. That is, as an example, a user may use a transformation tool to change the shape of an avatar's head from a humanoid head to a geometric head (e.g., block-shaped), which in some implementations may be performed by directly modifying (e.g., moving or otherwise manipulating) the segments and vertices of the cage of the humanoid head using the transformation tool. This technique can be utilized without having to present or use a body cage for the geometric head as a reference (target) and without having to perform interpolation between such a reference body cage and the body cage of the humanoid head. Thus, this technique may be considered a "free-form" approach to independently modifying the appearance of an avatar, where the appearance change is not related to any other avatar.

[0043] The various techniques described herein for dynamically modifying avatar bodies (with or without clothing) may be applied to avatars used in virtual experiences, which may be described in the context of electronic games. It is understood that describing such implementations in the context of electronic games is for convenience in providing examples and illustrations.

[0044] The techniques described herein can be used for other types of virtual experiences in three-dimensional (3D) environments that may not necessarily involve electronic games with one or more players represented by avatars. Examples of virtual experiences include virtual reality (VR) conferences, 3D sessions (e.g., online lectures or other types of presentations involving 3D avatars), or other types of 3D environments in which one or more users are represented in the 3D environment by one or more 3D avatars.

[0045] Layering may employ automated cage-to-cage fitting techniques for 3D avatars that allow any body geometry to be matched to any clothing geometry, including allowing a layer of clothing to be fitted over an underlying layer of clothing, thereby providing customization without restrictions imposed by predefined geometries or requiring complex calculations to make clothing items compatible with any body shape of the avatar or other clothing items.

[0046] Inter-cage fitting is also performed using various techniques employed by the gaming platform or software (or other virtual experience platform / software that operates to provide a 3D environment) without the need for the avatar creator (also referred to as an avatar body creator or body creator) or clothing item creator to perform complex calculations. As used herein, the terms "clothing" or "piece of clothing" or other similar terms are understood to include graphical representations of clothing and accessories, as well as any other items that can be placed on an avatar in relation to a particular portion of the avatar cage.

[0047] At run-time during a game or other virtual experience session, a player / user accesses a body library to select a particular avatar body and a clothing library to select clothing to place on the selected body. The 3D virtual environment platform presenting the avatar implements cage-to-cage fitting techniques to adjust the clothing (by automatically determining appropriate deformations) to fit the shape of the body (and any intermediate layers, if worn by the avatar).

[0048] When clothing is worn over the body and / or underlying clothing, the techniques described herein may be implemented to more accurately deform or otherwise fit the clothing to the avatar in terms of scale (e.g., proportions), shape, etc. The user may further select additional clothing to wear over the underlying clothing, where the additional clothing is deformed to match the geometry of the underlying clothing.

[0049] The implementations described herein are based on the concepts of cages and meshes. A body mesh (or render mesh) is the actual visible geometric shape of an avatar. A body mesh contains a graphical representation of body parts such as arms, legs, torso, and head, and can be of any shape, size, and geometric topology. Similarly, a clothing mesh (or render mesh) can be any mesh that graphically represents clothing or parts of clothing such as a shirt, pants, hat, or shoes.

[0050] In comparison, a cage represents an envelope of feature points around an avatar body, which envelope is simpler than the body mesh and has a weak correspondence with the corresponding vertices of the body mesh. As will be explained in more detail below, a cage can be used to represent not only a set of feature points on an avatar body, but also a set of feature points on clothing.

[0051] In some implementations, there is a dynamic body part modification mechanism implemented by extending the layering framework of the layering system to enable the use of user-specified cages to deform avatar body parts. This core functionality can be applied in at least two techniques. First, there can be a plug-in to the studio application that allows a user to change the entire body shape of an avatar based on the shape of another avatar, such as by dialing / adjusting a slider that interpolates between two body shapes. Second, there can be a virtual experience in which a player can incrementally update the avatar body by clicking other avatar body parts within the virtual experience.

[0052] Some existing techniques for dynamically modifying avatars may include Linear Blend Skinning (LBS), Facial Action Coding System (FACS), and Affine Skinning techniques, which serve as the foundation for current approaches including skin morphing and FACS pose morphing.

[0053] In Linear Blend Skinning (LBS), the i-th deformation position p' i in the vertex shader

[0054]

number

[0055] In this formula, the skin weight w i,j and the vertex bind position p i is a constant that is cached on the GPU. If less than four bones are used, w i,j can be 0. 3x4 bone transformation M j is calculated on the CPU and copied to the GPU for each frame.

[0056]

number

[0057] Inverse Binding Transformation

[0058]

number

[0059] is a cached constant, but the pose transformation P j is updated every frame. Global transformation B j and P j is a local transformation called a skeleton LB j and LP j is calculated from the hierarchy of B j =B j' ·LB, P j =P j' LB j ·LP j where j' is the index of the parent of the jth node, i.e. the root part. The skeleton hierarchy and transformations are provided by other engine systems, such as physics for bodies or FACS for dynamic heads. These are the various sources providing the animations that can be used to pose the avatar.

[0060] The FACS system provides the LBS data for the dynamic head mesh: skin weights w, skeleton hierarchy j', and local binding transformation LB j is derived from a content delivery network (CDN) mesh data structure that contains meshes and controls-to-joint-driver data, which is a mapping from FACS control values ​​to joint positions and rotations for a given mesh. The local pose transformation LP j is a 3x3 rotation matrix R j,k and the translation vector T j,k It is composed frame by frame from LP j =R j,k |T j,k where:

[0061]

number

[0062] and

[0063]

number

[0064] The Euler angle r j,k and the translation vector t j,k The mesh data structure ControlToJointDriver also derives from the mesh data structure, which defines the mapping of the mesh that converts FACS control values ​​into the positions and rotations of the skinned joints. The data structure can be an MxN matrix, where M is the number of FACS data channels and N is the number of joint transformation values. The matrix is ​​used to convert FACS controls into transformation values ​​that drive the facial skeleton joints.

[0065] The rotations are interpolated in Euler coordinates, so R j,k is a rigid body. 17≦n≦50 shape weights s' are computed as s'=applyCorrectives(s,C), where the applyCorrectives function expands the 17 raw FACS pose weights s into 50 shape weights s', also based on the CustomCorrections parameter C from the ControlToJointDriver structure. For selected combinations of two or three raw FACS weights, the default linear average shape can be replaced with a custom shape constructed for that combination. This is used to improve artist control and deformation quality.

[0066] Previously, the bind and pose transformations were of rigid CoordinateFrame type. A rigid transformation (also called a Euclidean transformation) is a geometric transformation in Euclidean space that preserves the Euclidean distance between all pairs of points. Previously, all Data Model (DM) and FACS transforms were rigid, so that was sufficient. Knowing this, we can transpose rotations when calculating their inverse. This can be a significant optimization, for example, during physics updates, when transform inversion is a hotspot. However, for LBS, the inversion

[0067]

number

[0068] Since only ^ is cached as a constant, the present technique does not benefit greatly from this optimization.

[0069] Approximating elastic skin is a common use case for LBS, but skin does not deform rigidly. Elastic deformations are usually approximated with rigid bone transformations, which is natural when deformations are driven by a rigid skeleton. However, facial muscles are not rigid. Facial muscles can shear and scale non-uniformly, so there is no physical reason to limit the facial "bones" to rigid transformations. Because affine transformations are a superset of rigid transformations, they can be used without modification in existing rigs and also add flexibility that can be leveraged in future rigs.

[0070] The dynamic head weight map divides the face into regions and provides the degrees of freedom necessary to achieve 17 to 50 poses. The pose transformations clearly represent those regions in response to 17 FACS controls. The bone transformations may be modified if the shape of the dynamic head changes and it is desirable for these poses to continue looking good. Skin weights can often be reused if the shape change does not change the semantic meaning of the vertices, i.e., cheek vertices no longer become nose vertices.

[0071] When skin weights are reused, they can represent which areas of the surface are influenced by each bone. One can also notice that pose transformations animate in a direction and amount relative to the surface shape in the areas the pose influences. This influence includes not only its own weight, but also the weights of any lower bones in the skeletal hierarchy. For example, when the neck rotates, the lip joint rotates with the neck, so the influence of the head joint includes the influence of the lip joint, etc. Also, since vertices influence the shape of the connected faces, these influence sets are extended by one edge. Unlike LBS, these influence weight sets are not normalized.

[0072] Skin morphing may involve using some linear algebra to calculate, for each bone, a 3x4 affine transformation that best fits the deformation of the point cloud that it affects. This correction is applied to the bone transformation in the skinning calculation,

[0073]

number

[0074] where the correction is made by the matrix M j This approach may work, but the technique is not designed to change the skinning pipeline, it is designed to change the rig. j is bubbled up through the skeleton computation, D j produces the same result as the new local binding bias LB' j Incorporated into LB' j =B j -1 D j ·B j' LB j For example, this produces the same result as the previous formulation, but with M j New local binding transformation LB' with deleted term j It can be a simple algebraic manipulation to solve

[0075] LB' j Note that after this, is no longer a rigid transformation, so the result can be orthonormalized to any data model bone, but the full affine transformation is retained for the dynamic head, greatly improving the results. This makes the result dependent on the affine skinning modifications discussed above.

[0076] Regarding FACS pose morphing, the current FACA rig contains many poses, and each pose is a local transformation P for each joint in the head rig. j The skin morphing technique involves a set of binding transformations LB' j , which changes the parent space of these local pose transforms.

[0077] Affine correction transformation D j is the resulting M j The affine correction transformation D j is the resulting M j Do not change the rotation of the transform. To further improve the results, use per-pose, per-joint rotations. j,k In the process, the per-pose, per-joint translation t j,k Further fine-tuning of the

[0078] In some implementations, fine-tuning can be performed one pose at a time. First, the original LBS-deformed mesh pose is calculated from the original head shape. For each joint in the pose, its skin load point is projected to the nearest point on the posed mesh, providing a new set of points. These points represent the locations on the original mesh that are closest to that joint's "destination" in that pose. The same 3x4 affine fit function can then be reused to calculate how these points transformed between the original and modified shapes.

[0079] In some implementations, the rigid body component of that transformation is extracted to fine-tune the final translation and rotation of the pose joints. Appropriate modifications are made to the Euler angles that fit that rotation to minimize changes to the Euler interpolation. For example, since Euler rotations can be expressed in an infinite number of ways with the same rotation matrix, appropriate techniques are used to modify the original angles as little as possible. These angles are calculated by decomposing the rotation matrix and incrementally inserting correction values.

[0080]

number

[0081] , then

[0082]

number

[0083] , then x'=x1+x2, y'=y1+y2, and z'=z1+z2.

[0084] Figure 1 - System Architecture 1 is a diagram of an exemplary system architecture including a 3D environment platform capable of supporting clothed 3D avatars, according to some implementations. FIG. 1 and other figures use similar reference numbers to indicate similar identifications. A letter following a reference number, such as "110," indicates that the text specifically refers to the element with that particular reference number. A reference number in the text without a following letter, such as "110," refers to any or all of the elements in the figure bearing that reference number (e.g., "110" in the text refers to reference numbers "110a," "110b," and / or "110n" in the figures).

[0085] System architecture 100 (also referred to herein as the “system”) includes an online virtual experience server 102, a data store 120, client devices 110a, 110b, and 110n (collectively referred to herein as “client devices 110”), and developer devices 130a and 130n (collectively referred to herein as “developer devices 130”). Virtual experience server 102, data server 120, client devices 110, and developer devices 130 are coupled via a network 122. In some implementations, client device 110 and developer device 130 may refer to the same device or the same type of device.

[0086] The online virtual experience server 102 may include, among other things, a virtual experience engine 104, one or more virtual experiences 106, and a graphics engine 108. In some implementations, the graphics engine 108 may be a system, application, or module that enables the online virtual experience server 102 to provide graphics and animation capabilities. In some implementations, the graphics engine 108 and / or the virtual experience engine 104 may perform one or more of the operations described below in connection with the flowcharts shown in Figures 18-21. The client device 110 may include a virtual experience application 112 and an input / output (I / O) interface 114 (e.g., input / output devices). The input / output devices may include one or more of a microphone, speakers, headphones, a display device, a mouse, a keyboard, a game controller, a touchscreen, a virtual reality console, etc.

[0087] The developer device 130 may include a virtual experience application 132 and an input / output (I / O) interface 134 (e.g., input / output devices). The input / output devices may include one or more of a microphone, speakers, headphones, a display device, a mouse, a keyboard, a game controller, a touchscreen, a virtual reality console, etc.

[0088] System architecture 100 is provided for illustrative purposes. In different implementations, system architecture 100 may include the same, fewer, more, or different elements, arranged in the same or different ways as shown in FIG.

[0089] In some implementations, the network 122 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network, a Wi-Fi network, or a wireless LAN (WLAN)), a cellular network (e.g., a 5G network, a Long Term Evolution (LTE) network, etc.), a router, a hub, a switch, a server computer, or a combination thereof.

[0090] In some implementations, data store 120 may be non-transitory computer-readable memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data. Data store 120 may also include multiple storage components (e.g., multiple drives or multiple databases), possibly spanning multiple computing devices (e.g., multiple server computers). In some implementations, data store 120 may include cloud-based storage.

[0091] In some implementations, the online virtual experience server 102 can include a server having one or more computing devices (e.g., a cloud computing system, a rack-mounted server, a server computer, a cluster of physical servers, etc.) In some implementations, the online virtual experience server 102 can be a standalone system, can include multiple servers, or can be part of another system or server.

[0092] In some implementations, the online virtual experience server 102 may include one or more computing devices (such as a rack-mounted server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memory, databases), networks, software components, and / or hardware components that may be used to execute operations on the online virtual experience server 102 and provide users with access to the online virtual experience server 102. The online virtual experience server 102 may also include websites (e.g., web pages) or application backend software that may be used to provide users with access to content provided by the online virtual experience server 102. For example, a user may access the online virtual experience server 102 using a virtual experience application 112 on a client device 110.

[0093] In some implementations, virtual experience session data is generated via online virtual experience server 102, virtual experience application 112, and / or virtual experience application 132 and stored in data store 120. With permission from the virtual experience participants, the virtual experience session data may include associated metadata, such as a virtual experience identifier, device data associated with the participant, participant demographic information, a virtual experience session identifier, chat transcripts, session start time, session end time, and session duration for each participant, relative positions of participant avatars within the virtual experience environment, purchases made within the virtual experience by one or more participants, accessories utilized by the participants, etc.

[0094] In some implementations, the online virtual experience server 102 may be a type of social network that provides connections between users, or a type of user-generated content system that allows users (e.g., end users or consumers) to communicate with other users on the online virtual experience server 102, where the communications may include voice chat (e.g., synchronous and / or asynchronous voice communication), video chat (e.g., synchronous and / or asynchronous video communication), or text chat (e.g., 1:1 and / or N:N synchronous and / or asynchronous text-based communication). Records of some or all communications may be stored in the data store 120 or within the virtual experience 106. The data store 120 may be utilized to store transcripts (e.g., text, audio, images, etc.) of chat exchanged between participants, with appropriate permission from players and in compliance with applicable regulations.

[0095] In some implementations, chat transcripts are generated by virtual experience application 112 and / or virtual experience application 132 and / or stored in data store 120. Chat transcripts may include chat content and associated metadata, such as the text content of the chat with each message having a corresponding sender and recipient, message formatting (e.g., bold, italic, loud, etc.), message timestamps, relative positions of participant avatars within the virtual experience environment, accessories utilized by virtual experience participants, etc. In some implementations, chat transcripts may include multilingual content, and messages in different languages ​​from different sessions of the virtual experience may be stored in data store 120.

[0096] In some implementations, chat transcripts may be stored in the form of conversations between participants based on timestamps. In some implementations, chat transcripts may be stored based on the originator of the message.

[0097] In some implementations of the present disclosure, a "user" may be represented as a single individual. Other implementations of the present disclosure encompass a "user" (e.g., a creating user) being a set of users or an entity controlled by an automated source. For example, a set of individual users federated as a community or group in a user-generated content system may be considered a "user."

[0098] In some implementations, the online virtual experience server 102 may be a virtual game server. For example, the game server may provide single-player or multiplayer games to a community of users, which users may access herein as the “system,” including the online virtual experience server 102, the data store 120, and clients, or may interact with the virtual experience using client devices 110 via the network 122. In some implementations, the virtual experience (including a virtual realm or world, a virtual game, or other computer-simulated environment) may be, for example, a two-dimensional (2D) virtual experience, a three-dimensional (3D) virtual experience (e.g., a 3D user-generated virtual experience), a virtual reality (VR) experience, or an augmented reality (AR) experience. In some implementations, users may participate in interactions (such as gameplay) with other users. In some implementations, the virtual experience may be experienced in real time with other users of the virtual experience.

[0099] In some implementations, virtual experience engagement may refer to the interaction of one or more participants using a client device (e.g., 110) within a virtual experience (e.g., 106), or the presentation of the interaction on a display or other output device (e.g., 114) of the client device 110. For example, virtual experience engagement may include the interaction of one or more participants within a virtual experience, and the presentation of the interaction on a display of the client device.

[0100] In some implementations, the virtual experience 106 may include electronic files that can be executed or loaded using software, firmware, or hardware configured to present virtual experience content (e.g., digital media items) to an entity. In some implementations, a virtual experience application 112 may be executed and the virtual experience 106 may be rendered in association with a virtual experience engine 104. In some implementations, the virtual experience 106 may have a common set of rules or a common goal, and the environments of the virtual experience 106 share a common set of rules or a common goal. In some implementations, different virtual experiences may have different rules or goals from one another.

[0101] In some implementations, a virtual experience may have one or more environments (also referred to herein as "virtual experience environments" or "virtual environments"), and multiple environments may be linked. An example of an environment may be a three-dimensional (3D) environment. One or more environments of the virtual environment 106 may be collectively referred to herein as a "world," or a "virtual experience world," or a "game world," or a "virtual world," or a "universe." An example of a world may be the 3D world of the virtual experience 106. For example, a user may build a virtual environment that is linked to another virtual environment created by another user. Characters in a virtual environment may cross virtual boundaries to enter an adjacent virtual environment.

[0102] It should be noted that a 3D environment or world uses graphics that use 3D representations of geometric data that represent the virtual experiential content (or at least presents the virtual experiential content to appear as 3D content, regardless of whether 3D representations of geometric data are used), and a 2D environment or world uses graphics that use 2D representations of geometric data that represent the virtual experiential content.

[0103] In some implementations, an online virtual experience server 102 may host one or more virtual experiences 106 and may allow users to interact with the virtual experiences 106 using virtual experience applications 112 on client devices 110. Users of the online virtual environment server 102 may play, create, interact with, or build the virtual environment 106, communicate with other users, and / or create and build objects (e.g., also referred to herein as “items,” “virtual experience objects,” or “virtual experience items”) of the virtual environment 106.

[0104] For example, when generating a user-generated virtual item, a user may create, among other things, a character, decorations for a character, one or more virtual environments for an interactive virtual experience, or a build structure used within the virtual environment 106. In some implementations, a user may buy, sell, or exchange virtual experience objects, such as in-platform currency (e.g., virtual currency), with other users of the online virtual experience server 102. In some implementations, the online virtual experience server 102 may transmit virtual experience content to a virtual experience application (e.g., 112). In some implementations, virtual experience content (also referred to herein as "content") may refer to any data or software instructions (e.g., virtual experience objects, virtual experiences, user information, videos, images, commands, media items, etc.) associated with the online virtual experience server 102 or the virtual experience application. In some implementations, a virtual experience object (e.g., also referred to herein as an "item" or "object" or "virtual experience item") may refer to an object used, created, shared, or otherwise depicted in a virtual experience 106 on an online virtual experience server 102 or a virtual experience application 112 on a client device 110. For example, a virtual experience object may include parts, models, characters, accessories, tools, weapons, clothing, buildings, vehicles, currency, plants, animals, components of the foregoing (e.g., windows of a building), etc.

[0105] Note that the online game server 102 hosting the game 106 is provided for illustrative purposes. In some implementations, the online virtual experience server 102 may host one or more media items, which may include communication messages from one user to one or more other users. With user permission and explicit user consent, the online virtual experience server 102 may analyze chat transcript data to improve the virtual experience platform. Media items may include, but are not limited to, digital videos, digital movies, digital photos, digital music, audio content, melodies, website content, social media updates, e-books, e-magazines, e-newspapers, digital audiobooks, e-journals, weblogs, real simple syndication (RSS) feeds, e-comic books, software applications, and the like. In some implementations, media items may be electronic files that can be executed or loaded using software, firmware, or hardware configured to present the digital media items to entities.

[0106] In some implementations, a virtual experience 106 may be associated with a particular user or a particular group of users (e.g., a private virtual experience), or may be made widely available to users of the online virtual experience server 102 (e.g., a public virtual experience). In some implementations, when the online virtual experience server 102 associates one or more virtual experiences 106 with a particular user or group of users, the online virtual experience server 102 may use user account information (e.g., a user account identifier such as a username and password) to associate the particular user with the virtual experience 106.

[0107] In some implementations, the online virtual experience server 102 or the client device 110 may include a virtual experience engine 104 or a virtual experience application 112. In some implementations, the virtual experience engine 104 may be used for developing or executing the virtual experience 106. For example, the virtual experience engine 104 may include functionality such as a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, scripting capabilities, an animation engine, an artificial intelligence engine, networking capabilities, streaming capabilities, memory management capabilities, threading capabilities, scene graph capabilities, or video support for cinematics, among other features. Components of the virtual experience engine 104 may generate commands (e.g., rendering commands, collision commands, physics commands, etc.) that help calculate and render the virtual experience. In some implementations, the virtual experience applications 112 of the client devices 110 may each operate independently, in cooperation with the virtual experience engine 104 of the online virtual experience server 102, or a combination of both.

[0108] In some implementations, both the online virtual experience server 102 and the client device 110 may run a virtual experience engine / application (104 and 112, respectively). The online virtual experience server 102, using its virtual experience engine 104, may perform some or all virtual experience engine functions (e.g., generating physics commands, rendering commands, etc.) or may offload some or all virtual experience engine functions to the virtual experience engine 104 of the client device 110. In some implementations, each virtual experience 106 may have a different ratio between the virtual experience engine functions executed on the online virtual experience server 102 and the virtual experience engine functions executed on the client device 110. For example, the virtual experience engine 104 of the online virtual experience server 102 may be used to generate physics commands when a collision exists between at least two virtual experience objects, while additional virtual experience engine functions (e.g., generating rendering commands) may be offloaded to the client device 110. In some implementations, the ratio of virtual experience engine functions executed on the online virtual experience server 102 and the client device 110 may be changed (e.g., dynamically) based on virtual experience engagement conditions. For example, if the number of users engaged in a particular virtual experience exceeds a threshold number, the online virtual experience server 102 may execute one or more virtual experience engine functions previously executed by the client device 110.

[0109] For example, a user may be playing a virtual experience 106 at a client device 110 and may send control instructions (e.g., user input such as right, left, up, down, user selection, or character position and velocity information) to the online virtual experience server 102. After receiving the control instructions from the client device 110, the online virtual experience server 102 may send experience instructions (e.g., position and velocity information of a character participating in a group experience, or commands such as rendering commands, collision commands, etc.) to the client device 110 based on the control instructions. For example, the online virtual experience server 102 may perform one or more logical operations on the control instructions (e.g., using the virtual experience engine 104) to generate experience instructions for the client device 110. In other examples, the online virtual experience server 102 may pass one or more of the control instructions from one client device 110 to other client devices participating in the virtual experience 106 (e.g., from client device 110a to client device 110b). Client device 110 may use the experience instructions to render the virtual experience for presentation on the display of client device 110 .

[0110] In some implementations, control instructions may refer to instructions indicating actions of a user's character within a virtual experience. For example, control instructions may include user inputs for controlling actions within the experience, such as right, left, up, down, user selection, gyroscope position and orientation data, force sensor data, etc. Control instructions may include character position and velocity information. In some implementations, control instructions may be transmitted directly to the online virtual experience server 102. In other implementations, control instructions may be transmitted from a client device 110 to another client device (e.g., from client device 110b to client device 110n), with the other client device generating the experience instructions using a local virtual experience engine 104. Control instructions may include instructions for playing a voice communication message or other sound from another user on an audio device (e.g., speaker, headphones, etc.), for example, a voice communication or other sound generated using audio spatialization techniques as described herein.

[0111] In some implementations, experience instructions may refer to instructions that enable a client device 110 to render a virtual experience, such as a multi-participant virtual experience. The experience instructions may include one or more of user input (e.g., control instructions), character position and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.).

[0112] In some implementations, a character (or a virtual experience object in general) is composed of components, one or more of which can be selected by the user and automatically combine to assist the user in editing.

[0113] In some implementations, a character is implemented as a 3D model, including a surface representation (also known as a skin or mesh) used to depict the character and a hierarchical set of interconnected bones (also known as a skeleton or rig). The rig may be utilized to animate the character and simulate movements and actions by the character. The 3D model may be represented as a data structure, one or more parameters of which may be modified to change various characteristics of the character, such as dimensions (height, width, girth, etc.), body type, movement style, number / type of body parts, proportions (e.g., shoulder-to-waist ratio), head size, etc.

[0114] One or more characters (also referred to herein as "avatars" or "models") may be associated with a user, who may control the characters to facilitate user interaction with the virtual experience 106.

[0115] In some implementations, characters may include components such as body parts (e.g., hair, arms, legs, etc.) and accessories (e.g., T-shirts, glasses, decorative images, tools, etc.). In some implementations, customizable character body parts include, among others, a head type, a body part type (arms, legs, torso, and hands), a face type, a hair type, and a skin type. In some implementations, customizable accessories include clothing (e.g., shirts, pants, hats, shoes, glasses, etc.), weapons, or other tools.

[0116] In some implementations, for some asset types, e.g., shirts, pants, etc., the online virtual experience platform may provide users with access to simplified 3D virtual object models represented by meshes with a low polygon count, e.g., between about 20 and about 30 polygons.

[0117] In some implementations, the user may also control the scale of the character (e.g., height, width, or depth) or the scale of components of the character. In some implementations, the user may control the proportions of the character (e.g., blocky, anatomical, etc.). Note that in some implementations, the character may not include character virtual experience objects (e.g., body parts, etc.), but the user may control the character (without the character virtual experience objects) to facilitate the user's interaction with the virtual experience (e.g., a puzzle game in which there is no rendered character game object, but the user still controls the character to control in-game actions).

[0118] In some implementations, components such as body parts may be primitive geometric shapes such as blocks, cylinders, spheres, or some other primitive shape such as a wedge, torus, tube, channel, etc. In some implementations, the creator module may publish the user's character for viewing or use by other users of the online virtual experience server 102. In some implementations, creating, modifying, or customizing a character, other virtual experience object, virtual experience 106, or virtual experience environment may be performed by a user using an I / O interface (e.g., a developer interface), with or without scripting (or with or without an application programming interface (API)). For illustrative purposes, characters are described as having a humanoid shape. Characters may have any form, such as a vehicle, an animal, an inanimate object, or other creative form.

[0119] In some implementations, the online virtual experience server 102 may store characters created by a user in the data store 120. In some implementations, the online virtual experience server 102 maintains a character catalog and a virtual experience catalog that can be presented to a user. In some implementations, the virtual experience catalog includes images of virtual experiences stored on the online virtual experience server 102. In addition, a user may select a character (e.g., a character created by the user or another user) from the character catalog to participate in a selected virtual experience. The character catalog includes images of characters stored on the online virtual experience server 102. In some implementations, one or more of the characters in the character catalog may be created or customized by the user. In some implementations, a selected character may have a character setting that defines one or more of the character's components.

[0120] In some implementations, a user's character (e.g., an avatar) can include a configuration of components, and the configuration and appearance of the components, and more generally the character's appearance, can be defined by a character setting. In some implementations, the character setting of a user's character can be selected at least in part by the user. In other implementations, a user can select a character with a default character setting or a character setting selected by another user. For example, a user can select a default character from a character catalog with predefined character settings, and the user can further customize the default character by changing some of the character settings (e.g., adding a shirt with a customized logo). A character setting can be associated with a particular character by the online virtual experience server 102.

[0121] In some implementations, the client devices 110 may each include a computing device such as a personal computer (PC), a mobile device (e.g., a laptop, a mobile phone, a smartphone, a tablet computer, or a notebook computer), a network-connected television, a game console, etc. In some implementations, the client devices 110 may also be referred to as "user devices." In some implementations, one or more client devices 110 may connect to the online virtual experience server 102 at any given moment. Note that the number of client devices 110 is provided by way of example. In some implementations, any number of client devices 110 may be used.

[0122] In some implementations, each client device 110 may include an instance of a virtual experience application 112. In one implementation, the virtual experience application 112 may enable a user to use and interact with the online virtual experience server 102, such as controlling a virtual character within a virtual experience hosted by the online virtual experience server 102, or viewing or uploading content such as the virtual experience 106, images, video items, web pages, documents, etc. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, search, present, or navigate content served by a web server (e.g., virtual characters within a virtual environment, etc.). In another example, the virtual experience application may be a native application (e.g., a mobile application, app, virtual experience program, or game program) that is installed and executed locally on the client device 110 and enables a user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content (e.g., a web page, media viewer) to the user. In one implementation, the virtual experience application may also include an embedded media player (e.g., a Flash or HTML5 player) embedded in a web page.

[0123] According to aspects of the present disclosure, the virtual experience application may be an online virtual experience server application through which a user builds, creates, edits, and uploads content to the online virtual experience server 120, as well as interacts with the online virtual experience server 102 (e.g., participates in a virtual experience 106 hosted by the online virtual experience server 102). Thus, the virtual experience application may be provided to the client device 110 by the online virtual experience server 102. In another example, the virtual experience application may be an application that is downloaded from a server.

[0124] In some implementations, each developer device 130 may include an instance of a virtual experience application 132. In one implementation, the virtual experience application 132 may enable a developer user to use and interact with the online virtual experience server 120, such as controlling a virtual character within a virtual experience hosted by the online virtual experience server 102, or viewing or uploading content such as the virtual experience 106, images, video items, web pages, documents, etc. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, search, present, or navigate content served by a web server (e.g., virtual characters within a virtual environment, etc.). In another example, the virtual experience application may be a native application (e.g., a mobile application, app, virtual experience program, or game program) that is installed and executed locally on the developer device 130 and allows a user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content (e.g., a web page, media viewer) to a user. In one implementation, the virtual experience application may also include an embedded media player (e.g., a Flash or HTML5 player) embedded in a web page.

[0125] According to aspects of the present disclosure, the virtual experience application 132 may be an online virtual experience server application through which a user builds, creates, edits, and uploads content to the online virtual experience server 120, as well as interacts with the online virtual experience server 102 (e.g., provides and / or participates in virtual experiences 106 hosted by the online virtual experience server 102). Thus, the virtual experience application may be provided to the developer device 130 by the online virtual experience server 102. In another example, the virtual experience application 132 may be an application downloaded from the server. The virtual experience application 132 may be configured to interact with the online virtual experience server 102 and obtain access to user credentials, user currency, etc., for one or more virtual experiences 106 developed, hosted, or provided by the virtual experience developer.

[0126] In some implementations, a user may log in to the online virtual experience server 102 through a virtual experience application. A user may access a user account by providing user account information (e.g., a username and password), and the user account is associated with one or more characters available to participate in one or more virtual experiences 106 on the online virtual experience server 102. In some implementations, with the appropriate credentials, a virtual experience developer may gain access to virtual experience virtual objects, such as in-platform currency (e.g., virtual currency), avatars, special abilities, accessories, etc., owned by or associated with other users.

[0127] In general, functionality described in one implementation as being performed by the online virtual experience server 102 may also be performed by the client device 110 or server in other implementations, where appropriate. In addition, functionality attributed to a particular component may be performed by a different component or multiple components operating together. The online virtual experience server 102 may also be accessed as a service offered to other systems or devices via an appropriate application programming interface (API) and, therefore, is not limited to use in a website.

[0128] Figure 2 - Exemplary body cage 2 is a diagram illustrating an exemplary body cage 200 according to some implementations. The body cage 200 in the example of FIG. 2 is an outer cage that encases or is superimposed on the outer surface / contour of a humanoid body form that serves as a mannequin. The underlying humanoid body form (mannequin, not shown) encased by the body cage 200 may be represented or constituted by a body mesh that includes multiple polygons and their vertices. The polygons of the body mesh (as well as the polygons of the garment mesh) are triangles, and the surface area of ​​each triangle provides a face or mesh surface.

[0129] The body cage 200 comprises a plurality of feature points 202 that define, otherwise identify, or correspond to the shape of the mannequin. In some implementations, the feature points 202 are formed by the vertices of segments / edges 204 of a plurality of polygons (or other geometric shapes) on the mannequin. According to various implementations (although not shown as such in FIG. 2 ), the polygons are triangles, and the surface area of ​​each triangle provides a face or cage face. In some implementations, the feature points 202 are not necessarily formed by the vertices of any polygons, but can be discrete points.

[0130] Body cage 200 of FIG. 2 is an example of a low-resolution body cage having 642 feature points (or some other number of feature points) for a fingerless humanoid body geometry. Other examples may use a body cage with 475 feature points (or some other number of feature points). A body cage for a fingerless humanoid geometry may have, for example, 1164 feature points (or some other number of feature points). A higher resolution body cage may include 2716 feature points (or some other number of feature points). These numbers (or ranges) of feature points are just some examples, and the number of feature points may vary from implementation to implementation depending on factors such as preferred resolution, the processing power of the 3D platform, user preferences, size / shape of the mannequin, etc.

[0131] Example 3 - Exemplary Body Cage 3 shows another exemplary body cage 300 according to some implementations. A cage may be provided for any avatar body shape or clothing shape. The body cage 300 in the example of FIG. 3 is an outer cage that encases or is superimposed on the exterior surface / contour of a typical gaming avatar body shape.

[0132] 2, the body cage 300 of FIG. 3 may have the same number of feature points. In some implementations, the body cage 300 may have a different number of feature points compared to the body cage 200, such as fewer or more feature points 302, as a result of a different (simpler or more complex) geometric shape of the game avatar and / or based on other factors. Thus, the number of feature points may vary from body cage to body cage and may be selected based on different body shapes or other body characteristics.

[0133] Figure 4 - Body cage parts FIG. 4 shows an example of portions of a body cage 400 grouped into corresponding body parts, according to some implementations.

[0134] In some implementations, for bandwidth and performance / efficiency purposes, or other reasons, the number of feature points in the cage may be reduced to fewer than the number provided above, such as 475 feature points (or some other number of feature points). Additionally, in some implementations, the feature points (vertices) in the body cage may be arranged in multiple groups (e.g., 15 groups), each representing a portion of the body shape.

[0135] More specifically, the 15 body parts shown in FIG. 4 (for a humanoid mannequin) are: head, torso, waist, right foot, left foot, left lower leg, right lower leg, left upper leg, right upper leg, left hand, right hand, left forearm, right forearm, left upper arm, and right upper arm. The number of parts in any body shape may be more or less than the 15 body parts shown. For example, a "one-armed" avatar character may have 12 body parts (as opposed to 15) due to the omission of the hand, forearm, and upper arm. Furthermore, other body shapes may include fewer or more body parts depending on factors such as body geometry, preferred resolution, processing power, and the type of avatar character (e.g., animal, alien, monster, etc.).

[0136] Each of the 15 groups / regions in Figure 4 includes feature points that define that region of the avatar body. Such groups of feature points may be mapped to corresponding clothing. For example, feature points within body cage 400 that define the left / right forearms, left / right upper arms, and torso may be used as an outer cage that is mapped to an inner cage of a jacket, and the graphical representation of the jacket would consist of a graphical mesh that renders the left / right arm and torso of the jacket logically and correspondingly fitting over the left / right arm and torso of the avatar body.

[0137] Furthermore, this separation into multiple groups (as shown in Figure 4) allows for customized fitting of clothing to atypical body shapes. For example, a 3D avatar may be in the form of a "one-armed" avatar character missing its left arm. Thus, the body cage for that 3D avatar would lack groups of feature points corresponding to the left hand, left forearm, and left upper arm.

[0138] Then, when a jacket is selected to fit that 3D avatar, the right forearm, right upper arm, and torso of the jacket can be deformed to fit the corresponding right forearm, right upper arm, and torso of the 3D avatar (body mannequin), and the left forearm and left upper arm of the jacket are not deformed (e.g., remain rigid in their original shape from their parent space) because there is no left arm cage to deform them into on the body mannequin.

[0139] Figure 5 - Deformed clothing layer on the body cage 5 shows an example of a clothing layer 500 deformed on a body cage (such as body cage 400 shown in FIG. 4 ) according to some implementations. Clothing layer 500 is a graphical representation of a jacket (shown in gray shading in FIG. 5 ) having portions that may be generated / rendered using a polygon mesh 502 (e.g., a clothing mesh) composed of a collection of vertices, edges, and faces (which may be triangular or other polygonal faces).

[0140] The clothing layer 500 includes an inner cage (not shown in FIG. 5) having feature points that correspond to feature points of the body cage 400. Specifically, the feature points of the inner cage of the clothing layer 500 are mapped to the feature points of the body cage 400, which comprises the left and right forearms, the left and right upper arms, and the torso.

[0141] In some implementations, this mapping involves directly mapping feature points of the inner cage of the garment layer 500 to the coordinate locations of corresponding feature points on the arms and torso of the body cage 400. Such mapping may involve a 1:1 correspondence if both cages have the same number of feature points, or the mapping may be n:1 or 1:n (where n is an integer greater than 1), in which case multiple feature points in one cage may map to the same feature points in the other cage (or some feature points may not map).

[0142] The garment layer 500 further includes an outer cage having feature points spaced apart and linked to corresponding feature points of the inner cage of the garment layer 500. The feature points of the outer cage of the garment layer 500 define or are otherwise positioned along the contour / geometry of the jacket's exterior, such as defining features such as the jacket's hood 504, cuffs 506, straight-cut torso 508, etc.

[0143] According to various implementations, spatial distances (e.g., the spatial distances between feature points of the inner cage of the clothing layer 500 and corresponding feature points of the outer cage of the clothing layer 500) are kept constant during the process of attaching the clothing layer 500 over the outer cage of an existing layer (or avatar body). In this way, feature points of the inner cage of the clothing layer 500 can be mapped to feature points of the body cage 400 to fit the inside of the jacket over the avatar's torso and arms.

[0144] Then, with the distance between the feature points of the inner cage of clothing layer 500 and the corresponding feature points of the outer cage of clothing layer 500 kept constant, the outer contour of the jacket may also be deformed to fit the shape of the avatar body, thereby resulting in at least partial preservation of the visual appearance (graphical representation) of the jacket's hood, cuffs, straight-cut torso, and other surface features, while matching the shape of the avatar body shown in Figure 5. In this way, clothing layer 500 may be deformed in any suitable manner to fit any shape / size of avatar body (body cage), such as tall, short, skinny, muscular, humanoid, animal, alien, etc.

[0145] Figure 6 - Exemplary outer cage 6 shows an example in which the clothing layers and portions of body cage 400 of FIG. 5 are used to form outer cage 600, according to some implementations. In some implementations, additional clothing layers may be placed over other clothing layers (e.g., in response to user selection). More specifically, outer cage features of clothing layer 500 of FIG. 5 are now combined with features of body cage 400 to result in composite outer cage 600 composed of features of exposed portions of body cage 400 and features along the outer surface of the jacket.

[0146] For example, the exposed outer surface 602 of the jacket (formed by the body, hood, and sleeves of the jacket) provides one set of feature points, the exposed legs, hands, head, and parts of the chest of the body not covered by the jacket provide another set of feature points, and these two sets of feature points (combined) provide the feature points of the outer cage 600.

[0147] The features of the outer cage 600 in Figure 6 that correspond to and define the outer surface / shape of the jacket may be the same features as the outer cage of the garment layer 500 in Figure 5. In some implementations, different and / or additional and / or fewer features may be used for areas of the jacket in the outer cage 600 in Figure 6 compared to the features of the outer cage of the jacket (garment layer 500) in Figure 5.

[0148] For example, if a higher resolution or more accurate fit is preferred for the next layer of clothing above outer cage 600, additional feature points may be calculated for outer cage 600 enclosed by the jacket region (compared to the outer cage of clothing layer 500 in FIG. 5 ). Similarly, if a lower resolution or less accurate fit is preferred for the next layer of clothing above outer cage 600, and / or due to other considerations such as improved processing / bandwidth efficiency offered by using fewer feature points where possible, feature points may be calculated for outer cage 600 enclosed by the jacket region (compared to the outer cage of clothing layer 500).

[0149] In operation, if the user provides input to place an additional clothing layer (such as an overcoat or other clothing) over the jacket (clothing layer 500) and / or other portions of the avatar body, feature points of the inner cage of such additional clothing layer are mapped to corresponding feature points of the outer cage 600. Thus, deformation may be performed in a manner similar to that described with respect to Figure 5. According to some implementations, radial basis function (RBF) techniques and / or other similar interpolation techniques may be used to deform clothing underlying the avatar or clothing attached over body parts.

[0150] 5 and 6 for layering, a first clothing layer (clothing layer 500) is made to encase the body by matching feature points of the avatar body's "outer cage" (body cage 400) with feature points of the first clothing layer's "inner cage." This matching can be done in the cage's UV space (UV refers to the coordinate system) so as not to rely on the number of feature points that match exactly between the inner and outer cages.

[0151] For example, a feature point may be a vertex that has both position and texture space coordinates. Texture space coordinates are typically expressed in the range [0,1] for each of the U and V coordinates. Texture space can be thought of as an "unwrapped" normalized coordinate space for vertices. By performing the correspondence of the two sets of vertices in UV space and not using their positions, the vertex-to-vertex correspondence can be performed in normalized space, thereby eliminating the difficult task of precise vertex-to-vertex index mapping.

[0152] Thus, in a technique for layering clothing, each avatar body and clothing item is associated with an "inner cage" and an "outer cage." For an avatar body, the inner cage represents the default "mannequin" (and different mannequins may be provided for different avatar body shapes), and the avatar body's "outer cage" represents the envelope around the avatar body's shape. For a clothing item, the "inner cage" represents the inner envelope used to define how the clothing item will encase the underlying body (or a body that already has a previous clothing layer on it), and the "outer cage" represents how the next clothing layer will encase this particular clothing item when worn on the avatar body.

[0153] According to various implementations, the various cages described herein may be invisible at runtime. For example, while participating in a virtual experience, including moving an avatar across the virtual 3D environment of the virtual experience, placing clothing on the avatar body, donning the clothing, animating the avatar, etc., the vertices and segments of the cage may be invisible to other users / viewers of the 3D environment. Also, the avatar and its transformed clothing may be presented at runtime in a cage-less manner, so that only the visual mesh of the transformed clothing, skin, avatar body parts, etc. is visible to the user; in fact, one or more cages may be present on the avatar for purposes described herein, such as transforming clothing, encasing avatar body parts and clothing items, modifying the avatar body, etc., but are invisible to the user at runtime. The cages may be made visible to the user (e.g., during a setup phase, via a view / edit cage command, etc.) so that the user can view and manipulate the cages as needed to modify the avatar body as described herein, to create the cages, or for other purposes.

[0154] Figure 7 - Interpolation between two body cages 7 shows an example of interpolation between two body cages to obtain a new body cage 700, according to some implementations. More specifically, the example interpolation of FIG. 7 may be performed in a situation where a user has a current avatar body and the user wants to change / transform the current avatar body to some other (target) avatar body that exists within the virtual experience, in a library, etc. The interpolation and corresponding modification of the current avatar body may be performed at runtime, such as during a session in a studio or other setting where the user can create and edit graphical objects while the avatar is participating in the virtual experience.

[0155] In the example of Figure 7, purely for purposes of illustration and explanation, the user's current avatar body may be a humanoid body having body cage 200 of Figure 2, and the user may wish to transform the current avatar body (or some portion thereof) into a target avatar body. The target avatar body selected by the user in the example of Figure 7 may be a geometric avatar body having body cage 300 of Figure 3.

[0156] The new avatar body is shown in Figure 7 as having a body cage 700. The new avatar body can be a complete transformation from one or more portions of the original avatar body to one or more portions of the target avatar body. Figure 7 shows both a complete transformation and a partial transformation from one or more portions of the original avatar body to one or more portions of the target avatar body.

[0157] As an example of a complete transformation, the shape of the torso 702 of the new avatar body has been transformed to perfectly match the rectangular shape of the target avatar's torso (with body cage 300), and the curved / tapered torso of the original avatar body (with body cage 200) is no longer present in the new avatar body, having been completely morphed or otherwise transformed into the rectangular torso 702.

[0158] As an example of a partial transformation, the shape of arm 704 of the new avatar body is a mixture / composite between the curved / tapered arm of the original avatar body (with body cage 200) and the rectangular arm of the target avatar body (with body cage 300). For example, arm 704 now has a more rectangular shape, similar to the arms of body cage 300, but still retains some of the curvature and tapering of the arms of body cage 200.

[0159] Thus, a complete body transformation can be a body transformation in which a body part is completely transformed, and a partial body transformation can be a body transformation in which one or more body parts are partially or not transformed.

[0160] The new avatar body's body cage 700 also represents a partial transformation in that not all portions of the entire avatar body are morphed. For example, only the torso 702 and one arm 704 undergo a transformation, while the shape of other portions of the new avatar body (e.g., the head, the other arm, the legs, etc.) remains unchanged relative to the original avatar body. In various implementations, various portions of the avatar body may undergo a partial or complete transformation, while other portions may not undergo any transformation. In some implementations, the entire avatar body may undergo a partial or complete transformation. In some implementations, morphing a portion of the first avatar body that is less than the entire first avatar body includes morphing a portion of the first avatar body to perform a partial transformation of the portion of the first avatar body.

[0161] The new avatar body (with body cage 700) can be the same or a different size than either or both of the original avatar body (with body cage 200) and the target avatar body (with body cage 300). In the example of Figure 7, the new avatar body (with body cage 700) is scaled to be smaller in size than both the original avatar body and the target avatar body.

[0162] To obtain the new avatar body (having body cage 700), one or more interpolation operations 706 may be performed in some implementations. For example, a linear or non-linear interpolation may be performed between body cage 200 and body cage 300. Interpolation may be performed between values / coordinates corresponding to vertices or segments of the two body cages (body cage 200 and body cage 300) to obtain the resulting vertices / segments of new body cage 700.

[0163] Alternatively, or additionally, at least some of the values / coordinates of the vertices / segments of the new body cage 700 can be calculated / generated as new values ​​that are not interpolated from other values. This may be done, for example, when attempting to create new vertices / segments for the new body cage 700 in a particular region of the avatar body, and there are no vertices / segments nearby the two body cages 200 and 300 that can form the basis for interpolation.

[0164] 7 corresponds to an implementation in which a new avatar body (with a new body cage) is generated based on or relative to two other avatar bodies (with their respective body cages). In some implementations, the new avatar body can be generated in a more freeform manner, not necessarily based on an existing target avatar body as a reference.

[0165] Figure 8 - Generating a new body cage 8 shows an example of such generation of a new body cage 800, according to some implementations. Purely for purposes of illustration and explanation, a user's current avatar body may be a humanoid body having body cage 200 of FIG. 2, and the user may wish to transform the current avatar body (or some portion thereof) into a target avatar body.

[0166] To perform this transformation, the user may use transformation tools to manipulate the vertices and / or segments of body cage 200. For example, the user may click and drag an arm vertex or segment to a new position, as shown at 800. The user may click and drag a torso vertex or segment to a new position, as shown at 802. Instead of clicking and dragging existing vertices / segments, the user may also use transformation tools to add or remove vertices / segments from the cage, draw / redraw portions of the cage, etc. during the process of generating a new avatar body.

[0167] As explained above with respect to cages and with respect to Figures 2-6, there are interrelationships and dependencies between multiple cages. For example, the body cage encapsulates (completely surrounds) the avatar body (including its body mesh), the inner cage of a first clothing item is mapped to the body cage, the outer cage encapsulates the first clothing item (including its clothing mesh), the inner cage of a second clothing item is mapped to the outer cage of the first clothing item, which encapsulates the second clothing item (including its clothing mesh), etc.

[0168] Given such interrelationships and dependencies, manipulation or other modification / transformation of at least one cage, in some implementations, causes an automatic and corresponding modification / transformation of one or more other cages. For example, if the body cage of a current avatar body is modified (as shown in FIGS. 7 and 8) to change the shape of the avatar body, if the avatar body is wearing clothing, the corresponding cages of one or more of the layered clothing items are also automatically modified / updated to deform / adapt the clothing items in a dynamic manner to match the changed shape of the avatar body. The modification in the cage, in turn, causes appropriate modifications to the mesh, skinning, and other visual aspects of the avatar body and / or its clothing.

[0169] In other examples, manipulating the cage of a clothing item can be performed instead of, or in addition to, modifying the body cage to change the shape or other appearance of the avatar. Using the example described above with a current humanoid avatar wearing a round baseball cap and a target alien with a conical head, the user can manipulate a transformation tool to reshape the outer cage of the baseball cap from rounded to conical. This reshaping of the outer cage then changes the visual appearance of the baseball cap from rounded to conical, and also modifies the inner cage of the baseball cap and the body cage of the underlying humanoid avatar body's head so that the new avatar body has a conical head.

[0170] According to various implementations, morphing or other transformation of the avatar body (including its clothing) may be performed at run time during the virtual experience. In such implementations, the cage is not necessarily visible to the user while participating in the virtual experience. Also, the user may select a target avatar or other target graphical object within the virtual experience, and a client-side or server-side virtual experience engine or other related component may perform the appropriate cage manipulation (as described above with respect to FIG. 6 ) in a manner transparent to the user, such as via a background process. Thus, the user may seamlessly see the altered / changing visual appearance of the avatar body during the virtual experience without actually seeing the cage itself being manipulated.

[0171] Figure 9 - Avatar body transformation during virtual experience 9 shows an example of an avatar body transformation while the avatar is participating in a virtual experience 900, according to some implementations such as those described above. In the virtual experience 900, at 902, the user's current avatar 904 is humanoid with generally geometric / rectangular shapes for its various body parts (e.g., head, arms, legs, torso, etc.). The user selects another avatar in the virtual experience 900 as a target avatar 906 (e.g., via a click with a mouse cursor or some other input tool).

[0172] If the user selects the target avatar 906, then at 908, a transformation of the current / original avatar 904 into a new (modified / changing) avatar 910 occurs. At 908, compared to the original avatar 904, the new avatar 910 has a larger left arm, a tapered torso, shrugged shoulders, etc., similar to the corresponding body parts of the target avatar 906. At 912, further morphing is performed such that the new avatar 910 is further modified to more closely resemble the target avatar 906 with its drooping head.

[0173] Figure 10 - Avatar body transformation during virtual experience Figure 10 shows another example of avatar body transformation while the avatar is participating in a virtual experience 1000, according to some implementations. Specifically, Figure 10 shows that a user may select multiple avatars as targets for morphing the avatar's current avatar body. In virtual experience 1000, at 1002, the user's current avatar 1004 has an alien body with a slim build / shape. A first target avatar 1006 is a humanoid avatar with a muscular torso shape. A second target avatar 1008 is a monster with a droopy monster head with horns.

[0174] At 1010, the user selected a first target avatar 1006, so the original avatar 1004 was transformed into a new avatar 1012 with a muscular torso like the first target avatar 1006. At 1014, the user selected a second target avatar 1008, so the new avatar 1012 continues to transform / morph to have a droopy monster head like the target avatar 1008 and be shorter in height.

[0175] Figure 11 - Avatar body transformation during virtual experience 11 illustrates another example of avatar body transformation while the avatar participates in a virtual experience 1100, according to some implementations. As previously described, portions of the avatar body can be modified (rather than modifying the entire avatar body), and then corresponding changes can be made to clothing, skinning, accessories, etc. associated with the modified portion of the avatar body.

[0176] At 1102 in virtual experience 1100, the user's current avatar 1104 has a humanoid head with a nose, hair, lips and lipstick, eyes and eyelashes, etc. At 1106, the user selects a different avatar head within virtual experience 1100, so the head of the current avatar 1104 begins to morph from a humanoid head to a different head shape (e.g., animal-like) for the new avatar 1108. This morphing also affects the appearance (e.g., shape and size) of the nose, lips, eyes, etc.

[0177] At 1110, the head of the new avatar 1108 continues to morph further into an animal head. Thus, the new avatar 1108 has a more pronounced animal-like appearance to the head, hair, nose, lips, eyes, etc.

[0178] Figure 12 - Avatar transformation in the settings 12 shows an example of avatar transformation in a configured environment 1200, according to some implementations. The configured environment 1200 may be a studio or other type of environment where a user can configure an avatar outside of the runtime environment of a virtual experience.

[0179] In this manner, the configuration environment 1200 may be an auxiliary feature that is not included in the virtual experience itself, but is tied to the virtual experience. Alternatively, or additionally, the configuration environment may be separate from any particular virtual experience, but its output (including avatars and other graphical objects) may be used and applied to the virtual experience.

[0180] In the setting 1200, there are multiple avatar models available: a user avatar 1202 (Bazooka), a first target avatar 1204 (Model 9), and a second target avatar 1206 (Roxy). The user avatar 1202 in this example has a chunky, alien-like avatar body and is skeletal in shape.

[0181] The user has selected target avatar 1206 as the target avatar. Thus, the user's avatar 1202 is transformed into a new avatar 1208 that is rendered in the setting 1200. The new avatar 1208 has a more human-like shape corresponding to target avatar 1206 and is taller, while retaining some of the skeletal features of the original avatar 1202.

[0182] According to various implementations, the configuration environment 1200 is provided with adjustment tools and / or other types of transformation tools 1210. For example, the transformation tool 1210 indicates that the transformation tool 1210 corresponds to a "Morph Body Plugin" and provides instructions such as "Select avatar A, then select B to morph A into B. The value is how much A morphs into B; if 0, no change; if 1, A looks like B."

[0183] 12, the transformation tools 1210 include a slide bar or other similar tool ("Morph Value") for controlling the amount of morphing (e.g., morph value) between the current avatar and the target avatar. There may be a button indicating "Current Deform Head" and a button indicating "Apply Morph and Value."

[0184] Thus, the new avatar can have a minimum morphing value of 0 (e.g., no change from the current avatar), a maximum morphing value (e.g., the new avatar is a complete transformation into the target avatar), or any other morphing value between the minimum and maximum morphing values ​​(e.g., the new avatar is a mixture between the original avatar and the target avatar, as shown in FIG. 12).

[0185] Although not shown in Figure 12, some implementations of the setup environment 1200 allow the user to manipulate the cage for purposes of modifying the avatar body, as shown in Figure 2. Various transformation tools may be provided in the setup environment 1200 to allow the user to click and drag, draw, delete, modify, etc., vertices of the cage.

[0186] Figure 13 - Layering clothing on an avatar body in a virtual experience 13 illustrates an example of layering clothing on an avatar body for an avatar participating in a virtual experience 1300, according to some implementations. At 1302, the user's avatar 1304 is a new avatar converted from a previous avatar, such as via techniques described herein, and is wearing several pieces of clothing over a body cage (not shown). The inner cages (not shown) of each outer clothing layer of the avatar 1304 have been deformed to accommodate / fit each outer clothing layer to the modified / new avatar 1304.

[0187] In virtual environment 1300, avatar 1304 is running toward a clothing item (e.g., coat 1306) from among a selection of other clothing items. When the avatar puts on coat 1306 at 1308, coat 1306 is deformed to fit and accommodate the underlying clothing layers. The deformation of the coat (e.g., cage mapping, cage deformation, etc.) can be performed seamlessly by an engine running virtual environment 1300 and / or by some other component such that the fit is seamless from the user's perspective (e.g., the user need only click on coat 1306, and coat 1306 is automatically fitted to avatar 1304).

[0188] In the above-described implementations and examples thereof, the original avatar body is changed into another (new) avatar body, such as by morphing one or more portions of the original avatar body in the manner illustrated and described above. The skinning / mesh and any clothing layers worn by the original avatar body are also deformed to adaptively fit the new avatar body.

[0189] After the geometry (e.g., avatar body, skinning, clothing layers, etc.) is changed / updated, the user may animate the avatar by running, laughing, blinking, waving arms, etc. Examples are provided below in Figures 14-17.

[0190] Figure 14 to Figure 17 - Avatar changes and animations Figures 14-17 show examples of transformations and animations of an avatar 1400, according to some implementations. Each of Figures 14-17 shows the head of the avatar 1400 and three different versions of the avatar 1400.

[0191] In Figure 14, avatar 1400 and the other three avatars have not yet been morphed, so the other three avatars are identical to avatar 1400. Avatar 1400 in Figure 14 has a neutral expression (e.g., neither smiling nor frowning) and wide-open eyes, and the other three avatars have this same appearance as well.

[0192] In Figure 15, avatar 1400 is shown in its original (unmorphed) form for reference, while the other three avatars have begun to morph into different head shapes. Avatar 1400 is also now animated to be smiling (lips parted, teeth visible). Corresponding smiles can also be seen in the animations of the other three avatars.

[0193] In Figure 16, avatar 1400 is shown in its original (unmorphed) form for reference, while the other three avatars continue its morphing into different head shapes with a smiling animation that corresponds to the smiling animation of avatar 1400. In Figure 17, avatar 1400 is animated with one eye partially blinking and lips grinning, and animations showing the same facial expression are performed on the other three avatars.

[0194] According to various implementations, an avatar may be provided with skinning such that the avatar's mesh is bound to the joints and bones of the avatar's skeleton. Thus, movement of the avatar's joints / bones results in corresponding skin deformation during animation, as shown in Figures 15-17 described above. In some implementations, the avatar's skeleton may be an estimated skeleton composed of virtual joints and virtual bones.

[0195] According to various implementations, when the geometry of the avatar body is changed as described in the examples above, such as by changing the geometry / shape of the arms, legs, torso, etc. for the new avatar, the skeleton is also updated for the new avatar. Updating the skeleton (including updating its joints and bones, etc.), as well as updating the skinning, ensures that the new avatar animates correctly, as shown in Figures 15-17.

[0196] In some implementations, such updates may be performed by interpolating attachment positions (e.g., joints) for the new avatar body, which may be obtained by interpolating between the attachment positions of the skeleton of the original avatar body and the attachment positions of the skeleton of the target avatar body.

[0197] In some implementations, interpolation may be performed between the vertices of the body cage of the original avatar body and the vertices of the body cage of the target avatar body to obtain attachment points for the skeleton of the new avatar body. Thus, once the original avatar body is morphed by manipulating its body cage, a similar morphing can be performed on the skeleton of the original avatar body to rearrange the joints and bones of the skeleton.

[0198] As an example, a body cage is associated with an avatar body mesh by encasing the avatar body mesh, and thus approximates the shape, size, contours, etc., of the body parts represented by the mesh. Given that the rig or rigging ties the avatar mesh to the joints and bones of the avatar skeleton, deformations of the body cage result in corresponding deformations of the mesh, and the same or similar deformations are applied to the skeleton (due to the binding of the mesh to the skeleton).

[0199] When interpolation is performed between vertices of a cage pair that have a direct 1:1 correspondence between the vertices, the result of the interpolation provides the vertices for the body cage of the avatar body. Additionally, there may be vertices on the body cage of the source avatar body that do not correspond to any of the vertices on the body cage of the target avatar body, or vice versa. In such situations, UV matching techniques for mapping between graphical objects, or other types of techniques, may be used to identify the closest vertices to use for interpolation. To further improve the accuracy of the interpolation, aligning the coordinate systems of the cage pair may also be performed.

[0200] Figure 18 - Changing the 3D avatar body 18 is a flowchart illustrating a computer-implemented method 1800 for modifying a three-dimensional (3D) avatar body, according to some implementations. For simplicity, various operations in method 1800 are described in the context of a virtual experience (VE) application on a client device that performs the operations.

[0201] 22 , some of the operations of method 1800 and / or any other methods described herein may alternatively or additionally be performed in whole or in part by a VE engine in a VE platform located at a server. Example method 1800 may include one or more operations represented by one or more blocks, such as blocks 1802-1806. Various blocks of method 1800 and / or any other processes described herein may be combined into fewer blocks, divided into additional blocks, supplemented with further blocks, and / or eliminated, based on the implementation.

[0202] The method 1800 of Figure 18 is described herein with reference to elements shown in Figures 2-17 and other figures. In some implementations, the operations of method 1800 may be performed in a pipelined sequential manner. In other implementations, some operations may be performed out of order, in parallel, etc.

[0203] In block 1802, a first avatar body to be modified is identified. For example, the first avatar body may be a user's current avatar body while the user is participating in a virtual experience or while the user is in a set environment, such as a studio. The first avatar body has a corresponding first body cage. Block 1802 may be followed by block 1804.

[0204] In block 1804, a target avatar is identified. For example, a user may identify a target avatar body in a virtual environment or a set environment. The target avatar body is a body to which the user wants to morph the current avatar body. The target avatar body has a corresponding target body cage. Block 1804 may be followed by block 1806.

[0205] At block 1806, a transformation of the first avatar body is performed. For example, interpolation may be performed between the first body cage and the target body cage to generate a second body cage corresponding to the second avatar body. Thus, the second avatar body may be a mixture between the first avatar body and the target avatar body, or may be completely transformed into the target avatar body. Additional details of how interpolation may be performed are discussed with reference to FIG. 20.

[0206] Figure 19 - Changing the 3D avatar body 19 is a flowchart illustrating another computer-implemented method 1900 for modifying a three-dimensional (3D) avatar body, according to some implementations. For example, method 1900 may be used for free-form transformation of an avatar body without necessarily using a target avatar body as a reference.

[0207] In block 1902, a first avatar body to be modified is identified. For example, the first avatar body may be the user's current avatar body while the user is in a set environment, such as a studio. In some implementations, the user's avatar may be present in a running virtual environment, so the avatar may be modified by pausing the virtual environment, exiting the virtual environment to enter the set environment, or modifying the avatar within the running virtual experience itself. The first avatar body has a corresponding first body cage. Block 1902 may be followed by block 1904.

[0208] At block 1904, a transformation of the first avatar body is performed. For example, the transformation may be performed by manipulating the first body cage (e.g., rearranging vertices / segments of the first body cage) to generate a second body cage corresponding to the second avatar body. The second body cage may be used to provide the transformation of the first avatar body to the second avatar body. The manipulation may include rearranging a portion of the first body cage. Additional details of how the transformation may be performed are discussed with reference to FIG. 20.

[0209] Figure 20 - Clothing layer deformed on the body cage 20 is a flowchart illustrating a computer-implemented method for performing skin morphing, according to some implementations. Figure 20 is a flowchart illustrating a computer-implemented method 2000 for modifying a three-dimensional (3D) avatar body, according to some implementations. For example, method 2000 may be used for free-form transformation of an avatar body without necessarily using a target avatar body as a reference. Method 2000 may begin at block 2002.

[0210] In block 2002, affine corrections are calculated for the bones. For each bone, linear algebra is used to calculate a 3x4 affine transformation that best fits the transformation of the affected point cloud. This allows for the skinning calculation to be modified to better fit changes in the mesh shape. j Block 2002 may be followed by block 2004.

[0211] In block 2004, an affine correction is applied to the bone transformations. This correction is applied to the bone transformations in the skinning calculations.

[0212]

number

[0213] Block 2004 may be followed by block 2006.

[0214] The corrections are bubbled up through skeleton calculations in block 2006. Block 2006 may be followed by block 2008.

[0215] In block 2008, the correction is incorporated into the local bind transform. Valve up and incorporate the correction into a new local bind transform LB' that produces the same result. j Incorporated into LB' j =B j -1 D j ·B j' LB j This is D j Remove the transformation and use the original LBS formulation to create a new binding transformation LB' that produces the same result. j Block 2008 may be followed by block 2010.

[0216] In block 2010, the new skin mesh is created and / or published. Created means that it exists as a mesh visible to local users. Published means that it has been pushed to the Roblox CDN so that other users can view or purchase it. This requires security coordination and other processes in order for the new skin mesh to be used in these ways. Block 2010 may be followed by block 2012.

[0217] In block 2012, translations and rotations are updated. The FACS rig contains translations and rotations for every joint in every pose of the rig. After the shape changes, these values ​​need to be updated so the pose continues to work with the new shape. Such updates can be performed using method 2100, as described with respect to FIG. 21.

[0218] Figure 21 - Clothing layer deformed on the body cage 21 is a flow chart illustrating a computer-implemented method 2100 for performing Facial Action Coding System (FACS) pose morphing. The method 2100 may begin at block 2102.

[0219] Regarding FACS pose morphing, the current FACS rig contains many poses, and each pose is a local transformation P for each joint in the head rig. j The skin morphing technique involves a set of binding transformations LB' j , which changes the parent space of these local pose transforms. j is the resulting M j The affine correction transformation D j is the resulting M j Do not change the rotation of the transform. To further improve the results, use per-pose, per-joint rotations. j,k In the process, the per-pose, per-joint translation t j,k Further fine-tuning of the pose may occur as well. Such a process may be performed one pose at a time.

[0220] In block 2102, a deformed head mesh pose is calculated from the original head mesh. Specifically, the original LBS deformed mesh pose is calculated from the original head shape. The pose is defined as a set of joint transformations. These joint transformations are used in conjunction with techniques such as linear blend skinning techniques to construct the mesh shape at that pose. Block 2102 may be followed by block 2104.

[0221] In block 2104, the skin load points are projected to the nearest point to generate new points. For each joint in the pose, the joint's skin load point is projected to the nearest point on the posed mesh to provide a new set of points. These points represent the locations on the original mesh that are closest to that joint's destination in that pose. For example, consider a smile pose, where the corners of the mouth slide into the cheek area. It can be useful to know how the cheeks have been reshaped, and whether or how the smile pose should be adjusted accordingly. Block 2104 can be followed by block 2106.

[0222] In block 2106, an affine correction is applied to the new points to generate a transformation. The same 3x4 affine fit function (used in block 2004) may be used to calculate how these points transformed between the original and modified shapes. Block 2106 may be followed by block 2108.

[0223] In block 2108, the rigid body components of the transformation are extracted. The rigid body components of the transformation can then be extracted. Some systems, including some rigid body simulation systems, cannot handle scaling and shear effectively, so these components must be removed before providing the correction to those systems. Block 2108 can be followed by block 2110.

[0224] In block 2110, the translation and / or rotation are fine-tuned based on the extracted rigid body components. Block 2110 may be followed by block 2112.

[0225] In block 2112, the Euler angles are calculated by decomposing the rotation matrix and interpolating the correction values. To minimize changes to the Euler interpolation, minimal changes are made to the Euler angles that fit the rotation. These angles are calculated by decomposing the rotation matrix and incrementally interpolating the correction values.

[0226]

number

[0227] , then

[0228]

number

[0229] (This shows how the correction rotation with the subscript of 2 should be incorporated into the rotation calculation), then x' = x1 + x2, y' = y1 + y2, and z' = z1 + z2 (This shows how the Euler angles from the correction matrix (the subscript of 2 in the previous formula) can be added directly to the original rotation to produce the same result). This shows the Euler rotation matrix decomposed into separate x, y, and z rotation matrices. Here, uppercase letters are translation matrices and lowercase letters are scalar values.

[0230] Figure 22 - An exemplary computing device FIG. 22 is a block diagram illustrating an exemplary computing device 2200 that may be used to implement one or more features described herein, according to some implementations. In one example, the computing device 2200 implements a computer device (e.g., 102 and / or 110 of FIG. 1 ) and may be used to perform implementations of suitable methods described herein. The computing device 2200 may be any suitable computer system, server, or other electronic or hardware device. For example, the computing device 2200 may be a mainframe computer, a desktop computer, a workstation, a portable computer, or an electronic device (such as a portable device, a mobile device, a mobile phone, a smartphone, a tablet computer, a television, a TV set-top box, a personal digital assistant (PDA), a media player, a gaming device, a wearable device, etc.). In some implementations, the computing device 2200 includes a processor 2202, a memory 2204, an input / output (I / O) interface 2206, and audio / video input / output device(s) 2214.

[0231] Processor 2202 may be one or more processors and / or processing circuits that execute program code and control the basic operations of computing device 2200. A "processor" includes any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. A processor may include a system with a general-purpose central processing unit (CPU), multiple processing units, dedicated circuits for achieving functions, or other systems. Processing need not be limited to a particular geometric location or have temporal limitations. For example, a processor may perform its functions in "real time," "offline," "batch mode," etc. Portions of processing may be performed at different times and in different locations by different (or the same) processing systems. A computer may be any processor in communication with memory.

[0232] The memory 2204 is typically provided within the computing device 2200 for access by the processor 2202 and may be any suitable processor-readable storage medium suitable for storing instructions for execution by the processor, and located separately from and / or integral with the processor 2202, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. The memory 2204 may store software operated on the computing device 2200 by the processor 2202, including an operating system 2208, a virtual experience application 2210, a 3D avatar modification application 2212, and other applications (not shown). In some implementations, the virtual experience application 2210 and / or the 3D avatar modification application 2212 may include instructions that enable the processor 2202 to perform (or control the functions of) some or all of the functions described herein, e.g., the methods described with respect to FIGS. 18-21 .

[0233] For example, the virtual experience application 2210 may include a 3D avatar modification application 2212 that can dynamically modify a 3D avatar within an online virtual experience server (e.g., 102) as described herein. The elements of software in memory 2204 may alternatively be stored on any other suitable storage location or computer-readable medium. Additionally, memory 2204 (and / or other connected storage devices) may store instructions and data used in the features described herein. Memory 2204, as well as any other type of storage (such as magnetic disks, optical disks, magnetic tape, or other tangible media), may be considered "storage" or "storage devices."

[0234] The I / O interface 2206 may provide functionality that allows the computing device 2200 to interface with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data store 120), and input / output devices may communicate through the I / O interface 2206. In some implementations, the I / O interface may connect to interface devices including input devices (keyboards, pointing devices, touchscreens, microphones, cameras, scanners, etc.) and / or output devices (display devices, speaker devices, printers, motors, etc.).

[0235] The audio / video input / output devices 2214 may include user input devices (e.g., a mouse, etc.) that can be used to receive user input, display devices (e.g., a screen, monitor, etc.) that can be used to provide graphical and / or visual output, and / or combined input and display devices.

[0236] For ease of explanation, FIG. 22 shows one block for each of the processor 2202, memory 2204, I / O interface 2206, and software blocks of the operating system 2208, virtual experience application 2210, and 3D avatar modification application 2212. These blocks may represent one or more processors or processing circuits, operating systems, memories, I / O interfaces, applications, and / or software engines. In other implementations, the computing device 2200 may not have all of the components shown and / or may have other elements, including other types of elements instead of or in addition to those illustrated herein. Although the online virtual experience server 102 is described as performing the operations as described in some implementations herein, any suitable component or combination of components of the online virtual experience server 102 or a similar system, or any suitable processor associated with such a system, may perform the described operations.

[0237] A user device can implement and / or use with the functionality described herein. An exemplary user device can be a computer device that includes several components similar to computing device 2200, e.g., processor 2202, memory 2204, and / or I / O interface 2206. An operating system, software, and applications suitable for the client device are provided in the memory and can be used by the processor. The I / O interface for the client device can be connected to a network communication device, as well as input and output devices, e.g., a microphone for capturing sound, a camera for capturing images or video, a mouse for capturing user input, a gesture device for recognizing user gestures, a touchscreen for detecting user input, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices. A display device within audio / video input / output devices 2214 may be connected to (or included within) computing device 2200 for displaying images, for example, pre- and post-processing as described herein, and such a display device may include any suitable display device, for example, an LCD, LED, or plasma display screen, a CRT, a television, a monitor, a touchscreen, a 3-D display screen, a projector, or other visual display device. Some implementations may provide an audio output device, for example, text-to-speech output or synthesis.

[0238] One or more methods described herein (e.g., methods 1800, 1900, 2000, 2100) can be implemented by computer program instructions or code that can be executed on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and can be stored on a computer program product that includes a non-transitory computer-readable medium (e.g., storage medium), such as a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, or a semiconductor storage medium, including semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), flash memory, rigid magnetic disks, optical disks, solid-state memory drives, etc. The program instructions can also be contained within or provided as an electronic signal, for example, in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented in hardware (such as logic gates) or a combination of hardware and software. Exemplary hardware may be a programmable processor (e.g., a field programmable gate array (FPGA), complex programmable logic device), general-purpose processor, graphics processor, application-specific integrated circuit (ASIC), etc. One or more methods may be implemented as part of or a component of an application running on the system, or as an application or software running in conjunction with other applications and the operating system.

[0239] One or more methods described herein may be implemented in a standalone program that can run on any type of computing device, a program running on a web browser, or a mobile application ("app") running on a mobile computing device (e.g., a mobile phone, a smartphone, a tablet computer, a wearable device (such as a watch, armband, jewelry, hat, goggles, glasses, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, e.g., a mobile computing device (as a client) sends user input data to a server device and receives final output data for output (e.g., for display) from the server. In another example, all computations may be performed within a mobile app (and / or other apps) on the mobile computing device. In another example, computations may be split between the mobile computing device and one or more server devices.

[0240] Although the present specification has been described with reference to specific implementations thereof, these specific implementations are merely exemplary and not limiting, and concepts shown in the examples may be applied to other examples and implementations.

[0241] The functional blocks, operations, features, methods, devices, and systems described in this disclosure may be combined or divided into different combinations of systems, devices, and functional blocks, as known to those skilled in the art. Any suitable programming language and programming techniques may be used to implement the routines of a particular embodiment. Various programming techniques, e.g., procedural or object-oriented, may be used. The routines may be executed on a single processing device or multiple processors. While steps, operations, or computations may be presented in a particular order, the order may be changed in different particular implementations. In some implementations, multiple steps or operations shown as sequential herein may be performed simultaneously. [Explanation of symbols]

[0242] 100 System Architecture 102 Online Virtual Experience Server, Virtual Experience Server 104 Virtual Experience Engine, Local Virtual Experience Engine 106 Virtual Experience 108 Graphics Engine 110 client devices 110a Client Device 110b client device 110n client devices 112 Virtual Experience Application 114 Input / Output (I / O) Interface 120 data stores 122 Network 130 Developer Devices 130a Developer Device 130n Developer Device 132 Virtual Experience Applications 134 Input / Output (I / O) Interface 200 Body Cage 202 minutiae 204 segments / edge 300 Body Cage 400 Body Cage 500 clothing layer 502 polygon meshes 504 Food 506 Cuffs 508 Straight Cut Body 600 Composite Outer Cage 602 Exposed Surface 700 Body Cage 702 fuselage 704 Arms 800 Body Cage 900 Virtual Experiences 904 Current Avatar, Current / Former Avatar 906 Target Avatar 910 New Avatar 1000 Virtual Experiences 1004 Current Avatar 1006 First Target Avatar 1008 Second Target Avatar 1012 New Avatar 1100 Virtual Experience 1104 Current Avatar 1108 New Avatar 1200 Setting Environment 1202 User avatar, original avatar 1204 First Target Avatar 1206 Second Target Avatar 1208 New Avatar 1210 Adjustment tools and / or other types of transformation tools, transformation tools 1300 Virtual Experience 1304 User Avatar, Avatar, Changed / New Avatar 1306 Coat 1400 Avatars 2200 Computing Devices 2202 processor 2204 memory 2206 Input / Output (I / O) Interface, I / O Interface 2208 Operating System 2210 Virtual Experience Application 2212 3D avatar change application 2214 Audio / Video Input / Output Device

Claims

1. 1. A computer-implemented method for modifying a three-dimensional (3D) avatar body, the computer-implemented method comprising: identifying a first avatar body having a first body cage; identifying a target avatar body having a target body cage; performing interpolation between the first body cage and the target body cage to obtain a second body cage corresponding to the second avatar body to provide a transformation of the first avatar body to a second avatar body; 11. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein performing the interpolation comprises performing the interpolation to generate the second body cage that perfectly matches the target body cage to provide a complete transformation.

3. 2. The computer-implemented method of claim 1, wherein performing the interpolation includes transforming the first avatar body into the second avatar body that is a mixture between the first avatar body and the target avatar body to provide a partial transformation.

4. The computer-implemented method of claim 1 , wherein performing the interpolation comprises morphing a portion of the first avatar body that is less than the entire first avatar body.

5. 5. The computer-implemented method of claim 4, wherein morphing the portion of the first avatar body that is less than the entire first avatar body comprises morphing the portion of the first avatar body to perform a partial transformation of the portion of the first avatar body.

6. 2. The computer-implemented method of claim 1, wherein the first avatar body is part of a virtual experience, the step of performing the interpolation is performed while the avatar is participating in the virtual experience, and the target avatar body is selected from a plurality of target avatar bodies in the virtual experience.

7. The computer-implemented method of claim 1 , wherein the step of performing interpolation is performed in a configured environment, and the target avatar body is selected from a plurality of target avatar bodies in a library within the configured environment.

8. 8. The computer-implemented method of claim 7, wherein the configuration environment includes a transformation tool that allows a user to control the amount of transformation of the first avatar body to obtain the second avatar body, and wherein performing the interpolation is based on the amount of transformation.

9. identifying a rig for the first avatar body, the rig including identifying a skeleton for the first avatar body and skinning for the first avatar body; After performing the interpolation, updating the rig of the first avatar body to correspond to the second body cage; animating the first avatar body by moving a skeleton of the updated rig and deforming skinning of the updated rig; The computer-implemented method of claim 1 further comprising:

10. 10. The computer-implemented method of claim 9, wherein moving the skeleton of the updated rig and deforming the skinning of the updated rig includes reusing skin weights from the skinning of the first avatar body based on determining areas of the skinning of the updated rig that are influenced by bones in the skeleton of the first avatar body.

11. 1. A computer-implemented method for modifying a three-dimensional (3D) avatar body, the computer-implemented method comprising: identifying a first avatar body having a corresponding first body cage; performing a manipulation of the first body cage to generate a second body cage corresponding to the second avatar body to provide a transformation of the first avatar body into a second avatar body; wherein the manipulating includes rearranging portions of the first body cage.

12. 12. The computer-implemented method of claim 11, wherein the step of performing the operation is performed in a configuration environment, the configuration environment including a transformation tool that enables a user to control aspects of the operation of the first body cage to obtain the second body cage, and the step of performing the operation is based on the aspects of the operation.

13. identifying a rig for the first avatar body, the rig including identifying a skeleton for the first avatar body and skinning for the first avatar body; after performing the manipulation, updating the rig of the first avatar body to correspond to the second body cage; animating the first avatar body by moving a skeleton of the updated rig and deforming skinning of the updated rig; The computer-implemented method of claim 11 , further comprising:

14. 14. The computer-implemented method of claim 13, wherein moving the skeleton of the updated rig and deforming the skinning of the updated rig comprises reusing skin weights from the skinning of the first avatar body based on determining areas of the skinning of the updated rig that are influenced by bones in the skeleton of the first avatar body.

15. 14. The computer-implemented method of claim 13, wherein transforming the first avatar body to the second avatar body includes performing an interpolation between the first body cage and the second body cage.

16. 14. The computer-implemented method of claim 13, wherein transforming the first avatar body into the second avatar body comprises morphing a portion of the first avatar body that is less than the entire first avatar body.

17. a memory in which instructions are stored; a processing device coupled to the memory, the processing device configured to access the memory and execute the instructions; and wherein the instructions cause the processing device to: identifying a first avatar body having a first body cage; identifying a target avatar body having a target body cage; performing interpolation between the first body cage and the target body cage to obtain a second body cage corresponding to the second avatar body to provide a transformation of the first avatar body to a second avatar body; A system that causes an operation including

18. 18. The system of claim 17, wherein performing the interpolation includes performing the interpolation to generate the second body cage that perfectly matches the target body cage to provide a perfect transformation.

19. 18. The system of claim 17, wherein performing the interpolation includes transforming the first avatar body into the second avatar body that is a mixture between the first avatar body and the target avatar body to provide a partial transformation.

20. 18. The system of claim 17, wherein performing the interpolation includes morphing a portion of the first avatar body that is less than the entire first avatar body.

Images