Dynamically changing avatar body in virtual experience

Abstract

Some embodiments relate to methods, systems, and computer-readable media for dynamically changing an avatar body during runtime during which an avatar associated with the avatar body participates in a virtual experience. In some embodiments, the method includes identifying a first avatar body having a first body cage, identifying a target avatar body having a target body cage, and performing interpolation between the first body cage and the target body cage to obtain a second body cage corresponding to a second avatar body, thereby providing a transformation of the first avatar body to the second avatar body. The body of the avatar can also be changed in the configuration environment. Altering the avatar body may involve interpolating between the body cages of the pair of cages, or by directly manipulating the body cages of the avatar body.

Description

Cross - Reference to Related Applications

[0001] This application claims the priority of U.S. Provisional Application No. 63 / 532,556, filed on August 14, 2023, with the title "DYNAMICALLY CHANGING AVATAR BODIES IN A VIRTUAL EXPERIENCE", the content of which is incorporated herein in its entirety. Technical Field

[0002] The present disclosure generally relates to computer graphics, and more specifically but not exclusively, to methods, systems, and computer - readable media for dynamically changing an avatar body (including representations of clothing worn by the avatar body) in a three - dimensional (3D) virtual environment. Background Art

[0003] Multi - user video games or other types of virtual experience environments may involve the use of avatars, where an avatar represents a user in the virtual experience. Different three - dimensional (3D) avatars vary in geometric form / shape. For example, an avatar can have different body shapes (e.g., tall, short, muscular, thin, etc.), can have different types (e.g., male, female, human, animal, alien, etc.), and can have any number and type of limbs, etc. An avatar can be customized in terms of multiple pieces of clothing and / or accessories worn (e.g., a shirt worn on the torso, a jacket worn over the shirt, a scarf worn over the jacket, a hat worn on the head, etc.).

[0004] When a user wishes to change some visual aspects of their respective avatar body and / or the clothing (including accessories) worn by the avatar body while participating in a virtual experience or other type of 3D environment, it is difficult to obtain satisfactory results in a computationally efficient manner.

[0005] Some embodiments are envisioned based on the above situation.

[0006] The background art description provided herein is intended to introduce the background of the present disclosure. The work done by the current inventors, insofar as it is described in this background section, and aspects of the specification that may not constitute prior art at the time of filing, whether explicitly or implicitly, shall not be regarded as prior art to the present disclosure. Summary of the Invention

[0007] Embodiments of the present disclosure relate to techniques for dynamically changing visual aspects of a user avatar (e.g., the visual appearance of an avatar associated with a user when the user participates in a virtual experience). The entire (original) avatar body can be changed or otherwise transformed as a whole into a new (different) avatar body, or only selectively change regions / parts of the original avatar body (e.g., only the head or other body parts), while other regions / parts of the original avatar body remain unchanged. Various techniques also provide specific methods for achieving aspects of dynamic changes (such as skinning deformation and facial action coding system (FACS) pose deformation) near real-time by using techniques that effectively manage computing resources.

[0008] A system of one or more computers can be used to perform specific operations or actions because software, firmware, hardware, or a combination thereof is installed on the system, causing the system to perform actions during operation. One or more computer programs can be used to perform specific operations or actions because the computer programs include instructions that cause the device to perform actions when executed by a data processing device.

[0009] According to one aspect, a computer-implemented method for modifying a three-dimensional (3D) avatar body is provided. The computer-implemented method includes: identifying a first avatar body having a first body cage; identifying a target avatar body having a target body cage; and performing interpolation between the first body cage and the target body cage to obtain a second body cage corresponding to a second avatar body, thereby providing a transformation from the first avatar body to the second avatar body.

[0010] Various embodiments of computer-implemented methods are described herein.

[0011] In some embodiments, performing interpolation includes performing interpolation to generate a second body cage that exactly matches the target body cage, thereby providing a complete transformation.

[0012] In some embodiments, performing interpolation includes transforming the first avatar body into a second avatar body that is a blend between the first avatar body and the target avatar body, thereby providing a partial transformation.

[0013] In some embodiments, performing interpolation includes deforming a part of the first avatar body that is less than the whole of the first avatar body.

[0014] In some embodiments, deforming a part of the first avatar body that is less than the whole of the first avatar body includes deforming a part of the first avatar body to perform a partial transformation of that part of the first avatar body.

[0015] In some embodiments, the first avatar body is part of a virtual experience, interpolation is performed when the avatar participates in the virtual experience, and a target avatar body is selected from a plurality of target avatar bodies in the virtual experience.

[0016] In some embodiments, interpolation is performed in a configured environment, and a target avatar body is selected from a plurality of target avatar bodies in a library in the configured environment.

[0017] In some embodiments, the configured environment includes a transformation tool that enables a user to control the amount of transformation of the first avatar body to obtain a second avatar body, and the interpolation is performed based on the amount of transformation.

[0018] In some embodiments, the computer-implemented method further includes identifying bindings of the first avatar body, the bindings including identifying a skeleton of the first avatar body and a skin of the first avatar body; after performing the interpolation, updating the bindings of the first avatar body to correspond to a second body cage; and animating the first avatar body by moving the skeleton of the updated bindings and deforming the skin of the updated bindings.

[0019] In some embodiments, moving the skeleton of the updated bindings and deforming the skin of the updated bindings includes: reusing skin weights from the skin of the first avatar body based on determining regions of the skin of the updated bindings that are affected by bones in the skeleton of the first avatar body.

[0020] According to another aspect, there is provided a computer-implemented method for modifying a three-dimensional (3D) avatar body, the computer-implemented method including: identifying a first avatar body having a corresponding first body cage; and performing an operation on the first body cage to generate a second body cage corresponding to a second avatar body, thereby providing a transformation from the first avatar body to the second avatar body, wherein the operation includes repositioning portions of the first body cage.

[0021] Various embodiments of computer-implemented methods are described herein.

[0022] In some embodiments, the operation is performed in a configured environment, and wherein the configured environment includes a transformation tool that enables a user to control aspects of the operation on the first body cage to obtain a second body cage, and wherein the operation is performed based on the aspects of the operation.

[0023] In some embodiments, the computer-implemented method further includes identifying bindings of the first avatar body, the bindings including identifying a skeleton of the first avatar body and a skin of the first avatar body; after performing the operation, updating the bindings of the first avatar body to correspond to the second body cage; and animating the first avatar body by moving the skeleton of the updated bindings and deforming the skin of the updated bindings.

[0024] In some embodiments, the computer-implemented method further includes moving the updated bound skeleton and deforming the updated bound skin, including: based on determining a region of the updated bound skin that is affected by bones in the skeleton of the first avatar body, reusing skin weights from the skin of the first avatar body.

[0025] In some embodiments, transforming the first avatar body into the second avatar body includes performing interpolation between a first body cage and a second body cage.

[0026] In some embodiments, transforming the first avatar body into the second avatar body includes deforming a portion of the first avatar body that is less than the whole of the first avatar body.

[0027] According to another aspect, a system is disclosed, including: a memory storing instructions thereon; and a processing device coupled to the memory, the processing device configured to access the memory, wherein, when executed by the processing device, the instructions cause the processing device to perform operations including: identifying a first avatar body having a first body cage; identifying a target avatar body having a target body cage; and performing interpolation between the first body cage and the target body cage to obtain a second body cage corresponding to a second avatar body, thereby providing a transformation of the first avatar body to the second avatar body.

[0028] Various embodiments of the system are described herein.

[0029] In some embodiments, performing interpolation includes performing interpolation to generate a second body cage that exactly matches the target body cage, thereby providing a complete transformation.

[0030] In some embodiments, performing interpolation includes transforming the first avatar body into a second avatar body that is a blend between the first avatar body and the target avatar body, thereby providing a partial transformation.

[0031] In some embodiments, performing interpolation includes deforming a portion of the first avatar body that is less than the whole of the first avatar body.

[0032] According to yet another aspect, portions, features, and implementation details of the system, method, and non-transitory computer-readable medium can be combined to form other aspects, including but not limited to some aspects that omit and / or modify some or part of each component or feature, include additional components or features, and / or other modifications, all of which modifications are within the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a diagram of an example system architecture of a 3D environment platform including a 3D avatar that can support clothing adapted thereon, according to some embodiments.

[0034] Figure 2 Shows an example body cage according to some embodiments.

[0035] Figure 3 Shows another example body cage according to some embodiments.

[0036] Figure 4 Shows an example of parts of a body cage grouped into corresponding body parts according to some embodiments.

[0037] Figure 5 Shows an example of a clothing layer deformed on a body cage according to some embodiments.

[0038] Figure 6 Shows an example of an outer cage formed based on Figure 5 parts of a clothing layer and a body cage according to some embodiments.

[0039] Figure 7 Shows an example of an interpolation between two body cages to obtain a new body cage according to some embodiments.

[0040] Figure 8 Shows an example of generating a new body cage according to some embodiments.

[0041] Figure 9 Shows an example of the transformation of an avatar body during a virtual experience according to some embodiments.

[0042] Figure 10 Shows another example of the transformation of an avatar body during a virtual experience according to some embodiments.

[0043] Figure 11 Shows another example of the transformation of an avatar body during a virtual experience according to some embodiments.

[0044] Figure 12 Shows an example of the transformation of an avatar in a configured environment according to some embodiments.

[0045] Figure 13 Shows an example of a layered clothing of an avatar body in a virtual experience according to some embodiments.

[0046] Figures 14 to 17 Shows an example of the transformation and animation of an avatar according to some embodiments.

[0047] Figure 18 Is a flowchart showing a computer-implemented method for changing an avatar body according to some embodiments.

[0048] Figure 19A flowchart showing another computer - implemented method for changing a three - dimensional (3D) avatar body according to some embodiments.

[0049] Figure 20 A flowchart showing a computer - implemented method for performing skinning deformation according to some embodiments.

[0050] Figure 21 A flowchart showing a computer - implemented method for performing Facial Action Coding System (FACS) pose deformation.

[0051] Figure 22 A block diagram showing an example computing device according to some embodiments. Detailed Description

[0052] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, like symbols typically identify like components unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject presented herein. Aspects of the present disclosure, as generally described herein and illustrated in the drawings, can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are within the scope of this consideration.

[0053] References in the specification to "one embodiment", "an embodiment", "an example embodiment", etc., mean that the described embodiment may include a particular feature, structure, or characteristic, but each embodiment may not necessarily include that particular feature, structure, or characteristic. Moreover, these phrases do not necessarily refer to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, whether or not explicitly described, such feature, structure, or characteristic can be implemented in connection with other embodiments.

[0054] The present disclosure describes techniques for dynamically changing the visual aspects of a user's avatar (e.g., the visual appearance of the avatar associated with the user when the user participates in a virtual experience). For example, when participating in a virtual experience, a user can change the avatar from a humanoid avatar body to an animal (non - humanoid) avatar body or other different avatar bodies. The entire (original) avatar body can be changed or otherwise transformed as a whole into a new (different) avatar body, or only selectively change regions / parts of the original avatar body (e.g., only the head or other body parts), while other regions / parts of the original avatar body remain unchanged.

[0055] The transformation from the original avatar body to the target avatar body can be a full transformation, in which the original (first) avatar body is completely transformed into the target avatar body, which becomes the new (second) avatar body. The transformation can also be a partial transformation, in which the new (second) avatar body is a mixture or other type of blending between the original (first) avatar body and the target avatar body.

[0056] According to various embodiments, a new avatar body may be obtained by performing interpolation between the body cage of an original (first) avatar body and the body cage of a target avatar body, such that the obtained new avatar body has its own body cage interpolated / generated from the body cages of the original avatar body and the target avatar body.

[0057] Dynamic editing / changing of the avatar body can be performed when there is no clothing on the avatar, and / or can be performed when there is one or more layers of clothing on the avatar body. When there is clothing on the avatar body, dynamically changing the avatar body (e.g., changing the shape of the body) will also cause a corresponding change in the one or more layers of clothing worn by the avatar.

[0058] For example, the original avatar may be a humanoid avatar wearing a baseball cap, so that the baseball cap has a rounded appearance. If the user changes the head of the humanoid avatar to an alien head (e.g., an alien avatar with a conical head), the baseball cap may also dynamically deform accordingly, so as to change from the original rounded appearance to a more pointed appearance that matches the conical head of the alien (target) avatar. The clothing (including accessories) of the dynamically deformed avatar may also be cage-based, as will be explained below.

[0059] According to various embodiments, during runtime while participating in a virtual experience, a user may select a target avatar body and its clothing. For example, the user may select a target avatar body by selecting (e.g., clicking) another avatar in the virtual experience, by selecting the target avatar body from a library, by directly manipulating the original (current) avatar body (e.g., by changing the body cage) without selecting the target avatar body in the virtual experience or from a library, etc. Adjustment tools such as sliders may be provided on the user interface so that the user can control the amount of transformation between the two avatar bodies.

[0060] In some embodiments, adjustment tools and / or some other transformation tools can be used to dynamically change / transform the current avatar body in a direct manner without involving interpolation between two avatar bodies. That is, and by way of example, a user can use a transformation tool to change the shape of the avatar's head from a humanoid head to a geometric head (e.g., blocky) - in some embodiments, such a transformation can be performed by directly changing (e.g., moving or otherwise operating on) the line segments and vertices of the cage of the humanoid head. The application of this technique may neither require the presence or use of a body cage of a geometric head as a reference (target) nor the performance of interpolation between such a reference body cage and the body cage of the humanoid head. Thus, this technique can be regarded as a "free form" method of independently changing the appearance of an avatar, where the change in appearance is independent of any other avatar.

[0061] The various techniques described herein for dynamically changing an avatar body (with or without a clothing layer thereon) can be applied to avatars used in virtual experiences. Such virtual experiences are sometimes described herein in the context of video games. It should be noted that describing these embodiments in the context of video games is for the convenience of providing examples and illustrations only.

[0062] The techniques described herein can be used in other types of virtual experiences in a three-dimensional (3D) environment, which do not necessarily involve video games with one or more players represented by avatars. Examples of virtual experiences can include virtual reality (VR) meetings, 3D sessions (e.g., online lectures or other types of presentations involving 3D avatars), augmented reality (AR) sessions, or other types of 3D environments in which one or more users are represented by one or more 3D avatars in a 3D environment.

[0063] For layered clothing, an automatic cage-to-cage fitting technique can be used for 3D avatars. This technique allows for the fitting of any body geometry to any clothing geometry, including fitting each layer of clothing to the underlying layer of clothing, thus providing customization without being restricted by predefined geometries and without the need for complex calculations to make the clothing compatible with any body shape of a virtual avatar or with other clothing items.

[0064] Cage-to-cage adaptation is also performed using various techniques employed by a game platform or game software (or other virtual experience platforms / software for providing 3D environments), without requiring complex calculations to be performed by an avatar creator (also referred to as an avatar body creator or body creator) or a clothing item creator. As used herein, the term "clothing" or "a piece of clothing" or other similar terms shall be understood to include graphical representations of clothing and accessories, as well as any other items that can be placed on an avatar and are related to specific parts of the avatar cage.

[0065] During the runtime of a game or other virtual experience session, a player / user accesses a body library to select a specific avatar body and accesses a clothing library to select clothing to be placed on the selected body. The 3D virtual environment platform presenting the avatar employs cage-to-cage adaptation techniques (by automatically determining appropriate deformations) to adjust a piece of clothing to fit the shape of the body, thereby automatically fitting the piece of clothing onto the body (and any intermediate layers that the avatar may be wearing).

[0066] When a piece of clothing is fitted onto a virtual avatar's body and / or underlying clothing, the techniques described herein can be performed to make the piece of clothing fit the avatar more precisely in terms of dimensions (e.g., coordination), shape, etc. by deforming or adapting. The user can further select additional clothing to be fitted onto the underlying clothing, where the additional clothing is deformed to match the geometry of the underlying clothing.

[0067] The various embodiments described herein are based on the concepts of "cage" and "mesh". The body mesh (or rendering mesh) is the actual visible geometry of the avatar. The body mesh includes graphical representations of body parts such as arms, legs, torso, head, etc., and can have any shape, size, and geometric topology. Similarly, a clothing mesh (or rendering mesh) can be any mesh that graphically represents a piece of clothing (such as a shirt, pants, hat, shoes, etc.) or a part thereof.

[0068] In contrast, a cage represents an envelope around the feature points of an avatar body, which is simpler than the body mesh and has a weaker correspondence with the corresponding vertices of the body mesh. As will be elaborated further below, a cage can be used not only to represent a set of feature points on an avatar body but also to represent a set of feature points on a piece of clothing.

[0069] In some embodiments, there is a dynamic body part modification mechanism implemented by extending the layering clothing framework of a layering clothing system, allowing the use of a user-specified cage to deform an avatar body part. This core functionality can be applied to at least two techniques. First, there can be a plugin for studio applications that enables a user to change the overall body shape of one avatar based on the overall body shape of another avatar, for example, by toggling / adjusting a slider that interpolates between two body shapes. Second, there can be a virtual experience where a player can gradually update the avatar body by clicking on other avatar body parts in the virtual experience.

[0070] Some existing techniques for dynamically changing an avatar can include linear blend skinning (LBS), Facial Action Coding System (FACS), and affine skinning techniques. These techniques form the basis of current methods and include skinning deformation and Facial Action Coding System (FACS) pose deformation.

[0071] In linear blend skinning (LBS), the i-th deformed position p′ is calculated in the vertex shader by computing the i-th deformed position p′ i . In this equation, the skinning weight w i,j and the vertex bind position p i are constants and cached in the GPU. If fewer than 4 bones are used, w i,j can be 0. The 3x4 bone transformation M is calculated on the CPU as follows j and copied to the GPU frame by frame:

[0072] The inverse bind transformation is a cached constant, while the pose transformation P j is updated every frame. The global transformation B j and P j are calculated from the local transformations LB j and LP j of a hierarchy (called the skeleton) as follows: B j = B j′ · LB, P j = P j′ · LB j · LP j , where j′ is the index of the parent or root part of the j-th node. The skeleton hierarchy and transformations are provided by other engine systems such as a physics engine for the body or FACS for the dynamic head. These engine systems are various sources for animating the avatar into poses.

[0073] The FACS system provides LBS data for the dynamic head mesh. Skinning weights w, skeleton hierarchy j′, and local bind transform LB j From a content delivery network (CDN) mesh data structure with mesh and controls-to-joint-driver data, where the controls-to-joint-driver data is a mapping of a given mesh from FACS control values to joint positions and rotations. Local pose transform LP j Each frame consists of a 3x3 rotation matrix R j,k and a translation vector T j,k : LP j = R j,k | T j,k , where Euler angles r j,k and translation vector t j,k also come from the ControlToJointDriver structure of the mesh data structure, which defines the mapping of the mesh that converts FACS control values into the joint positions and rotations of the skinning. The data structure can be an MxN matrix, where M is the number of FACS data channels and N is the number of joint transformation values. The matrix is used to transform FACS controls into transformation values for driving the facial bone joints.

[0074] The rotation is interpolated in Euler coordinates, so R j,k is rigid. Calculate 17 <= n <= 50 shape weights s ′ as follows: s′ = applyCorrectives(s, C), where the applyCorrectives function extends 17 original FACS pose weights s to up to 50 shape weights s′ based on the CustomCorrections parameter C, which also comes from the ControlToJointDriver structure. For selected combinations of 2 or 3 original FACS weights, the default linear average shape can be replaced with a custom shape constructed for that combination. This is used to improve artistic control and deformation quality.

[0075] Previously, both the bind transform and the pose transform were of the rigid CorrdinateFrame type. A rigid transform (also known as an Euclidean transform) is a geometric transform of Euclidean space that preserves the Euclidean distance between every pair of points. This method was applicable because each data model (DM) and FACS transform was rigid. Knowing this, the rotation can be transposed when calculating its inverse. For example, during the physical update process, when transform inversion is a key issue, this can be a significant optimization. However, for LBS, the only inversion Cached as a constant, so this optimization does not provide a significant gain for this technology.

[0076] Approximate elastic skinning is a common use case for LBS, but the skin does not deform rigidly. Elastic deformation is typically approximated by rigid bone transforms, which is natural when the deformation is driven by rigid bones. However, facial muscles are not rigid. Facial muscles can undergo shear and non-uniform scaling, so there is no physical reason to restrict facial "bones" to rigid transforms. Since affine transforms are a superset of rigid transforms, affine transforms can be used without changing the existing bindings, while increasing flexibility for future bindings.

[0077] The weight maps of the dynamic head divide the face into regions, providing the necessary degrees of freedom for 17 to 50 poses. The pose transforms of the dynamic head articulate these regions in response to 17 FACS controls. If the shape of the dynamic head changes and good visual effects need to be maintained, the bone transforms may need to change. If the shape change does not change the semantics of the vertices, i.e., cheek vertices do not become nose vertices, etc., the skinning weights can usually be reused.

[0078] If the skinning weights are reused, the surface area affected by each bone can be represented. It can also be noted that the pose transforms animate the direction and amount related to the surface shape of the region affected by the pose. This influence includes not only its own weight but also the weights of any descendant bones in the bone hierarchy. For example, the influence of a head joint includes the influence of a lip joint, etc., because when the neck rotates, the lip joint rotates with the neck. Additionally, since the vertices affect the shape of the connected faces, these influence sets extend along an edge. Different from LBS, these influence weight sets are not normalized.

[0079] For each bone, the skin deformation can include using some linear algebra to calculate a 3x4 affine transform that best fits the deformed point cloud of its affected points. This correction is applied to the bone transform in the skinning calculation: where the correction is matrix M j . This method may be effective, but these techniques do not intend to change the skinning process but rather aim to update the bindings. Therefore, D is bubbled through the skeleton calculation / , and D / is merged into the new local binding transform LB' j which generates the same result: LB' j = B j -1 ·D j ·B j′ ·LB j . For example, this can be a simple algebraic operation to solve for the new local binding transform LB'j , the new local binding transformation produces the same result as the previous formula, but without the M j term.

[0080] Note that after this, LB′ j is no longer a rigid transformation. Therefore, for any data model skeleton, the result can be orthonormalized, but for the dynamic head, the full affine transformation is retained and the result is significantly improved. This makes the result depend on the affine skinning changes discussed above.

[0081] Regarding FACS pose deformation, this FACS binding contains many poses, and each pose includes the local transformation P j of each joint in the head binding. The skinning deformation technology updates their binding transformation LB′ j , thus changing the parent space of these local pose transformations.

[0082] The affine correction transformation D j changes the translation of the resulting transformation M j , including its direction and magnitude, but the affine correction transformation D j does not change the rotation of the resulting transformation M j . To further improve the result, the rotation r j,k of each joint for each pose will also be updated. During this process, the translation t j,k of each joint for each pose can also be further fine-tuned.

[0083] In some embodiments, the fine-tuning of each pose can be performed one at a time. First, the original LBS deformed mesh pose is calculated based on the original head shape. For each joint in the pose, its skinned weighted points are projected onto the nearest point on the pose grid, thus providing a new set of points. These points represent the positions on the original grid that are closest to the "destination" of that joint in that pose. Then the same 3x4 affine adaptation function can be reused to calculate how these points transform between the original shape and the modified shape.

[0084] In some embodiments, the rigid components of the transformation are extracted and the final translation and rotation of the pose joints are fine-tuned. To minimize the change in Euler interpolation, appropriate changes are made to the Euler angles that match the rotation. For example, an Euler rotation can represent the same rotation matrix in countless ways, so appropriate techniques are used to change the original angles as little as possible. These angles are calculated by decomposing the rotation matrix and gradually inserting corrections. Then Then x ′ = x1 + x2, y ′ = y1 + y2, and z ′ = z1 + z2. Figure 1 - System architecture

[0085] Figure 1 FIG. is a diagram of an exemplary system architecture of a 3D environment platform according to some embodiments that can support a 3D avatar with clothing adapted thereon. Figure 1 Like reference numerals are used in the other drawings to identify like elements. Characters following a reference numeral, such as "110", indicate the element in the text that specifically refers to the element with that particular reference numeral. A reference numeral without a character in the text (e.g., "110") refers to any or all of the elements in the drawing with that reference numeral (e.g., "110" in the text refers to reference numerals "110a", "110b", and / or "110n" in the drawing).

[0086] System architecture 100 (also referred to herein as "the system") includes an online virtual experience server 102, a data storage area 120, client devices 110a, 110b, and 110n (generally referred to herein as "client devices 110"), and developer devices 130a and 130n (generally referred to herein as "developer devices 130"). The virtual experience server 102, the data storage area 120, the client devices 110, and the developer devices 130 are coupled via a network 122. In some embodiments, the client devices 110 and the developer devices 130 may refer to the same or the same type of device.

[0087] In addition, the online virtual experience server 102 may include a virtual experience engine 104, one or more virtual experiences 106, a graphics engine 108, etc. In some embodiments, the graphics engine 108 may be a system, application, or module that allows the online virtual experience server 102 to provide graphics and animation capabilities. In some embodiments, the graphics engine 108 and / or the virtual experience engine 104 may perform one or more operations described in connection with the flowchart shown in Figures 18 to 21 The client device 110 may include a virtual experience application 112 and an input / output (I / O) interface 114 (e.g., an input / output device). The input / output device may include one or more of a microphone, a speaker, headphones, a display device, a mouse, a keyboard, a game controller, a touch screen, a virtual reality console, etc.

[0088] The developer device 130 may include a virtual experience application 132 and an input / output (I / O) interface 134 (e.g., an input / output device). The input / output device may include one or more of a microphone, a speaker, headphones, a display device, a mouse, a keyboard, a game controller, a touch screen, a virtual reality console, etc.

[0089] A system architecture 100 is provided for illustration. In different embodiments, the system architecture 100 may include the same, fewer, more, or different components configured in the same or different ways as Figure 1 shown.

[0090] In some embodiments, the network 122 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., Ethernet), a wireless network (e.g., an 802.11 network, a network, or a wireless LAN (WLAN)), a cellular network (e.g., a 5G network, a long term evolution (LTE) network, etc.), routers, hubs, switches, server computers, or a combination thereof.

[0091] In some embodiments, the data store 120 may be a non-transitory computer-readable memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data. The data store 120 may also include multiple storage components (e.g., multiple drives or multiple databases), which may also span multiple computing devices (e.g., multiple server computers). In some embodiments, the data store 120 may include cloud-based storage.

[0092] In some embodiments, the online virtual experience server 102 may include a server having one or more computing devices (e.g., a cloud computing system, a rack server, a server computer, a physical server cluster, etc.). In some embodiments, the online virtual experience server 102 may be a stand-alone system, may include multiple servers, or may be a part of another system or server.

[0093] In some embodiments, the online virtual experience server 102 may include one or more computing devices (e.g., rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc.), data storage areas (e.g., hard disks, memories, databases), networks, software components, and / or hardware components, which can be used to perform operations on the online virtual experience server 102 and provide users with access to the online virtual experience server 102. The online virtual experience server 102 may also include a website (e.g., a web page) or application backend software that can be used to provide users with access rights to the content provided by the online virtual experience server 102. For example, a user can use the virtual experience application 112 on the client device 110 to access the online virtual experience server 102.

[0094] In some embodiments, the virtual experience session data is generated by the online virtual experience server 102, the virtual experience application 112, and / or the virtual experience application 132 and stored in the data storage area 120. With the permission of the virtual experience participants, the virtual experience session data may include relevant metadata, such as a virtual experience identifier; device data associated with the participant; demographic information of the participant; virtual experience session identifier; chat records; session start time, session end time, and session duration of each participant; relative positions of the participant avatars within the virtual experience environment; purchases made by one or more participants in the virtual experience; accessories used by the participant, etc.

[0095] In some embodiments, the online virtual experience server 102 may be a social network that provides connections between users, or a user-generated content system that allows users (e.g., end users or consumers) to communicate with other users on the online virtual experience server 102, where the communication may include voice chat (e.g., synchronous and / or asynchronous voice communication), video chat (e.g., synchronous and / or asynchronous video communication), or text chat (e.g., 1:1 and / or N:N synchronous and / or asynchronous text-based communication). Records of some or all user communications may be stored in the data storage area 120 or the virtual experience 106. The data storage area 120 can be used to store chat records (text, audio, images, etc.) exchanged between participants, subject to the appropriate permission of the players and compliance with applicable regulations.

[0096] In some embodiments, the chat record is generated by virtual experience application 112 and / or virtual experience application 132 and stored in data storage area 120. The chat record may include chat content and related metadata, for example, the chat text content of each message with the corresponding sender and recipient; the message format (e.g., bold, italic, loud, etc.); the message timestamp; the relative positions of the participant avatars within the virtual experience environment, accessories used by the virtual experience participants, etc. In some embodiments, the chat record may include content in multiple languages, and messages in different languages for different sessions of the virtual experience may be stored in data storage area 120.

[0097] In some embodiments, the chat record may be stored in the form of a conversation between participants based on the timestamp. In some embodiments, the chat record may be stored based on the originator of the message.

[0098] In some embodiments of the present disclosure, a "user" may represent a single individual. The "users" covered by other embodiments of the present disclosure (e.g., creative users) are entities controlled by a group of users or an automated source. For example, a group of individual users united as a community or group in a user-generated content system may be considered a "user".

[0099] In some embodiments, the online virtual experience server 102 may be a virtual game server. For example, the game server may provide single-player or multi-player games to a user community that may access (referred to herein as "the system") including the online virtual experience server 102, the data storage area 120, the client, or interact with the virtual experience using the client device 110 via the network 122. In some embodiments, the virtual experience (including virtual realms or worlds, virtual games, other computer-simulated environments) may be, for example, a two-dimensional (2D) virtual experience, a three-dimensional (3D) virtual experience (e.g., a 3D user-generated virtual experience), a virtual reality (VR) experience, or an augmented reality (AR) experience. In some embodiments, users may participate in interactions (e.g., games) with other users. In some embodiments, the virtual experience may be experienced in real time with other users of the virtual experience.

[0100] In some embodiments, virtual experience engagement may refer to one or more participants using a client device (e.g., 110) to interact within a virtual experience (e.g., 106), or presenting the interaction on a display or other output device (e.g., 114) of the client device 110. For example, virtual experience engagement may include interacting with one or more participants within the virtual experience or presenting the interaction on the display of the client device.

[0101] In some embodiments, the virtual experience 106 may include an electronic file that may be executed or loaded using software, firmware, or hardware for presenting virtual experience content (e.g., digital media items) to an entity. In some embodiments, the virtual experience application 112 may be executed and combined with the virtual experience engine 104 to render the virtual experience 106. In some embodiments, the virtual experience 106 may have a set of common rules or common goals, and the environments of the virtual experience 106 share this set of common rules or common goals. In some embodiments, different virtual experiences may have rules or goals that are different from each other.

[0102] In some embodiments, a virtual experience may have one or more environments (also referred to herein as "virtual experience environments" or "virtual environments"), where multiple environments may be connected. Examples of environments may be three-dimensional (3D) environments. One or more environments of the virtual experience 106 may be collectively referred to herein as a "world", "virtual experience world", "game world", "virtual world", or "universe". An example of a world may be the 3D world of the virtual experience 106. For example, a user may construct a virtual environment that may be connected to another virtual environment created by another user. Characters of the virtual experience may cross virtual boundaries and enter adjacent virtual environments.

[0103] It can be noted that the graphics used in a 3D environment or 3D world use a three-dimensional representation of the geometric data representing the virtual experience content (or at least display the virtual experience content as 3D content regardless of whether a 3D representation of geometric data is used). The graphics used in a 2D environment or 2D world use a two-dimensional representation of the geometric data representing the virtual experience content.

[0104] In some embodiments, the online virtual experience server 102 may host one or more virtual experiences 106 and may allow users to interact with the virtual experiences 106 using the virtual experience application 112 of the client device 110. Users of the online virtual experience server 102 may play the virtual experiences 106, create the virtual experiences 106, interact with the virtual experiences 106, or construct the virtual experiences 106, communicate with other users, and / or create and construct objects of the virtual experiences 106 (e.g., also referred to herein as "items", "virtual experience objects", or "virtual experience items").

[0105] For example, when generating user-generated virtual projects, a user can create characters, decorations for the characters, one or more virtual environments for an interactive virtual experience, or structures used in building the virtual experience 106, etc. In some embodiments, a user can buy, sell, or trade virtual experience objects, such as in-platform currency (e.g., virtual currency), with other users of the online virtual experience server 102. In some embodiments, the online virtual experience server 102 can send virtual experience content to a virtual experience application (e.g., 112). In some embodiments, virtual experience content (also referred to herein as "content") can refer to any data or software instructions associated with the online virtual experience server 102 or the virtual experience application (e.g., virtual experience objects, virtual experiences, user information, videos, images, commands, media items, etc.). In some embodiments, a virtual experience object (e.g., also referred to herein as an "item", "object", "virtual object", or "virtual experience item") can refer to an object used, created, shared, or otherwise depicted in the virtual experience 106 of the online virtual experience server 102 or the virtual experience application 112 of the client device 110. For example, virtual experience objects can include parts, models, characters, accessories, tools, weapons, clothing, buildings, vehicles, currency, flora, fauna, components of the above objects (e.g., windows of a building), etc.

[0106] It can be noted that, for illustration purposes, an online virtual experience server 102 that hosts the virtual experience 106 is provided. In some embodiments, the online virtual experience server 102 can host one or more media items, and the above media items can include communication messages from one user to one or more other users. With the user's permission and explicit user consent, the online virtual experience server 102 can analyze chat record data to improve the virtual experience platform. Media items can include, but are not limited to, digital videos, digital movies, digital photos, digital music, audio content, melodies, website content, social media updates, e-books, e-magazines, digital newspapers, digital audiobooks, e-journals, web blogs, real simple syndication (RSS) feeds, digital comic books, software applications, etc. In some embodiments, a media item can be an electronic file that can be executed or loaded using software, firmware, or hardware for presenting a digital media item to an entity.

[0107] In some embodiments, the virtual experience 106 may be associated with a particular user or group of users (e.g., a private virtual experience), or be widely available to users who can access the online virtual experience server 102 (e.g., a public virtual experience). In some embodiments, when the online virtual experience server 102 associates one or more virtual experiences 106 with a particular user or group of users, the online virtual experience server 102 may use user account information (e.g., a user account identifier such as a username and password) to associate the particular user with the virtual experience 106.

[0108] In some embodiments, the online virtual experience server 102 or the client device 110 may include a virtual experience engine 104 or a virtual experience application 112. In some embodiments, the virtual experience engine 104 may be used for the development or execution of the virtual experience 106. For example, the virtual experience engine 104 may include a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, a scripting function, an animation engine, an artificial intelligence engine, a network function, a streaming function, a storage management function, a threading function, a scene graph function, or animation video support, and other functions. The components of the virtual experience engine 104 may generate commands (e.g., rendering commands, collision commands, physics commands, etc.) that assist in computing and rendering the virtual experience. In some embodiments, the virtual experience application 112 of the client device 110 may work independently, in cooperation with the virtual experience engine 104 of the online virtual experience server 102, or a combination of both.

[0109] In some embodiments, both the online virtual experience server 102 and the client device 110 can execute the virtual experience engine 104 / virtual experience application 112. The online virtual experience server 102 using the virtual experience engine 104 can execute some or all of the virtual experience engine functions (e.g., generating physical commands, rendering commands, etc.), or divert some or all of the virtual experience engine functions to the virtual experience engine 104 of the client device 110. In some embodiments, the ratio between the virtual experience engine functions executed on the online virtual experience server 102 and the virtual experience engine functions executed on the client device 110 for each virtual experience 106 can be different. For example, the virtual experience engine 104 of the online virtual experience server 102 can be used to generate physical commands in the case of a collision occurring between at least two virtual experience objects, while additional virtual experience engine functions (e.g., generating rendering commands) can be diverted to the client device 110. In some embodiments, the ratio of the virtual experience engine functions executed on the online virtual experience server 102 and the client device 110 can change based on virtual experience participation conditions (e.g., dynamically). For example, if the number of users participating in a specific virtual experience 106 exceeds a threshold number, the online virtual experience server 102 can execute one or more virtual experience engine functions previously executed by the client device 110.

[0110] For example, a user can play the virtual experience 106 on the client device 110 and can send control instructions (e.g., user inputs such as right, left, up, down, user selection, or character position and speed information, etc.) to the online virtual experience server 102. After receiving the control instructions from the client device 110, the online virtual experience server 102 can send experience instructions (e.g., the position and speed information of the characters participating in the group experience, or commands such as rendering commands, collision commands, etc.) to the client device 110 based on the control instructions. For example, the online virtual experience server 102 can perform one or more logical operations on the control instructions (e.g., using the virtual experience engine 104) to generate experience instructions for the client device 110. In other instances, the online virtual experience server 102 can transfer one or more control instructions from one client device 110 to other client devices participating in the virtual experience 106 (e.g., from client device 110a to client device 110b). The client device 110 can use the experience instructions and render the virtual experience to present on the display of the client device 110.

[0111] In some embodiments, a control instruction may refer to an instruction that indicates an action of a user role in a virtual experience. For example, a control instruction may include user input that controls an action in the experience, such as right, left, up, down, user selection, gyroscopic position and orientation data, force sensor data, etc. A control instruction may include role position and velocity information. In some embodiments, the control instruction is sent directly to the online virtual experience server 102. In other embodiments, the control instruction may be sent from the client device 110 to another client device (e.g., from the client device 110b to the client device 110n), where the other client device uses the local virtual experience engine 104 to generate an experience instruction. A control instruction may include an instruction to play a voice communication message or other sound of another user on an audio device (e.g., a speaker, headphones, etc.), such as a voice communication or other sound generated using audio spatialization techniques as described herein.

[0112] In some embodiments, an experience instruction may be an instruction that enables the client device 110 to render a virtual experience (such as a multi - participant virtual experience). An experience instruction may include one or more of user input (e.g., a control instruction), role position and velocity information, or commands (e.g., physical commands, rendering commands, collision commands, etc.).

[0113] In some embodiments, a role (or generally, a virtual experience object) is composed of components, where one or more of these components can be selected by the user, and these components are automatically connected together to assist the user in editing.

[0114] In some embodiments, a role is implemented as a 3D model and includes a hierarchical collection of surface representations (also called skins or meshes) for drawing the role and interconnected bones (also called skeletons or rigs). The rig can be used to animate the role and to simulate the movement and actions of the role. The 3D model can be represented as a data structure, and one or more parameters of the data structure can be modified to change various attributes of the role, such as dimensions (height, width, perimeter, etc.); body type; movement style; number / type of body parts; proportions (e.g., shoulder - to - hip ratio); head size, etc.

[0115] One or more roles (also referred to herein as “avatars” or “models”) may be associated with a user, where the user can control the role to facilitate the user's interaction with the virtual experience 106.

[0116] In some embodiments, a character can include components such as body parts (e.g., hair, arms, legs, etc.) and accessories (e.g., t-shirts, glasses, decorative images, tools, etc.). In some embodiments, the body parts of a customizable character include head types, body part types (arms, legs, torso, and hands), face types, hair types, and skin types, etc. In some embodiments, customizable accessories include clothing (e.g., shirts, pants, hats, shoes, glasses, etc.), weapons, or other tools.

[0117] In some embodiments, for some asset types (e.g., shirts, pants, etc.), an online virtual experience platform can provide users with access to a simplified 3D virtual object model represented by a mesh with a low polygon count (e.g., between approximately 20 and 30 polygons).

[0118] In some embodiments, a user can also control the size of a character (e.g., height, width, or depth) or the size of components of the character. In some embodiments, a user can control the proportion of the character (e.g., blocky, anatomical, etc.). It can be noted that in some embodiments, a character may not include a character virtual experience object (e.g., body parts, etc.), but a user can (in the absence of a character virtual experience object) control the character to facilitate the user's interaction with the virtual experience (e.g., a puzzle game where there are no rendered character game objects, but the user still controls the character to control in-game actions).

[0119] In some embodiments, components such as body parts can be basic geometric shapes such as blocks, cylinders, spheres, etc., or some other basic shapes such as wedges, rings, tubes, channels, etc. In some embodiments, a creator module can publish a user's character for other users of the online virtual experience server 102 to view or use. In some embodiments, a user can use an I / O interface (e.g., a developer interface) and perform creation, modification, or customization of a character, other virtual experience objects, virtual experience 106, or a virtual experience environment with or without using scripts (or with or without using an application programming interface (API)). It should be noted that for illustration purposes, a character is described as having a humanoid form. It can also be noted that a character can have any form, such as a vehicle, an animal, an inanimate object, or other creative forms.

[0120] In some embodiments, the online virtual experience server 102 can store the user-created character in the data storage area 120. In some embodiments, the online virtual experience server 102 maintains a character directory and a virtual experience directory that can be presented to the user. In some embodiments, the virtual experience directory includes images of virtual experiences stored on the online virtual experience server 102. Additionally, the user can select a character (e.g., a character created by the user or another user) from the character directory to participate in the selected virtual experience. The character directory includes images of characters stored on the online virtual experience server 102. In some embodiments, one or more characters in the character directory may have been created or customized by the user. In some embodiments, the selected character can have character settings that define one or more components of the character.

[0121] In some embodiments, the user's character (e.g., an avatar) can include a configuration of components, where the configuration and appearance of the components and more generally the appearance of the character can be defined by the character settings. In some embodiments, at least part of the character settings of the user's character can be selected by the user. In other embodiments, the user can select a character with default character settings or character settings selected by other users. For example, the user can select a default character from a character directory with predefined character settings, and the user can further customize the default character by changing some character settings (e.g., adding a shirt with a custom logo). The online virtual experience server 102 can associate the character settings with a specific character.

[0122] In some embodiments, the client device 110 can include a computing device such as a personal computer (PC), a mobile device (e.g., a laptop computer, a mobile phone, a smartphone, a tablet computer, or a netbook computer), an Internet TV, a game console, etc. In some embodiments, the client device 110 can also be referred to as a "user device". In some embodiments, one or more client devices 110 can be connected to the online virtual experience server 102 at any given moment. It should be noted that the number of client devices 110 provided is for illustrative purposes. In some embodiments, any number of client devices 110 can be used.

[0123] In some embodiments, each client device 110 may respectively include an instance of the virtual experience application 112. In one embodiment, the virtual experience application 112 may allow a user to use and interact with the online virtual experience server 102, such as controlling a virtual character in a virtual experience hosted by the online virtual experience server 102, or viewing or uploading content (e.g., virtual experience 106, images, video items, web pages, documents, etc.). In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, present, or navigate content provided by a web server (e.g., virtual characters in a virtual environment, etc.). In another example, the virtual experience application may be a native application (e.g., a mobile application, app, virtual experience program, or game program) that is installed on the client device 110 and executed locally, and allows the user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content for the user (e.g., web pages, media viewers). In an embodiment, the virtual experience application may also include an embedded media player (e.g., or an HTML5 player).

[0124] According to aspects of the present disclosure, the virtual experience application may be an online virtual experience server application for a user to build, create, edit, upload content to the online virtual experience server 102, and interact with the online virtual experience server 102 (e.g., participate in the virtual experience 106 hosted by the online virtual experience server 102). Thus, the online virtual experience server 102 may provide the virtual experience application to the client device 110. In another example, the virtual experience application may be an application downloaded from a server.

[0125] In some embodiments, each developer device 130 may respectively include an instance of the virtual experience application 132. In one embodiment, the virtual experience application 132 may allow a developer user to use and interact with the online virtual experience server 102, such as controlling a virtual character in a virtual experience hosted by the online virtual experience server 102, or viewing or uploading content (e.g., virtual experience 106, images, video items, web pages, documents, etc.). In one example, the virtual experience application may be a web application (e.g., an application operating in conjunction with a web browser) that can access, retrieve, present, or navigate content provided by a web server (e.g., virtual characters in a virtual environment, etc.). In another example, the virtual experience application may be a native application (e.g., a mobile application, app, virtual experience program, or game program) that is installed on the developer device 130 and executed locally, and allows a user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content for the user (e.g., a web page, a media viewer). In an embodiment, the virtual experience application may further include an embedded media player (e.g., or an HTML5 player).

[0126] According to aspects of the present disclosure, the virtual experience application 132 may be an online virtual experience server application for a user to build, create, edit, upload content to the online virtual experience server 102, and interact with the online virtual experience server 102 (e.g., provide and / or participate in the virtual experience 106 hosted by the online virtual experience server 102). Thus, the online virtual experience server 102 may provide the virtual experience application to the developer device 130. In another example, the virtual experience application 132 may be an application downloaded from a server. The virtual experience application 132 may be used to interact with the online virtual experience server 102 and obtain access to user credentials, user currency, etc. for one or more virtual experiences 106 developed, hosted, or provided by a virtual experience developer.

[0127] In some embodiments, a user may log in to the online virtual experience server 102 through the virtual experience application. The user may access a user account by providing user account information (e.g., a username and password), where the user account is associated with one or more characters that can be used to participate in one or more virtual experiences 106 of the online virtual experience server 102. In some embodiments, with appropriate credentials, a virtual experience developer may obtain access to virtual experience virtual objects (e.g., in-platform currency (e.g., virtual currency), avatars, special abilities, accessories owned by or associated with other users).

[0128] Generally, functions described as being performed by the online virtual experience server 102 in one embodiment may, if appropriate, also be performed by the client device 110 or a server in other embodiments. Additionally, functions attributed to a particular component may be performed by different or multiple components operating together. The online virtual experience server 102 may also be accessed as a service provided to other systems or devices through a suitable application programming interface (API) and is thus not limited to use within a website. Figure 2 - Exemplary body cage

[0129] Figure 2 An example body cage 200 according to some embodiments is shown. Figure 2 In the example, the body cage 200 is an outer cage that surrounds or is superimposed on the outer surface / contour of a humanoid body shape that serves as a mannequin. The underlying humanoid body shape (mannequin, not shown) surrounded by the body cage 200 may be represented by a body mesh including a plurality of polygons and their vertices, or may be composed of a body mesh including a plurality of polygons and their vertices. The polygons of the body mesh (as well as the polygons of the clothing mesh) may be triangles, and the surface area of each triangle provides a face or a mesh face.

[0130] The body cage 200 includes a plurality of feature points 202 that define or otherwise identify or correspond to the shape of the mannequin. In some embodiments, the feature points 202 are formed by the vertices of line segments / edges 204 of a plurality of polygons (or other geometric shapes) on the mannequin. According to various embodiments (although not shown as such in Figure 2 ), the polygons may be triangles, and the surface area of each triangle provides a face or a cage face. In some embodiments, the feature points 202 may be discrete points and are not necessarily formed by the vertices of any polygon.

[0131] Figure 2 The body cage 200 shown is an example of a low-resolution body cage with 642 feature points (or some other number of feature points) for a humanoid body geometry lacking fingers. Other examples may use a body cage with 475 feature points (or some other number of feature points). For example, a body cage for a humanoid geometry including fingers may have 1164 feature points (or some other number of feature points). A high-resolution body cage may include 2716 feature points (or some other number of feature points). These numbers (and their ranges) of feature points are just some examples - the number of feature points may vary in different embodiments depending on factors such as the preferred resolution, the processing power of the 3D platform, user preferences, the size / shape of the mannequin, etc. Figure 3 - Example body cage

[0132] Figure 3 Shows another example body cage 300 according to some embodiments. The cage can be provided for any arbitrary avatar body shape or clothing shape. Figure 3 In the example of, the body cage 300 is an outer cage that surrounds or overlays the outer surface / contour of the body mesh of a general game avatar body shape.

[0133] Compared with Figure 2 the body cage 200 of, Figure 3 the body cage 300 of can have the same number of feature points. In some embodiments, the number of feature points of the body cage 300 can be different from the number of feature points of the body cage 200, for example, having fewer or more feature points 302 due to different (simpler or more complex) geometries of the game avatar and / or based on other factors. Thus, the number of feature points of different body cages can be different, and the number of feature points can be selected based on different body shapes or other body attributes. Figure 4 - Each part of the body cage

[0134] Figure 4 Shows an example of the parts of a body cage 400 grouped into corresponding body parts according to some embodiments.

[0135] In some embodiments, for bandwidth and performance / efficiency purposes or other reasons, the number of feature points of the cage can be reduced to less than the number provided above, for example, 475 feature points (or some other number of feature points). Additionally, in some embodiments, the feature points (vertices) in the body cage can be arranged into multiple groups (e.g., 15 groups), with each group representing a part of the body shape.

[0136] More specifically, Figure 4 the 15 body parts shown (for a humanoid body model) are: head, torso, hips, right foot, left foot, left calf, right calf, left thigh, right thigh, left hand, right hand, left lower arm, right lower arm, left upper arm, and right upper arm. The number of parts in any body shape can be greater than or less than the 15 body parts shown. For example, a "one-armed" avatar character may have 12 (instead of 15) body parts because one hand, lower arm, and upper arm are omitted. Additionally, other body shapes may involve fewer or more body parts, depending on factors such as body geometry, preferred resolution, processing power, type of avatar character (e.g., animal, alien, monster, etc.).

[0137] Figure 4Each of the 15 groups / parts in [ ] includes feature points that define the part of the avatar body. Such a group of feature points can in turn be mapped to a corresponding piece of clothing. For example, since the graphical representation of a jacket consists of a graphical mesh that logically corresponds to and fits onto the left / right arms and torso of the avatar body, the feature points in the body cage 400 that define the left / right lower arms, left / right upper arms, and torso can be used as the outer cage to be mapped to the inner cage of the jacket.

[0138] In addition, such separation into multiple groups (as Figure 4 shown) enables a piece of clothing to be custom-fitted to an atypical body shape. For example, a 3D avatar can be in the form of a "one-armed" avatar character lacking a left arm. Accordingly, the body cage of the 3D avatar lacks the group of feature points corresponding to the left hand, left lower arm, and left upper arm.

[0139] When the jacket is subsequently selected to be fitted to the 3D avatar, the right lower arm, right upper arm, and torso of the jacket can be deformed to fit onto the corresponding right lower arm, right upper arm, and torso of the 3D avatar (body mannequin), and since there is no left arm cage in the body mannequin to deform, the left lower arm and left upper arm of the jacket do not deform (e.g., remain rigid in their original form from its parent space). Figure 5 -A deformed clothing layer on the body cage

[0140] Figure 5 Shows an example of a clothing layer 500 deformed on a body cage (e.g., Figure 4 the body cage 400 shown). The clothing layer 500 is a graphical representation of a jacket (shown in gray shade in Figure 5 ) and has portions that can be generated / rendered using a polygon mesh 502 (e.g., a clothing mesh), the polygon mesh 402 consisting of a collection of vertices, edges, and faces (which can be triangular faces or other polygon faces).

[0141] The clothing layer 500 includes an inner cage ( Figure 5 not shown in [ ]), the feature points of which correspond to the feature points of the body cage 400. Specifically, the feature points of the inner cage of the clothing layer 500 are mapped to the feature points of the body cage 400 that make up the left lower arm, right lower arm, left upper arm, right upper arm, and torso.

[0142] In some embodiments, the mapping includes directly mapping the feature points of the inner cage of the garment layer 500 to the coordinate positions of the corresponding feature points of the arms and torso of the body cage 400. When the two cages have the same number of feature points, such mapping can involve a 1:1 correspondence, and the mapping can be n:1 or 1:n (where n is an integer greater than 1), in which case multiple feature points in one cage can be mapped to the same feature point in the other cage (or some feature points may not be mapped).

[0143] The garment layer 500 further includes an outer cage, the feature points of which are spaced apart from and connected to the corresponding feature points of the inner cage of the garment layer 500. The feature points of the outer cage of the garment layer 500 are defined along the outer surface contour / geometry of the jacket or otherwise position the outer surface contour / geometry of the jacket so as to define the features of the jacket (e.g., the hood 504, cuffs 506, straight torso 508, etc.).

[0144] According to various embodiments, during the process of fitting the garment layer 500 onto the outer cage of an existing layer (or avatar body), the spatial distance (e.g., the spatial distance between the feature points of the inner cage of the garment layer 500 and the corresponding feature points of the outer cage of the garment layer 500) remains constant. In this way, the feature points of the inner cage of the garment layer 500 can be mapped to the feature points of the body cage 400 to "fit" the inside of the jacket to the torso and arms of the avatar.

[0145] Then, with the distance between the feature points of the inner cage of the garment layer 500 and the corresponding feature points of the outer cage of the garment layer 500 remaining unchanged, the outer contour of the jacket can also be deformed to match the shape of the avatar body, so that at least the visual appearance (graphical representation) of the hood, cuffs, straight torso, and other surface features of the jacket is at least partially maintained while matching the shape of the avatar body as Figure 5 shown. In this way, the garment layer 500 can be deformed in any suitable manner to fit any arbitrary shape / size of the avatar body (body cage), such as tall, short, slender, strong, humanoid, animal, alien, etc. Figure 6 - Example outer cage

[0146] Figure 6 Shows an example of the parts of the garment layer and the body cage 400 for forming the outer cage 600 according to some embodiments. In some embodiments, an additional garment layer can be placed on top of another garment layer (e.g., in response to user selection). More specifically, Figure 5 the feature points of the outer cage of the garment layer 500 are now combined with the feature points of the body cage 400, resulting in a composite outer cage 600 composed of the feature points of the exposed part of the body cage 400 and the feature points along the outer surface of the jacket. Figure 5 the feature points of the outer cage of the garment layer 500 are now combined with the feature points of the body cage 400, resulting in a composite outer cage 600 composed of the feature points of the exposed part of the body cage 400 and the feature points along the outer surface of the jacket.

[0147] For example, the exposed outer surface 602 of the jacket (formed by the body, hood, and sleeves of the jacket) provides a set of feature points, and the exposed legs, hands, head, and portions of the chest of the body not covered by the jacket provide another set of feature points, and these two sets of feature points (combined) provide the feature points of the outer cage 600.

[0148] Figure 6 The feature points in the outer cage 600 that correspond to and define the outer surface / shape of the jacket can be Figure 5 the same feature points of the outer cage of the garment layer 500. In some embodiments, compared with Figure 5 the feature points of the outer cage of the jacket (garment layer 500), different feature points and / or additional feature points and / or fewer feature points can be used for Figure 6 the jacket area in the outer cage 600.

[0149] For example, if the next layer of clothing above the outer cage 600 requires a higher resolution or a more precise fit, additional feature points of the outer cage 600 enclosed by the jacket area can be calculated (compared with Figure 5 the outer cage of the garment layer 500). Similarly, if the next layer of clothing above the outer cage 600 requires a lower resolution or a less precise fit and / or due to other considerations (such as improving processing / bandwidth efficiency by using as few feature points as possible), the feature points of the outer cage 600 enclosed by the jacket area can be calculated (compared with the outer cage of the garment layer 500).

[0150] In operation, if the user provides an input to fit an additional layer of clothing (such as a coat or other garment) on the jacket (garment layer 500) and / or other parts of the avatar body, the feature points of the inner cage of such an additional layer of clothing are mapped to the corresponding feature points of the outer cage 600. Thus, deformation can be performed in a manner similar to that described with reference to Figure 5 According to some embodiments, radial basis function (RBF) techniques and / or other similar interpolation techniques can be used to deform a garment adapted to a lower layer of clothing or a body part of the avatar.

[0151] Thus, according to Figure 5 and Figure 6 the example of the layered clothing, by matching the feature points of the "outer cage" (body cage 400) of the avatar body with the feature points of the "inner cage" of the first layer of clothing, the first layer of clothing (garment layer 500) is wrapped around the body. This matching can be performed in the UV space of the cage (e.g., UV refers to the coordinate system), so that it is not necessary to rely on the number of feature points that exactly match between the inner cage and the outer cage.

[0152] For example, the feature points can be vertices with position and texture space coordinates. The texture space coordinates are typically represented in the range [0,1], with each range for the U and V coordinates. The texture space can be considered as the "unwrapped" normalized coordinate space of the vertices. By performing the correspondence of two sets of vertices in the UV space and not using the positions of these vertices, the vertex-to-vertex correspondence can be accomplished in the normalized space by eliminating the hard requirement of an exact vertex-to-vertex index mapping.

[0153] According to the techniques described herein, each avatar body and clothing item is thus associated with an "inner cage" and an "outer cage". In the case of an avatar body, the inner cage represents the default "mannequin" (and different mannequins can be provided for different avatar body shapes), and the "outer cage" of the avatar body represents the outer shell surrounding the avatar body shape. For a clothing item, the "inner cage" represents the inner envelope used to define how the clothing item wraps around the underlying body (or around the body already fitted with a previous clothing layer), while the "outer cage" represents the way the next layer of clothing wraps around that particular clothing item when worn on the avatar body.

[0154] According to various embodiments, the various cages described herein can be invisible during runtime. For example, when participating in a virtual experience, including an avatar navigating through the virtual 3D environment of the virtual experience, placing clothing on the avatar body, wearing the clothing, animating the avatar, etc., the vertices and line segments of the cages can be invisible to the user and other users / viewers of the 3D environment. Additionally, during runtime, the avatar and the deformed clothing of the avatar appear to have no cages, such that only the visual meshes of the deformed clothing, skinned, avatar body parts, etc. are visible to the user - in fact, one or more cages can exist on the avatar for the purposes described herein, to deform the clothing, surround the avatar body parts and clothing items, change the avatar body, etc., but are invisible to the user during runtime. The cages can be visible to the user (e.g., such as through a view / edit cage command, during a configuration phase, etc.) such that the user can view and manipulate the cages when necessary to change the avatar body described herein, create cages, or for other purposes. Figure 7 -Interpolation between two body cages

[0155] Figure 7 An example of obtaining an interpolation between two body cages for a new body cage 700 is shown according to some embodiments. More specifically, Figure 7Example interpolations can be performed in the following cases: The user has a current avatar body and wishes to change / transform that current avatar body into some other (target) avatar body that exists in a virtual experience, library, etc. During a session in a studio or other configured environment where the user can create and edit graphical objects, etc., the interpolation of the current avatar body and the corresponding change can be performed while the avatar is participating in the running of a virtual experience.

[0156] In Figure 7 the example of, for illustrative and explanatory purposes only, the user's current avatar body can be a humanoid body with a Figure 2 body cage 200, and the user may wish to transform the current avatar body (or some parts of the current avatar body) into a target avatar body. In Figure 7 the example of, the target avatar body selected by the user can be a geometric avatar body with a Figure 3 body cage 300.

[0157] Figure 7 The new avatar body shown has a body cage 700. The new avatar body can be a complete transformation from one or more parts of the original avatar body to one or more parts of the target avatar body. Figure 7 Shows a complete transformation and a partial transformation from one or more parts of the original avatar body to one or more parts of the target avatar body.

[0158] As an example of a complete transformation, the shape of the torso 702 of the new avatar body has been transformed to exactly match the rectangular shape of the torso of the target avatar (with body cage 300) - the curved / tapered torso of the original avatar body (with body cage 200) no longer exists in the new avatar body and has been completely deformed or otherwise transformed into a rectangular torso 702.

[0159] As an example of a partial transformation, the shape of the arm 704 of the new avatar body is a hybrid / blend between the curved / tapered arm of the original avatar body (with body cage 200) and the rectangular arm of the target avatar body (with body cage 300). For example, now the shape of the arm 704 is more like a rectangle, similar to the arm of body cage 300, but still retains some of the curvature and taper of the arm of body cage 200.

[0160] Thus, a complete transformation can be a body transformation where all body parts are completely transformed, and a partial body transformation can be a body transformation where one or more body parts are partially transformed or not transformed.

[0161] The body cage 700 of the new avatar body also represents a partial transformation, as not every part of the entire avatar body has undergone deformation. For example, only the torso 702 and one arm 704 have undergone transformation, while the shapes of the other parts of the new avatar body (e.g., the head, the other arm, the legs, etc.) remain unchanged relative to the original avatar body. In various embodiments, different parts of the avatar body can undergo partial or full transformation, while other parts do not undergo any transformation. In some embodiments, the entire avatar body can undergo partial or full transformation. In some embodiments, deforming a part of the first avatar body that is less than the whole of the first avatar body includes deforming that part of the first avatar body to perform a partial transformation of that part of the first avatar body.

[0162] The size of the new avatar body (with body cage 700) can be the same as or different from the original avatar body (with body cage 200) and / or the target avatar body (with body cage 300). In Figure 7 the example, the new avatar body (with body cage 700) is shrunk so that its size is smaller than the original avatar body and the target avatar body.

[0163] To obtain the new avatar body (with body cage 700), in some embodiments, one or more interpolation operations 706 can be performed. For example, linear interpolation or non-linear interpolation can be performed between body cage 200 and body cage 300. Interpolation can be performed between the values / coordinates of the vertices or line segments corresponding to the two body cages (body cage 200 and body cage 300) in order to obtain the resulting vertices / line segments of the new body cage 700.

[0164] Optionally, or additionally, at least some of the values / coordinates of the vertices / line segments of the new body cage 700 can be calculated / generated as new values that have not been interpolated from other values. For example, this can be done if new vertices / line segments of the new body cage 700 are to be created in a specific region of the avatar body and there are no vertices / line segments near the two body cages 200 and 300 that can form the basis for interpolation.

[0165] Figure 7 The example corresponds to embodiments in which the new avatar body (with the new body cage) is generated based on or relative to two other avatar bodies (with the corresponding body cages). In some embodiments, the new avatar body can be generated in a more free-form manner, not necessarily based on an existing target avatar body as a reference. Figure 8 - Generation of a new body cage

[0166] Figure 8 Illustrates an example of generating a new body cage 800 according to some embodiments. For illustrative and explanatory purposes only, the user's current avatar body can be one with Figure 2the humanoid body of the body cage 200, and the user may wish to transform the current avatar body (or some parts of the current avatar body) into a target avatar body.

[0167] To perform this transformation, the user can use the transformation tool to manipulate the vertices and / or line segments of the body cage 200. For example, as shown at 800, the user can click and drag a vertex or line segment of an arm to a new position. As shown at 802, the user can click and drag a vertex or line segment of the torso to a new position. In addition to clicking and dragging existing vertices / line segments, during the process of generating a new avatar body, the user can also use the transformation tool to delete or add vertices / line segments to the cage, draw / redraw parts of the cage, etc.

[0168] Regarding the cage, as described above with respect to Figures 2 to 6 there are interrelationships and dependencies between multiple cages. For example, the body cage encloses (completely encloses) the avatar body (including the body mesh), the inner cage of the first clothing item is mapped to the body cage, the outer cage encloses the first clothing item (including the clothing mesh of the first clothing item), the inner cage of the second clothing item is mapped to the outer cage of the first clothing item, the outer cage encloses the second clothing item (including the clothing mesh of the second clothing item), etc.

[0169] Given such interrelationships and dependencies, in some embodiments, an operation or other change / transformation of at least one cage causes an automatic and corresponding change / transformation of one or more other cages. For example, if the body cage of the current avatar body changes (as shown in Figure 7 and Figure 8 ), thereby changing the shape of the avatar body, then if the avatar body is wearing clothing, the corresponding cages of one or more clothing items covering it will also automatically change / update, so as to deform / fit the clothing dynamically to match the changed shape of the avatar body. The changes in the cage will in turn cause appropriate changes in the mesh, skinning, and other visual aspects of the avatar body and / or its clothing.

[0170] In other examples, to change the shape or other appearance of the avatar, the cage of the clothing item can be optionally or additionally manipulated while changing the body cage. Using the example of the current humanoid avatar wearing a round baseball cap and the target alien with a conical head described above, the user can operate the transformation tool to reshape the outer cage of the baseball cap from round to conical. Then this reshaping of the outer cage correspondingly changes the visual appearance of the baseball cap from round to conical, and also changes the inner cage of the baseball cap and the underlying body cage of the head of the humanoid avatar body, so that the new avatar body now has a conical head.

[0171] According to various embodiments, deformation or other transformations of an avatar body (including its clothing) can be performed during the runtime of a virtual experience. In such embodiments, the cage may not necessarily be visible while the user is participating in the virtual experience. Additionally, the user can select a target avatar or other target graphical object in the virtual experience, and the virtual experience engine or other relevant components on the client side or server side can perform appropriate cage operations (as described above with respect to Figure 6 shown) in a manner transparent to the user (e.g., through a background process). Thus, the user is able to seamlessly view the changed / varying visual appearance of the avatar body during the virtual experience without actually viewing the cage itself being operated on. Figure 9 - Transformation of the avatar body during virtual experience

[0172] Figure 9 An example of the transformation of an avatar body when the avatar participates in a virtual experience 900 according to some embodiments, as explained above, is shown. In the virtual experience 900 at 902, the user's current avatar 904 is humanoid, and the respective body parts of the humanoid (e.g., head, arms, legs, torso, etc.) have a generally geometric / rectangular shape. The user has selected another avatar in the virtual experience 900 as the target avatar 906 (e.g., by clicking with a mouse cursor or through some other input tool).

[0173] If the user selects the target avatar 906, then at 908, a transformation of the current / original avatar 904 to a new (changed / changing) avatar 910 occurs. At 908, and compared to the original avatar 904, the new avatar 910 has a larger left arm similar to the corresponding body part of the target avatar 906, a tapered waist, raised shoulders, etc. At 912, a further deformation is performed such that the new avatar 910 is further modified to more closely match the target avatar 906 with a drooping head. Figure 10 - Transformation of the avatar's body during virtual experience

[0174] Figure 10 Another example of the transformation of an avatar body when the avatar participates in a virtual experience 1000 according to some embodiments is shown. Specifically, Figure 10 It is shown that the user can select multiple avatars as the target for the current avatar body deformation. At 1002, in the virtual experience 1000, the user's current avatar 1004 has an alien body with a slender body shape / form. The first target avatar 1006 is a humanoid avatar and has a muscular torso shape. The second target avatar 1008 is a monster and has a horned, drooping monster head.

[0175] At 1010, the user has selected a first target avatar 1006, and thus the original avatar 1004 has been transformed into a new avatar 1012, which has a robust torso like the first target avatar 1006. At 1014, the user has selected a second target avatar 1008, and thus the new avatar 1012 continues to transform / deform to have a drooping monster head and be shorter, like the second target avatar 1008. Figure 11 - Transformation of the avatar body during virtual experience

[0176] Figure 11 Another example of the transformation of an avatar's body when the avatar participates in a virtual experience 1100 is shown. As previously described, a part of the avatar's body (rather than the entire avatar's body) can be changed, and then corresponding changes can be made to the clothing, skinning, accessories, etc. associated with the changed part of the avatar's body.

[0177] At 1102, in the virtual experience 1100, the user's current avatar 1104 has a humanoid head with a nose, hair, lips and lipstick, eyes and eyelashes, etc. At 1106, the user has selected the head of another avatar in the virtual experience 1100, and thus the head of the current avatar 1104 begins to deform from the humanoid head into the different head shape (e.g., animal-like) of the new avatar 1108. This deformation also affects the appearance (e.g., shape and size) of the nose, lips, eyes, etc.

[0178] At 1110, the head of the new avatar 1108 continues to further deform into an animal head. Thus, the head, hair, nose, lips, eyes, etc. in the new avatar 1108 have a more distinct animal-like appearance. Figure 12 -Transform of the avatar in the configured environment

[0179] Figure 12 An example of the transformation of an avatar in a configuration environment 1200 according to some embodiments is shown. The configuration environment 1200 can be a studio or other type of environment where the user can configure the avatar outside of the runtime environment of the virtual experience.

[0180] Thus, the configuration environment 1200 can be an auxiliary feature that is related to the virtual experience but not within the virtual experience itself. Optionally, or additionally, the configuration environment can be decoupled from any specific virtual experience, but the output of the configuration environment (including the avatar and other graphical objects) can be used and applied to the virtual experience.

[0181] In the configuration environment 1200, there are multiple available avatar models: the user's avatar 1202 (bazooka), the first target avatar 1204 (Model9), and the second target avatar 1206 (roxie). In this example, the avatar body of the user's avatar 1202 is shaped like a stocky alien and is skeletal in form.

[0182] The user has selected the target avatar 1206 as the target avatar. Accordingly, the user's avatar 1202 transforms into a new avatar 1208, which is rendered in the configuration environment 1200. Although the new avatar 1208 retains some of the skeletal features of the original avatar 1202, it is now more humanoid in shape, corresponding to the target avatar 1206, and taller.

[0183] According to various embodiments, the configuration environment 1200 may be provided with adjustment tools and / or other types of transformation tools 1210. For example, the transformation tool 1210 indicates that the transformation tool 1210 corresponds to a "morph body plugin" and provides the instruction "Select avatar A and then select B to morph A into B. The value is the degree to which A will morph into B. When the value is 0, there is no change. When the value is 1, A will look like B."

[0184] In Figure 12 the example, the transformation tool 1210 includes a slider or other similar tool ("MorphValue") to control the amount of morphing (e.g., the morph value) between the current avatar and the target avatar. There may be buttons indicating "Current Deform Head" and "Apply Morph and Value"

[0185] Accordingly, the new avatar may have a minimum morph value of 0 (e.g., the current avatar does not change), a maximum morph value (e.g., the new avatar is a complete transformation to the target avatar), and any other morph value between the minimum and maximum morph values (e.g., as Figure 12 shown, the new avatar is a blend between the original avatar and the target avatar).

[0186] Although not shown in Figure 12 , as Figure 2 shown, some embodiments of the configuration environment 1200 enable the user to operate a cage for the purpose of changing the avatar body. Various transformation tools may be provided in the configuration environment 1200 to enable the user to click and drag, draw, delete, modify, etc. the vertices of the cage. Figure 13 - Layered clothing for an avatar body in virtual experience

[0187] Figure 13An example of a layered clothing of an avatar body participating in a virtual experience 1300 according to some embodiments is shown. At 1302, the user's avatar 1304 is a new avatar transformed from a previous avatar (e.g., by the techniques described herein), and is wearing some clothing on a body cage (not shown). The inner cage (not shown) of each outer layer of clothing of avatar 1304 has been deformed so that each outer layer of clothing conforms / fits to the changed / new avatar 1304.

[0188] In the virtual experience 1300, the avatar 1304 runs towards one of a series of other clothing items (e.g., coat 1306). When the avatar puts on the coat 1306 at 1308, the coat 1306 deforms to fit and conform to the underlying clothing layer. The engine running the virtual experience 1300 and / or some other components can seamlessly perform the deformation of the coat (e.g., cage mapping, cage deformation, etc.) such that the fitting is performed seamlessly from the user's perspective (e.g., the user simply clicks on the coat 1306 and the coat 1306 automatically fits onto the avatar 1304).

[0189] In the foregoing embodiments and examples, one or more parts of the original avatar body are deformed in the manner shown and described above, changing the original avatar body into another (new) avatar body. The skin / mesh and any clothing layers worn by the original avatar body also deform so as to conformingly fit the new avatar body.

[0190] After changing / updating the geometry (e.g., avatar body, skin, clothing layer, etc.), the user can animate the avatar, for example, by running, smiling, blinking, waving an arm, etc. Next, examples are provided in Figures 14 to 17 In Figures 14 to 17 -Transformation and animation of the avatar

[0191] Figures 14 to 17 An example of the transformation and animation of an avatar 1400 according to some embodiments is shown. Figures 14 to 17 Each of

[0192] In Figure 14 shows the head of the avatar 1400, and three different versions of the avatar 1400. Figure 14 In

[0193] In Figure 15In [the figure], the avatar 1400 is shown in its original form (undeformed form) for reference, and the other three avatars begin to deform into different head shapes. The avatar 1400 is also animated to be smiling (with lips parted and teeth visible). A corresponding smile can also be seen in the animations of the other three avatars.

[0194] In Figure 16 In [the figure], the avatar 1400 is shown in its original (undeformed form) for reference, and the other three avatars continue to deform into different head shapes and have smile animations corresponding to the smile animation of the avatar 1400. In Figure 17 In [the figure], the avatar 1400 is animated to blink one eye partially and grin with lips, and the same expressions are animated on the other three avatars.

[0195] According to various embodiments, an avatar can be provided with skin such that the mesh of the avatar is bound to the joints and bones of the avatar skeleton. Thus, as shown in the above description Figures 15 to 17 During the animation process, the movement of the joints / bones of the avatar causes corresponding skin deformation. In some embodiments, the skeleton of the avatar can be an inferred skeleton composed of virtual joints and virtual bones.

[0196] According to various embodiments, when the geometric form of the avatar body changes as described in the above examples (e.g., by changing the geometric form / shape of the arms, legs, torso, etc. for a new avatar), the skeleton is also updated for the new avatar. As Figures 15 to 17 shown, updating the skeleton (including updating its joints and bones, etc.) and updating the skin ensure accurate animation effects for the new avatar.

[0197] In some embodiments, such updating can be performed by interpolating the attachment positions (e.g., joints) for the new avatar body. The interpolated attachment positions can be obtained by interpolating between the attachment positions of the skeleton of the original avatar body and the attachment positions of the skeleton of the target avatar body.

[0198] In some embodiments, interpolation can be performed between the vertices of the body cage of the original avatar body and the vertices of the body cage of the target avatar body to obtain the attachment points of the skeleton of the new avatar body. Thus, when deforming the original avatar body by operating the body cage of the original avatar body, a similar deformation can be performed on the skeleton of the original avatar body to reposition the joints and bones of the skeleton.

[0199] As an example, a body cage is associated with an avatar body through a mesh that surrounds the avatar body. Thus, the body cage approximates the shape, size, outline, etc. of the body part represented by the mesh. Considering rigging, the mesh of the avatar is bound to the joints and bones of the avatar skeleton, and the deformation of the body cage causes the corresponding mesh to deform. The same or similar deformation (due to the binding of the mesh to the skeleton) is also applied to the skeleton.

[0200] When performing interpolation between the vertices of a pair of cages with a direct 1:1 vertex correspondence, the result of the interpolation provides vertices for the body cage of the avatar body. Additionally, there may be vertices on the body cage of the original avatar body that do not correspond to any vertices on the body cage of the target avatar body, and vice versa. In such cases, UV correspondence techniques or other types of techniques for mapping between graphical objects can be used to identify the nearest vertices for interpolation. The coordinate systems of the paired cages can also be aligned to further improve the accuracy of the interpolation. Figure 18 - Changing the body of a 3D avatar

[0201] Figure 18 is a flowchart showing a computer-implemented method 1800 for changing a three-dimensional (3D) avatar body according to some embodiments. For simplicity, the various operations in method 1800 are described in the context of a virtual experience (VE) of a client device performing the operations.

[0202] Further, as referred to below Figure 22 as described, some operations of method 1800 and / or any other method described herein can be optionally or additionally performed in whole or in part by a VE engine of a VE platform located on a server. Example method 1800 can include one or more operations shown by one or more blocks (e.g., block 1802 to block 1806). The various blocks of method 1800 and / or any other process described herein can be combined into fewer blocks, divided into additional blocks, supplemented with other blocks, and / or eliminated based on the embodiments.

[0203] As referred to herein Figures 2 to 17 and the elements shown in other figures to illustrate Figure 18 method 1800. In some embodiments, the operations of method 1800 can be performed in a pipelined sequential manner. In other embodiments, some operations can be performed out of order, in parallel, etc.

[0204] In block 1802, identify a first avatar body to be changed. For example, when the user is participating in a virtual experience or when the user is in a configuration environment such as a studio, the first avatar body can be the user's current avatar body. The first avatar body has a corresponding first body cage. After block 1802 can be block 1804.

[0205] At block 1804, identify a target avatar body. For example, a user may identify a target avatar body in a virtual experience or in a configured environment. The target avatar body is a body into which the user may desire the user's current avatar body to morph. The target avatar body has a corresponding target body cage. After block 1804 may be block 1806.

[0206] At block 1806, perform a transformation of the first avatar body. For example, an interpolation may be performed between the first body cage and the target body cage to generate a second body cage corresponding to a second avatar body. Thus, the second avatar body may be a blend between the first avatar body and the target avatar body, or may be fully transformed into the target avatar body. Additional details on how to perform the interpolation are referenced Figure 20 for discussion. Figure 19 - Changing the body of a 3D avatar

[0207] Figure 19 is a flow chart showing another computer-implemented method 1900 for changing a three-dimensional (3D) avatar body according to some embodiments. For example, method 1900 may be used for free-form transformation of an avatar body without having to use a target avatar body as a reference.

[0208] At block 1902, identify a first avatar body to be changed. For example, when the user is in a configured environment such as a studio, the first avatar body may be the user's current avatar body. In some embodiments, the user's avatar may exist in a running virtual experience and may thus be changed by pausing the virtual experience, by exiting the virtual experience to enter the configured environment, or by changing the avatar within the running virtual experience itself. The first avatar body has a corresponding first body cage. After block 1902 may be block 1904.

[0209] At block 1904, perform a transformation of the first avatar body. For example, the transformation may be performed by operating on the first body cage (e.g., repositioning the vertices / segments of the first body cage) to generate a second body cage corresponding to a second avatar body. The second body cage may be used to provide the transformation of the first avatar body to the second avatar body. The operation may include repositioning various parts of the first body cage. Additional details on how to perform the transformation are referenced Figure 20 for discussion. Figure 20 - A deformed clothing layer on the body cage

[0210] Figure 20 is a flow chart showing a computer-implemented method for performing skinning morphing according to some embodiments. Figure 20FIG. 2000 is a flowchart of a computer-implemented method for changing a three-dimensional (3D) avatar body according to some embodiments. For example, method 2000 can be used for free-form transformation of an avatar body without having to use a target avatar body as a reference. Method 2000 can start at block 2002.

[0211] At block 2002, an affine correction of the skeleton is calculated. For each bone, a 3x4 affine transformation that best fits the deformation of its affected point cloud is calculated using linear algebra. This can be a D j term, D j that can change the skinning calculation to better fit the change in the mesh shape. After block 2002 can be block 2004.

[0212] At block 2004, the affine correction is applied to the bone transformation. This correction is applied to the bone transformation in the skinning calculation: After block 2004 can be block 2006.

[0213] At block 2006, the correction is bubbled through the skeleton calculation. After block 2006 can be block 2008.

[0214] At block 2008, the correction is incorporated into the local bind transform. Bubbling and combining merge the correction into a new local bind transform LB′ j such that the transform produces the same result: LB′ j = B j -1 · D j · B j′ · LB j . This is an algebraic operation that uses the original LBS formula, removes the transform D j and replaces it with a new bind transform LB′ that produces the same result j . After block 2008 can be block 2010.

[0215] At block 2010, a new skinned mesh is created and / or published. Creating just means that the skinned mesh exists as a mesh for local users to view. Publishing means pushing the skinned mesh to the Roblox CDN so that other users can also see or purchase it. Using the new skinned mesh in this way requires passing through security reviews and other processes. After block 2010 can be block 2012.

[0216] At block 2012, the translation and rotation are updated. The FACS bind contains the translation and rotation of each joint in each pose of the bind. After a shape change, these values need to be updated so that the poses can continue to work with the new shape. As described with respect to Figure 21 this update can be performed using method 2100. Figure 21 - A deformed clothing layer on the body cage

[0217] Figure 21 is a flowchart showing a computer - implemented method 2100 for performing Facial Action Coding System (FACS) pose deformations. Method 2100 can start from block 2102.

[0218] Regarding FACS pose deformations, this FACS binding contains many poses, and each pose includes a set of local transformations P for each joint in the head binding. j Skinning deformation techniques update their bound transformations LB′ j , thus changing the parent space of these local pose transformations. The affine correction transformation D j changes the translation of the resulting M j transformation, including the direction and magnitude of the translation, but the affine correction transformation D j does not change the rotation of the resulting transformation M j . To further improve the result, the rotation r of each pose and each joint j,k will also be updated. During this process, the translation t of each pose and each joint can also be further fine - tuned. j,k Such a process can be performed one pose at a time.

[0219] At block 2102, a deformed head - mesh pose is calculated from the original head mesh. Specifically, an original LBS deformed - mesh pose is calculated based on the original head shape. A pose is defined as a set of joint transformations. These joint transformations are used with techniques such as linear - blend skinning to construct the mesh shape in that pose. After block 2102 can be block 2104.

[0220] At block 2104, skinned - weighted points are projected to the nearest points to generate new points. For each joint in the pose, the skinned - weighted points of the joint are projected to the nearest points on the pose mesh, thus providing a set of new points. These points represent the positions on the original mesh that are closest to the destination of that joint in that pose. For example, when considering a smiling pose, the corners of the mouth slide into the cheek area. Understanding how the cheek reshapes can be helpful to understand whether and how to correspondingly adjust the smiling pose. After block 2104 can be block 2106.

[0221] At block 2106, an affine correction is applied to the new points to generate a transformation. The same 3x4 affine - fitting function (used at block 2004) can be used to calculate how these points transform between the original shape and the modified shape. After block 2106 can be block 2108.

[0222] At block 2108, extract the transformed rigid components. Subsequently, the transformed rigid components can be extracted. Some systems, including some rigid body simulation systems, cannot effectively handle scaling and shearing, so these components must be removed before providing the correction to these systems. After block 2108 can be block 2110.

[0223] At block 2110, fine-tune the translation and / or rotation based on the extracted rigid components. After block 2110 can be block 2112.

[0224] At block 2112, calculate the Euler angles by decomposing the rotation matrix and inserting the correction. To minimize the change to Euler interpolation, make the smallest change to the Euler angles that match the rotation. These angles are calculated by decomposing the rotation matrix and gradually inserting the correction. Then (This shows how to incorporate the corrected rotation with subscript 2 into the rotation calculation), then x ′ = x1 + x2, y ′ = y1 + y2, and z ′ = z1 + z2 (This shows how to directly add the Euler angles from the correction matrix (subscript 2 in the previous equation) to the original rotation and produce the same result). This shows the Euler rotation matrix decomposed into separate x, y, z rotation matrices. Here, the capital letters are transformation matrices and the lowercase letters are scalar values. Figure 22 - Example computing device

[0225] Figure 22 is a block diagram showing an example computing device 2200 that can be used to implement one or more features described herein. In one example, the computing device 2200 can be used to implement a computer device (e.g., Figure 1 102 and / or 110 of), and perform the appropriate method implementations described herein. The computing device 2200 can be any suitable computer system, server, or other electronic or hardware device. For example, the computing device 2200 can be a mainframe computer, desktop computer, workstation, portable computer, or electronic device (portable device, mobile device, cellular phone, smart phone, tablet computer, television, set-top box, personal digital assistant (PDA), media player, gaming device, wearable device, etc.). In some embodiments, the computing device 2200 includes a processor 2202, a memory 2204, an input / output (I / O) interface 2206, and an audio / video input / output device 2214.

[0226] The processor 2202 can be one or more processors and / or processing circuits to execute program code and control the basic operations of the computing device 2200. A "processor" includes any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. The processor can include a system with a general-purpose central processing unit (CPU), multiple processing units, dedicated circuits for implementing functions, or other systems. The processing is not limited to a specific geographical location and has no time limit. For example, the processor can perform its functions in "real-time", "offline", "batch mode", etc. Various parts of the processing can be executed by different (or the same) processing systems at different times and in different locations. A computer can be any processor that communicates with a memory.

[0227] The memory 2204 is typically provided in the computing device 2200 for access by the processor 2202 and can be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. The memory 2204 is suitable for storing instructions for execution by the processor and is separate from and / or integrated with the processor 2202. The memory 2204 can store software for operation by the processor 2202 on the computing device 2200, including an operating system 2208, a virtual experience application 2210, a 3D avatar modification application 2212, and other applications (not shown). In some embodiments, the virtual experience application 2210 and / or the 3D avatar modification application 2212 can include instructions that enable the processor 2202 to perform (or control) the functions described herein. For example, regarding Figures 18 to 21 some or all of the methods described.

[0228] For example, the virtual experience application 2210 can include the 3D avatar modification application 2212, and as described herein, the 3D avatar modification application 2212 can dynamically change a 3D avatar within an online virtual experience server (e.g., 102). The software elements in the memory 2204 can alternatively be stored at any other suitable storage location or computer-readable medium. Additionally, the memory 2204 (and / or other connected storage devices) can store instructions and data used in the features described herein. The memory 2204 and any other type of storage (disk, optical disc, magnetic tape, or other tangible media) can be considered a "storage area" or "storage device".

[0229] The I / O interface 2206 can provide functions that enable the computing device 2200 to interface with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data storage area 120), and input / output devices can communicate through the I / O interface 2206. In some embodiments, the I / O interface can be connected to an interface device that includes input devices (keyboard, pointing device, touch screen, microphone, camera, scanner, etc.) and / or output devices (display device, speaker device, printer, motor, etc.).

[0230] The audio / video input / output device 2214 can include user input devices (e.g., mouse, etc.) that can be used to receive user input, display devices (e.g., screen, monitor, etc.) that can be used to provide graphical and / or visual output, and / or combined input and display devices.

[0231] For ease of illustration, Figure 22 each of the software boxes of the processor 2202, the memory 2204, the I / O interface 2206, and the operating system 2208, the virtual experience application 2210, and the 3D avatar modification application 2212 is shown as a box. These blocks can represent one or more processors or processing circuits, operating systems, memories, I / O interfaces, applications, and / or software engines. In other embodiments, the computing device 2200 may not have all of the components shown, and / or may have other elements, including other types of elements not shown herein, or in addition to those shown herein. Although the online virtual experience server 102 is described as performing the operations described in some embodiments herein, any suitable component or combination of components of the online virtual experience server 102 or a similar system, or any suitable processor associated with such a system, can perform the operations.

[0232] User equipment may also implement and / or be used in conjunction with the features described herein. An example user equipment may be a computing device including some components similar to those of computing device 2200 (e.g., processor 2202, memory 2204, and I / O interface 2206). An operating system, software, and applications suitable for the client device may be provided in the memory and used by the processor. The I / O interface for the client device may be connected to network communication devices, as well as input and output devices, such as a microphone for capturing sound, a camera for capturing images or video, a mouse for capturing user input, a gesture device for recognizing user gestures, a touch screen for detecting user input, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices. For example, a display device within the audio / video input / output device 2214 may be connected to (or included in) the computing device 2200 to display pre-processed and post-processed images as described herein, where such display device may include any suitable display device, such as an LCD, LED, or plasma display screen, CRT, television, monitor, touch screen, 3D display, projector, or other visual display device. Some embodiments may provide an audio output device, such as voice output or synthesized speech for reading text.

[0233] One or more of the methods described herein (e.g., methods 1800, 1900, 2000, and 2100) can be implemented by computer program instructions or code that can be executed on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and can be stored on a computer program product that includes a non-transitory computer-readable medium (e.g., a storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid-state memory, magnetic tape, removable computer floppy disk, random access memory (RAM), read-only memory (ROM), flash memory, hard magnetic disk, optical disk, solid-state storage drive, etc. The program instructions can also be embodied in and provided as an electronic signal, e.g., in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more of the methods can be implemented using hardware (e.g., logic gates, etc.) or a combination of hardware and software. Example hardware can be a programmable processor (e.g., a field-programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application specific integrated circuit (ASIC), etc. One or more of the methods can be executed as part of or a component of an application running on a system, or as an application or software running together with other applications and an operating system.

[0234] One or more of the methods described herein can run in a stand-alone program that can run on any type of computing device, a program running on a web browser, or a mobile application (“app”) that runs on a mobile computing device (e.g., a cell phone, a smart phone, a tablet, a wearable device (watch, armband, jewelry, headgear, goggles, glasses, etc.), a laptop computer, etc.). In one example, a client / server architecture can be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives final output data from the server for output (e.g., for display). In another example, all computations are performed within a mobile application (and / or other applications) on the mobile computing device. In another example, the computations can be split between the mobile computing device and one or more server devices.

[0235] Although the specification has been described with respect to its specific embodiments, these specific embodiments are for illustration only and not for limitation. The concepts illustrated in the examples can be applied to other examples and embodiments.

[0236] As is known to those skilled in the art, the functional blocks, operations, features, methods, devices, and systems described in this disclosure can be integrated or divided into different combinations of systems, devices, and functional blocks. Any suitable programming language and programming technique can be used to implement the routines of a particular embodiment. Different programming techniques can be employed, for example, procedural or object-oriented. The routines can be executed on a single processing device or multiple processors. Although steps, operations, or calculations may be presented in a particular order, the order can be changed in different particular embodiments. In some embodiments, multiple steps or operations shown as sequential in this specification can be executed simultaneously.

Claims

1. A computer-implemented method for modifying a three-dimensional (3D) avatar body, the computer-implemented method comprising: identifying a first avatar body having a first body cage; identifying a target avatar body having a target body cage; as well as Interpolation is performed between the first body cage and the target body cage to obtain a second body cage corresponding to a second avatar body, thereby providing a transformation of the first avatar body to the second avatar body.

2. The computer-implemented method of claim 1 , wherein: Performing the interpolation includes performing the interpolation to generate the second body cage that exactly matches the target body cage, thereby providing a complete transformation.

3. The computer-implemented method of claim 1 , wherein: Performing the interpolation includes transforming the first avatar body into the second avatar body, the second avatar body being a hybrid between the first avatar body and the target avatar body, thereby providing a partial transformation.

4. The computer-implemented method of claim 1 , wherein: Performing the interpolation includes deforming a portion of the first avatar body that is smaller than the entirety of the first avatar body.

5. The computer-implemented method of claim 4, wherein: Deforming the portion of the first avatar body that is smaller than the entirety of the first avatar body includes deforming the portion of the first avatar body to perform a partial transformation of the portion of the first avatar body.

6. The computer-implemented method of claim 1 , wherein: The first avatar body is part of a virtual experience, the interpolation is performed while the avatar is participating in the virtual experience, and the target avatar body is selected from a plurality of target avatar bodies in the virtual experience.

7. The computer-implemented method of claim 1 , wherein: The interpolation is performed in a configuration environment, and the target avatar body is selected from a plurality of target avatar bodies in a library in the configuration environment.

8. The computer-implemented method of claim 7, wherein: The configuration environment includes a transformation tool that enables a user to control an amount of the transformation of the first avatar body to obtain the second avatar body, and wherein the interpolation is performed based on the amount of the transformation.

9. The computer-implemented method of claim 1 , further comprising: identifying a rig of the first avatar body, the rig comprising identifying a skeleton of the first avatar body and a skin of the first avatar body; After performing the interpolation, updating the binding of the first avatar body to correspond to the second body cage; and The first avatar body is animated by moving the updated bound skeleton and deforming the updated bound skin.

10. The computer-implemented method of claim 9, wherein: Moving the updated bound skeleton and deforming the updated bound skin includes reusing skin weights of the skin from the first avatar body based on determining areas of the updated bound skin that are affected by bones in the skeleton of the first avatar body.

11. A computer-implemented method for modifying a three-dimensional (3D) avatar body, the computer-implemented method comprising: identifying a first avatar body having a corresponding first body cage; as well as Manipulations are performed on the first body cage to generate a second body cage corresponding to a second avatar body to provide a transformation of the first avatar body to the second avatar body, wherein the manipulations include repositioning portions of the first body cage.

12. The computer-implemented method of claim 11, wherein: The operation is performed in a configuration environment, and wherein the configuration environment includes a transformation tool that enables a user to control aspects of the operation of the first body cage to obtain the second body cage, and wherein the operation is performed based on the aspects of the operation.

13. The computer-implemented method of claim 11 , further comprising: identifying a rig of the first avatar body, the rig comprising identifying a skeleton of the first avatar body and a skin of the first avatar body; After performing said operation, updating said binding of said first avatar body to correspond to said second body cage; as well as The first avatar body is animated by moving the updated bound skeleton and deforming the updated bound skin.

14. The computer-implemented method of claim 13, wherein: Moving the updated bound skeleton and deforming the updated bound skin includes reusing skin weights of the skin from the first avatar body based on determining areas of the updated bound skin that are affected by bones in the skeleton of the first avatar body.

15. The computer-implemented method of claim 13, wherein: Transforming the first avatar body to the second avatar body includes performing interpolation between the first body cage and the second body cage.

16. The computer-implemented method of claim 13, wherein: Transforming the first avatar body to the second avatar body includes deforming a portion of the first avatar body that is smaller than the entirety of the first avatar body.

17. A system comprising: a memory having instructions stored therein; as well as A processing device is coupled to the memory, the processing device is configured to access the memory and execute the instructions, wherein the instructions cause the processing device to perform operations including: identifying a first avatar body having a first body cage; identifying a target avatar body having a target body cage; and Interpolation is performed between the first body cage and the target body cage to obtain a second body cage corresponding to a second avatar body, thereby providing a transformation of the first avatar body to the second avatar body.

18. The system of claim 17, wherein: Performing the interpolation includes performing the interpolation to generate the second body cage that exactly matches the target body cage, thereby providing a complete transformation.

19. The system of claim 17, wherein: Performing the interpolation includes transforming the first avatar body into the second avatar body, the second avatar body being a hybrid between the first avatar body and the target avatar body, thereby providing a partial transformation.

20. The system of claim 17, wherein: Performing the interpolation includes deforming a portion of the first avatar body that is smaller than the entirety of the first avatar body.

Images