Automatic skin migration and rigid automatic skin

Abstract

Some embodiments relate to methods, systems, and computer readable media for migrating a skin from a target geometry to an accessory geometry to be attached to the target geometry. Some embodiments include obtaining skin information of an avatar body, migrating the skin information to an accessory to be fitted to the avatar body, generating an automatic skin accessory grid based at least in part on the skin information and an original accessory grid, fitting the accessory to the avatar body based at least in part on the automatic skin accessory grid, and generating a skin accessory to be fitted to the avatar body based at least in part on the original accessory grid. And after the adaptation, animating the avatar body wherein the adapted accessory is animated corresponding to the animated avatar body based at least in part on the automated skin accessory grid. These techniques enable accurate deformation and movement of the accessory without the need to build a skin into the accessory. For an unskinned rigid body, a skin may be derived from the rigid body component itself.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 523,900, filed June 28, 2023, entitled “AUTOMATIC SKINNING TRANSFER AND RIGID AUTOMATIC SKINNING”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to computer graphics, and more specifically, but not exclusively, to methods, systems, and computer-readable media for automatically migrating skins and automatically generating skins for rigid incarnations. Background Technology

[0003] Creating visually compelling animated avatars is a time-consuming process that requires a high level of expertise in three-dimensional (3D) skinning and animation. Skinning a 3D avatar body (including hair, eyebrows, etc.) and accessories (such as clothing) adapted to it in a way that allows accessories to precisely fit and follow the movements of the underlying avatar body can be extremely difficult and time-consuming.

[0004] Based on the above, some implementation methods have been proposed.

[0005] The background description provided herein is intended to introduce the background of this disclosure. The work done by the present inventor, with respect to what is described in this background section, and to any aspects of the specification that may not constitute prior art at the time of application, whether express or implied, should not be considered prior art to this disclosure. Summary of the Invention

[0006] Implementations of this application relate to automatic skinning migration and rigid automatic skinning. For example, the creation process utilizes various techniques to create high-quality avatars while minimizing manual labor. These techniques include migrating skinning from the avatar to accessories to create alternative meshes, which facilitate fitting the accessories to the avatar and animate the avatar. Techniques also exist for inferred skinning used for rigid avatars lacking pre-existing skinning information.

[0007] A system of one or more computers can be configured to perform specific operations or actions by means of software, firmware, hardware, or a combination thereof installed on the system, which, when operated, cause the system to perform the actions. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause that device to perform the actions.

[0008] According to one aspect, a computer implementation method is provided for adapting an accessory to an avatar body in a three-dimensional (3D) environment. The computer implementation method includes: obtaining skinning information of the avatar body; transferring the skinning information to the accessory to be adapted to the avatar body; generating an auto-skinned accessory mesh based at least in part on the skinning information and an original accessory mesh; adapting the accessory to the avatar body based at least in part on the auto-skinned accessory mesh; and animate the avatar body after adaptation, wherein the adapted accessory is animated correspondingly to the animated avatar body based at least in part on the auto-skinned accessory mesh.

[0009] This article describes various implementation methods of the computer implementation approach.

[0010] In some implementations, skinning information includes at least one from a group consisting of: weights, skeletal effects, binding information, joints, other features of the avatar body, and combinations thereof.

[0011] In some embodiments, the accessories to be adapted to the avatar's body include at least one from a group consisting of: body hair, clothing components, and combinations thereof.

[0012] In some implementations, the skinning information of the avatar body is incomplete, and the computer implementation method further includes: performing automatic skinning technology to generate derived skinning information of the avatar body; and generating an automatic skinning accessory mesh based on the derived skinning information of the avatar body.

[0013] In some implementations, performing automatic skinning techniques to generate derived skinning information for the avatar body includes: labeling each body part of the avatar as either a skinned body part or a rigid body part; performing a correspondence search to identify one or more body parts within a threshold distance of the accessory; for each of the identified body parts, in response to the identified body part being a skinned body part, skinning sampling of the corresponding vertex of the avatar body at the spatial location of the nearest identified body part; and in response to the identified body part being a rigid body part, assigning original weights to the corresponding vertices of the avatar body at the location of the identified body part; and applying an automatic skinning smoothing operation to the rigid vertices of the avatar body based on the associated weights of the rigid vertices of the avatar body.

[0014] In some implementations, generating an automatic skinning accessory mesh includes supplementing the original accessory mesh and the accessory wrapping deformer (WD) cage associated with the accessory with information from the avatar mesh, the avatar wrapping deformer (WD) cage, and the skinning information of the avatar body.

[0015] In some implementations, generating an auto-skinned accessory mesh includes performing a geometric correspondence search that uses an accessory-wrapped deformer cage and an avatar-wrapped deformer cage to find a correspondence between the vertices of the original accessory mesh and the nearest corresponding position on the surface of the avatar mesh to which the accessory is to be attached.

[0016] In some implementations, generating an auto-skinned accessory mesh further includes performing centroid sampling of avatar skinning information on at least one vertex of the original accessory mesh to generate the auto-skinned accessory mesh.

[0017] In some implementations, the adapter accessory also includes attaching the accessory to the avatar body by providing skinning information to the wrapper deformer frame using an automatic skinning accessory mesh.

[0018] In some implementations, animate the avatar body on which the accessories are adapted includes: using an auto-skinned accessory mesh to render the animation of the accessories based on the motion of the avatar body.

[0019] According to another aspect, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium stores instructions that, in response to execution by a processing device, cause the processing device to perform operations including: obtaining skinning information of an avatar body in a three-dimensional (3D) environment; transferring the skinning information to an accessory to be adapted to the avatar body; generating an auto-skinned accessory mesh based at least in part on the skinning information and an original accessory mesh; adapting the accessory to the avatar body based at least in part on the auto-skinned accessory mesh; and animate the avatar body after adaptation, wherein the adapted accessory is animated at least in part on the auto-skinned accessory mesh corresponding to the animated avatar body.

[0020] This document describes various implementations of non-transitory computer-readable media.

[0021] In some implementations, the skinning information of the avatar body is incomplete, and the above operations also include: performing automatic skinning technology to generate derived skinning information of the avatar body; and generating an automatic skinning accessory mesh based on the derived skinning information of the avatar body.

[0022] In some implementations, generating an automatic skinning accessory mesh includes supplementing the original accessory mesh and the accessory wrapping deformer (WD) cage associated with the accessory with information from the avatar mesh, the avatar wrapping deformer (WD) cage, and the skinning information of the avatar body.

[0023] In some implementations, generating an auto-skinned accessory mesh includes performing a geometric correspondence search that uses an accessory-wrapped deformer cage and an avatar-wrapped deformer cage to find a correspondence between the vertices of the original accessory mesh and the nearest corresponding position on the surface of the avatar mesh to which the accessory is to be attached.

[0024] In some implementations, the adapter accessory also includes attaching the accessory to the avatar body by providing skinning information to the wrapper deformer frame using an automatic skinning accessory mesh.

[0025] According to another aspect, a system is disclosed, comprising: a memory storing instructions thereon; and a processing device coupled to the memory for accessing the memory, wherein the instructions, when executed by the processing device, cause the processing device to perform operations including: obtaining skinning information of an avatar body in a three-dimensional (3D) environment; transferring the skinning information to an accessory to be adapted to the avatar body; generating an auto-skinned accessory mesh based at least in part on the skinning information and an original accessory mesh; adapting the accessory to the avatar body based at least in part on the auto-skinned accessory mesh; and animate the avatar body after adaptation, wherein the adapted accessory is animate at least in part on the auto-skinned accessory mesh corresponding to the animate avatar body.

[0026] This article describes various implementations of the system.

[0027] In some embodiments, the system further includes a mesh content cache coupled to a computing device for storing autoskinned accessory meshes for at least one of adaptation or animation, and the mesh content cache is coupled to a file repository, wherein the operations further include: determining that no autoskinned accessory mesh is present in the mesh content cache; retrieving an autoskinned accessory mesh from the file repository in response to determining that no autoskinned accessory mesh is present in the mesh content cache; storing the retrieved autoskinned accessory mesh in the mesh content cache; and accessing the autoskinned accessory mesh from the mesh content cache after storage.

[0028] In some implementations, the skinning information of the avatar body is incomplete, and the above operations also include: performing automatic skinning technology to generate derived skinning information of the avatar body; and generating an automatic skinning accessory mesh based on the derived skinning information of the avatar body.

[0029] In some implementations, generating an automatic skinning accessory mesh includes supplementing the original accessory mesh and the accessory wrap deformer (WD) cage associated with the accessory with information from the avatar mesh, the avatar wrap deformer (WD) cage, and the skinning information of the avatar body.

[0030] In some implementations, the adapter accessory also includes attaching the accessory to the avatar body by providing skinning information to the wrapper deformer frame using an automatic skinning accessory mesh.

[0031] According to another aspect, parts, features, and implementation details of a system, method, and non-transitory computer-readable medium may be combined to form additional aspects, including aspects that omit and / or modify some or part of a single component or feature, including additional components or features and / or other modifications, and all such modifications are within the scope of this disclosure. Attached Figure Description

[0032] Figure 1 This is a diagram of an example system architecture for a 3D environment platform that supports automatic skinning migration and rigid automatic skinning, according to some implementation methods.

[0033] Figure 2 An example workflow for adapting a beard accessory to an avatar's head, according to some implementations, is shown.

[0034] Figure 3 Examples of applying different types of beard accessories to different avatars are shown according to some implementations.

[0035] Figure 4 Other examples of applying different types of beard accessories to different avatars are shown according to some implementations.

[0036] Figure 5 The existing and new phases of an accessory migration pipeline according to some implementations are shown.

[0037] Figure 6 An example of matching an accessory mesh to a head mesh using a cage, according to some implementations, is shown.

[0038] Figure 7 Examples of skinning migration masks and / or regions are shown according to some implementations.

[0039] Figure 8 An example of using centroid coordinates when migrating a mesh is shown according to some implementations.

[0040] Figure 9 A pipeline for storing and using grids according to some implementations is shown.

[0041] Figure 10 Another pipeline for storing and using grids according to some implementations is shown.

[0042] Figure 11 Examples of enabling and disabling the function when adapting accessories to an avatar are shown according to some implementations.

[0043] Figure 12 The problematic avatar and the corrected avatar according to some implementations are shown.

[0044] Figure 13 Examples of fitting accessories to a costume are shown according to some implementations.

[0045] Figure 14 Examples of avatars with problematic accessories and avatars with repaired accessories, according to some implementations, are shown.

[0046] Figure 15 Examples of clothing components adapted to an avatar and the correction of such clothing components are shown according to some embodiments.

[0047] Figure 16 An example of a smoothed avatar grid according to some implementations is shown.

[0048] Figure 17 Smoothed and unsmoothed example incarnations are shown according to some implementations.

[0049] Figure 18 Examples of the use of adaptation techniques according to some implementations are shown.

[0050] Figure 19 Other embodiments of the use of adaptation techniques according to some implementations are shown.

[0051] Figure 20 Other embodiments of the use of adaptation techniques according to some implementations are shown.

[0052] Figure 21 Examples of rigid automatic skinning and existing asset skinning according to some implementation methods are shown.

[0053] Figure 22 Other examples of rigid automatic skinning and existing asset skinning according to some implementation methods are shown.

[0054] Figure 23 Other examples of rigid automatic skinning and existing asset skinning according to some implementation methods are shown.

[0055] Figure 24 This is a flowchart illustrating a computer implementation method for performing automatic skinning migration of a 3D avatar according to some embodiments.

[0056] Figure 25 This is a flowchart illustrating a computer implementation method for performing rigid autoskinning of a 3D avatar according to some embodiments.

[0057] Figure 26This is a block diagram illustrating an example computing device according to some implementations. Detailed Implementation

[0058] In the following detailed description, reference is made to the accompanying drawings, which form an integral part of this document. In the drawings, like reference numerals generally identify like parts unless the context otherwise requires. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein. As generally described herein and shown in the drawings, aspects of this disclosure can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are considered herein.

[0059] The embodiments described in this specification, such as "some embodiments," "implementations," and "example embodiments," may include specific features, structures, or characteristics; however, each embodiment does not necessarily include specific features, structures, or characteristics. Furthermore, these phrases do not necessarily refer to the same embodiment. Additionally, when a specific feature, structure, or characteristic is described in connection with an embodiment, whether explicitly stated or not, that feature, structure, or characteristic may be implemented in combination with other embodiments.

[0060] This disclosure primarily relates to a technique for automatically transferring skinning from an avatar's body to an accessory (e.g., a garment component fitted to the avatar's body) so that, when the avatar moves (is animated), the accessory can move in a manner that follows the movement of the avatar's head / body to which it is attached. Furthermore, in some cases, the avatar's body or body part may not have skinning associated with it. Such an avatar's body or body part can be considered rigid. Rigid automatic skinning functions, implemented in various ways, allow skinning to be obtained from the rigid body / part itself and then applied to the accessory.

[0061] The automatic skinning transfer function is characterized by its ability to transfer skin from a target geometry (e.g., head, body, etc.) to the geometry of the accessory being attached, using a layered garment system. For example, when transferring skin, the target head or body may have a pre-existing associated skin that is already available when performing the automatic skinning transfer. However, not all avatar heads and / or bodies have pre-existing associated skins. Therefore, rigid automatic skinning techniques described below can be used to provide such associated skins.

[0062] Automatic skinning migration can be presented to the user via an on / off toggle or other user interface. If this toggle is enabled for any specific accessory, the techniques and implementations described herein can be enabled and applied. This feature can improve the movement and motion performance of accessories within a virtual experience (VE) engine, such as a game engine.

[0063] Without the techniques described herein, accessories embodying a body may not move accurately and / or follow the movements of the head or body to which they are attached. Enabling this feature allows attached accessories to move more accurately (e.g., in a natural way) in accordance with the movements of the head or body to which they are attached. If an accessory initially has no skin to which it is associated, it may not move accurately without the described techniques. However, with this feature enabled, even unskinned accessories can have skin transferred to them. Therefore, such unskinned accessories can move accurately along with the head or body to which they are attached.

[0064] In some implementations, the automatic skinning migration technique described herein enables accessories to deform along with the dynamic head or body to which they are attached. Such implementations can be valuable because automatically deforming accessories (with cages) to match any moving, deforming, or animated skinned head or body when the platform utilizes dynamic heads and bodies can preserve combinatorial aspects of the platform (such as a virtual environment). For example, such preservation could include extending the wrapper deformer frame, thereby also (optionally) migrating skinning weights and skeleton bindings from the head or body to which the accessory is attached to to the accessory in real time at runtime.

[0065] The various implementations described herein provide an automatic skinning migration method that can be integrated as an optional feature into an existing wrap deformer framework. If this optional feature is enabled when applying an accessory, skinning information (weights, bone effects, binding information, etc.) will automatically migrate from the avatar's head or body to the accessory when attaching the accessory using the wrap deformer.

[0066] Subsequently, the automatically skinned mesh is used for rendering and animation within the game engine / virtual experience engine. The automatically skinned mesh allows attached accessories to deform naturally with the avatar as it moves and animates. This approach is a real-time (or near real-time) process applied during the execution of the real-time wrap deformer.

[0067] For example, a user applies a beard accessory to their avatar using a wrap deformer, and the beard does not have an associated skin. In one implementation, skin information is automatically added to the beard's mesh based on existing skin found in the avatar's head and / or body mesh. Therefore, the newly skinned beard can then follow the face as it expresses emotions and itself.

[0068] In this example, facial accessories are applied to a dynamic head. While much of this disclosure relates to facial accessories and dynamic heads, there are no limitations on what accessories can be used, or whether the technique is applied to the avatar's head or body. The embodiments described herein provide a general automatic skinning transfer method that can be applied to head and / or body-related accessories such as shirts, trousers, beards, hair, shoes, eyebrows, etc.

[0069] In some implementations, the head and / or body to which the accessory is attached may have a pre-existing skin. In some implementations, skinning migration can also support unskinned, traditional blocky avatars (such as R15 avatars). In these cases, some implementations may assume that the avatar's head and body parts have some original skinning weights (e.g., 1.0). For example, in some implementations, the original assets are objects rigidly bound to a skeleton. Therefore, their weights will be 1.0, with no weight gradient between parts. In other implementations, the weights can be drawn by an artist using tools to create assets for 3D applications.

[0070] Automatic skinning migration involves a geometric search that uses a wrapper deformer cage to find correspondences. The migration also involves sampling the centroid of the skinning information from the source avatar mesh (i.e., the source of the skinning migration) for each accessory mesh vertex. The result of this process is a new, edited accessory mesh (e.g., a FileMeshData object), which is used in place of the original, unmodified accessory mesh during the rendering phase.

[0071] This new, edited mesh has a different structure than the original mesh. This difference in structure is because new skinning information requires the computation and optimization of new skinning subsets on its surface, which alters the mesh's structure. The number of polygons is the same as the original mesh, but the number of vertices can be significantly different because these new mesh segmentation patterns are part of this new skinned mesh.

[0072] Because of this different structure, a completely new mesh needs to be built and stored in an updated format to allow the mesh structure to be bound to the skin itself. Therefore, the implementation provides a method for storing this edited accessory mesh so that the rendering stage can use it during its mesh preparation techniques.

[0073] Successfully implementing the above changes may also involve some other related technical issues. One issue is integration with the existing wrap deformer framework. The implementation specifies that the skin transfer method is integrated as part of the existing wrap deformer by adding new processing stages that are activated (where appropriate) after the current deformer / solver phase is completed. The wrap deformer's state machine is extended through these additional skin transfer stages. Because these new stages only begin after the wrapping phase is completed, the operation and performance of the existing wrap deformer are not adversely affected.

[0074] The implementation may also provide specific skinning transfer techniques to deliver high-quality results. Once the nearest location on the head or body mesh surface is found for each vertex of the accessory mesh, the implementation can sample skinning information from that mesh using a centroid sampling technique. This information may include which bones in the skeletal rig influence the skinning at that location, and / or the weight of the influence associated with each of those bones. The implementation can use correspondences to calculate skinning weights and may not explicitly store the correspondence information.

[0075] The overall capability of these implementations is to extend existing wrap deformer frames with an optional automatic skinning migration method, wherein skinning information is automatically migrated from the avatar body and / or head to which the accessory is attached. This automatic skinning migration functionality will be integrated with existing wrap deformer frames. The automatic skinning migration functionality will not adversely affect the performance of existing wrap deformer adaptation phases or associated hidden surface removal (HSR) functions.

[0076] Some implementations assume that the head and / or body to which the accessory is attached actually have a pre-existing skin, and such implementations do not support skin migration for unskinned R15 block avatars. However, other implementations do not make this assumption and can handle unskinned avatars.

[0077] When integrated into a game engine (also known as a virtual experience engine), new wrapper deformer skinning migration parameters (flags, enumerations, etc.) are introduced into the WrapTarget instance, and the new internal functionality is initially hidden. The new skinning migration feature is disabled by default and must be explicitly enabled by the user for their specific accessory.

[0078] The automatic skinning migration process runs as a background process alongside the existing wrapper deformer stage. The new skinning takes effect at some point after this process is complete (ideally very quickly). However, during this migration, the avatar itself remains active and interacts with the experience as usual. However, associated accessories may not move correctly (i.e., may become static) until the skinning migration process is complete and the new skinning / edit mesh is applied.

[0079] Another aspect of the implementation is to apply automatic skinning to an unskinned rigid R15 avatar. Some implementations support this technique, providing automatic skinning for these situations. Such implementations support automatic skinning of rigid R15 avatars using this technique. For example, in this case, the implementation first assumes that the skinning weight of the rigid avatar is 1.0 (or other preset weight value) across all body parts. Such implementations perform skinning migration based on this assumption. The implementation also applies smoothing to the final skin to avoid noticeable breaks between different body parts.

[0080] Figure 1 -System Architecture Figure 1 This is a diagram of an example system architecture for a 3D environment platform that supports automatic skinning migration and rigid automatic skinning, according to some implementation methods.

[0081] Figure 1 The same reference numerals are used to identify the same elements as in other figures. The characters following the reference numerals, such as "110", indicate that the text specifically refers to the element with that particular reference numeral. Reference numerals without characters such as "110" in the text refer to any or all elements in the figure that have that reference numeral (e.g., "110" in the text refers to reference numerals "110a", "110b", and / or "110n" in the figure).

[0082] System architecture 100 (also referred to herein as the "system") includes an online virtual experience server 102, a data storage device 120, client devices 110a, 110b, and 110n (generally referred to herein as "client device 110"), and developer devices 130a and 130n (generally referred to herein as "developer device 130"). The virtual experience server 102, data storage device 120, client device 110, and developer device 130 are coupled via network 122. In some implementations, client device 110 and developer device 130 may refer to the same device or devices of the same type.

[0083] The online virtual experience server 102 may include (but is not limited to) a virtual experience engine 104, one or more virtual experiences 106, and a graphics engine 108. In some embodiments, the graphics engine 108 may be a system, application, or module that allows the online virtual experience server 102 to provide graphics and animation capabilities. In some embodiments, the graphics engine 108 may perform the following combinations Figure 2 and Figure 3 The flowchart shown describes one or more operations. Client device 110 may include virtual experience application 112 and input / output (I / O) interface 114 (e.g., input / output device). Input / output device may include one or more of a microphone, speaker, headphones, display device, mouse, keyboard, game controller, touch screen, virtual reality console, etc.

[0084] Developer device 130 may include virtual experience application 132 and input / output (I / O) interface 134 (e.g., input / output device). Input / output device may include one or more of the following: microphone, speaker, headphones, display device, mouse, keyboard, game controller, touch screen, virtual reality console, etc.

[0085] A system architecture 100 is provided for illustration. In different implementations, the system architecture 100 may include components that are compatible with... Figure 1 The same, fewer, more, or different elements configured in the same or different ways as shown.

[0086] In some implementations, network 122 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., Ethernet), a wireless network (e.g., an 802.11 network, a Wi-Fi® network, or a wireless LAN (WLAN)), a cellular network (e.g., a 5G network, a long term evolution (LTE) network, etc.), a router, a hub, a switch, a server computer, or a combination thereof.

[0087] In some implementations, data storage 120 may be a non-transitory computer-readable storage device (e.g., random access memory), a cache, a drive (e.g., a hard disk drive), a flash drive, a database system, or another type of component or device capable of storing data. Data storage 120 may also include multiple storage components (e.g., multiple drives or multiple databases) that can span multiple computing devices (e.g., multiple server computers). In some implementations, data storage 120 may include cloud-based storage.

[0088] In some implementations, the online virtual experience server 102 may include a server with one or more computing devices (e.g., a cloud computing system, rack server, server computer, physical server cluster, etc.). In some implementations, the online virtual experience server 102 may be a standalone system, may include multiple servers, or may be part of another system or server.

[0089] In some implementations, the online virtual experience server 102 may include one or more computing devices (such as rack servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, etc.), data storage (e.g., hard disks, memory, databases), networks, software components, and / or hardware components that can be used to perform operations on the online virtual experience server 102 and provide users with access to the online virtual experience server 102. The online virtual experience server 102 may also include website (e.g., web pages) or application backend software that can be used to provide users with access to content offered by the online virtual experience server 102. For example, a user can access the online virtual experience server 102 using a virtual experience application 112 on a client device 110.

[0090] In some implementations, virtual experience session data is generated via an online virtual experience server 102, a virtual experience application 112, and / or a virtual experience application 132, and stored in a data storage device 120. With the permission of the virtual experience participants, the virtual experience session data may include associated metadata, such as: virtual experience identifier; device data associated with the participant; participant demographic information; virtual experience session identifier; chat logs; session start time, session end time, and session duration for each participant; the relative location of the participant's avatar in the virtual experience environment; purchases made by one or more participants in the virtual experience; accessories used by the participant; and so on.

[0091] In some implementations, the online virtual experience server 102 may be a social network providing connectivity between users or a user-generated content system allowing users (e.g., end users or consumers) to communicate with other users on the online virtual experience server 102. In this user-generated content system, communication may include voice chat (e.g., synchronous and / or asynchronous voice communication), video chat (e.g., synchronous and / or asynchronous video communication), or text chat (e.g., 1:1 and / or N:N synchronous and / or asynchronous text-based communication). Records of some or all user communication may be stored in data storage 120 or within the virtual experience 106. Data storage 120 may be used to store chat logs (text, audio, images, etc.) exchanged between participants.

[0092] In some implementations, chat logs are generated via virtual experience applications 112 and / or 132 and stored in data storage 120. Chat logs may include chat content and associated metadata, such as the chat text content for each message with a corresponding sender and receiver; message formatting (e.g., bold, italics, uppercase, etc.); message timestamps; the relative positions of participants' avatars within the virtual experience environment; accessories used by virtual experience participants; etc. In some implementations, chat logs may include multilingual content, and messages in different languages ​​from different sessions within the virtual experience may be stored in data storage 120.

[0093] In some implementations, chat logs can be stored based on timestamps as a form of conversation between participants. In other implementations, chat logs can be stored based on the message initiator.

[0094] In some embodiments of this disclosure, "user" may refer to a single individual. However, other embodiments of this disclosure encompass "user" as an entity controlled by a group of users or an automated source (e.g., a creative user). For example, a group of individual users united as a community or group in a user-generated content system can be considered "users".

[0095] In some implementations, the online virtual experience server 102 may be a virtual game server. For example, the game server may provide single-player or multiplayer games to a user community that can access the “system” described herein, including the online game server 102, data storage 120, and clients, or interact with the virtual experience via network 122 using client devices 110. In some implementations, for example, the virtual experience (including virtual realms or worlds, virtual games, other computer-simulated environments) may be a two-dimensional (2D) virtual experience, a three-dimensional (3D) virtual experience (e.g., a 3D user-generated virtual experience), a virtual reality (VR) experience, or an augmented reality (AR) experience. In some implementations, users can interact with other users (e.g., playing games). In some implementations, the virtual experience can be experienced in real-time with other users of the virtual experience.

[0096] In some implementations, virtual experience participation can refer to the interaction of one or more participants in a virtual experience (e.g., 106) using a client device (e.g., 110), or the presentation of the interaction on a display or other output device (e.g., 114) of the client device 110. For example, virtual experience participation can include interaction with one or more participants in the virtual experience, or the presentation of the interaction on a display of the client device.

[0097] In some implementations, virtual experience 106 may include electronic files that can be executed or loaded using software, firmware, or hardware for presenting virtual experience content (e.g., digital media projects) to entities. In some implementations, virtual experience application 112 may be executed and combined with virtual experience engine 104 to render virtual experience 106. In some implementations, virtual experience 106 may have a set of common rules or common goals, and the environment of virtual experience 106 may share this set of common rules or common goals. In some implementations, different virtual experiences may have different rules or goals from each other.

[0098] In some implementations, a virtual experience may have one or more environments in which multiple environments can be linked (also referred to herein as a "virtual experience ring" or "virtual environment"). An example of an environment may be a three-dimensional (3D) environment. The one or more environments of virtual experience 106 may be collectively referred to herein as a "world," a "virtual experience world," a "game world," a "virtual world," or a "universe." An example of a world may be a 3D world of virtual experience 106. For example, a user can construct a virtual environment linked to another virtual environment created by another user. Characters in a virtual experience can cross virtual boundaries to enter adjacent virtual environments.

[0099] It should be noted that a 3D environment or 3D world uses graphics that represent a three-dimensional representation of the geometric data representing the virtual experience content (or, regardless of whether a 3D representation of the geometric data is used, at least the virtual experience content is rendered as 3D content). A 2D environment or 2D world uses graphics that represent a two-dimensional representation of the geometric data representing the virtual experience content.

[0100] In some implementations, the online virtual experience server 102 may host one or more virtual experiences 106 and may allow users to interact with the virtual experiences 106 using the virtual experience application 112 on the client device 110. Users of the online virtual experience server 102 can play with, create, interact with, or build virtual experiences 106, communicate with other users, and / or create and build objects of virtual experiences 106 (e.g., also referred to herein as “projects”, “virtual experience objects”, or “virtual experience projects”).

[0101] For example, when generating user-generated virtual projects, users can create characters, character accessories, one or more virtual environments for interactive virtual experiences, or structures used in building virtual experience 106. In some implementations, users can buy, sell, or trade virtual experience objects, such as platform currency (e.g., virtual currency), with other users on online virtual experience server 102. In some implementations, online virtual experience server 102 can transfer virtual experience content to virtual experience applications (e.g., 112). In some implementations, virtual experience content (also referred to herein as "content") can refer to any data or software instructions (e.g., virtual experience objects, virtual experiences, user information, videos, images, commands, media items, etc.) associated with online virtual experience server 102 or virtual experience applications. In some implementations, virtual experience objects (e.g., also referred to herein as "projects," "objects," "virtual objects," or "virtual experience projects") can refer to objects used, created, shared, or otherwise depicted in virtual experience application 106 on online virtual experience server 102 or virtual experience application 112 on client device 110. For example, virtual experience objects may include components, models, accessories, tools, weapons, clothing, buildings, vehicles, currency, flora, fauna, and components of the above (e.g., windows of buildings).

[0102] It should be noted that the online virtual experience server 102 providing the hosting virtual experience 106 is for illustrative purposes. In some embodiments, the online virtual experience server 102 may host one or more media items, which may include communication messages from one user to one or more other users. With user permission and explicit user consent, the online virtual experience server 102 may analyze chat log data to improve the virtual experience platform. Media items may include, but are not limited to, digital videos, digital movies, digital photos, digital music, audio content, melodies, website content, social media updates, e-books, e-magazines, digital newspapers, digital audiobooks, e-journals, web blogs, realsimple syndication (RSS) feeds, e-comic books, software applications, etc. In some embodiments, media items may be electronic files that can be executed or loaded using software, firmware, or hardware used to present digital media items to entities.

[0103] In some implementations, virtual experience 106 may be associated with a specific user or a specific group of users (e.g., a private virtual experience), or it may be widely available to users with access to the online virtual experience server 102 (e.g., a public virtual experience). In some implementations, where the online virtual experience server 102 associates one or more virtual experiences 106 with a specific user or group of users, the online virtual experience server 102 may use user account information (e.g., a user account identifier such as a username and password) to associate the specific user with the virtual experience 106.

[0104] In some implementations, the online virtual experience server 102 or client device 110 may include a virtual experience engine 104 or a virtual experience application 112. In some implementations, the virtual experience engine 104 may be used for the development or execution of a virtual experience 106. For example, the virtual experience engine 104 may include a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, scripting capabilities, an animation engine, an artificial intelligence engine, networking capabilities, streaming capabilities, storage management capabilities, threading capabilities, scene graph capabilities, or animation video support, and other capabilities. Components of the virtual experience engine 104 may generate commands (e.g., rendering commands, collision commands, physics commands, etc.) to aid in the calculation and rendering of the virtual experience. In some implementations, the virtual experience application 112 of the client device 110 may operate independently and / or collaborate with the virtual experience engine 104 of the online virtual experience server 102.

[0105] In some implementations, both the online virtual experience server 102 and the client device 110 can execute virtual experience engines (104 and 112, respectively). The online virtual experience server 102, using virtual experience engine 104, can execute some or all of the virtual experience engine functions (e.g., generating physics commands, rendering commands, etc.), or offload some or all of the virtual experience engine functions to the virtual experience engine 104 on the client device 110. In some implementations, the ratio between the virtual experience engine functions executed on the online virtual experience server 102 and those executed on the client device 110 for each virtual experience 106 can be different. For example, the virtual experience engine 104 of the online virtual experience server 102 can be used to generate physics commands in the event of a collision between at least two virtual experience objects, while additional virtual experience engine functions (e.g., generating rendering commands) can be offloaded to the client device 110. In some implementations, the ratio of virtual experience engine functions executed on the online virtual experience server 102 and the client device 110 can be changed based on virtual experience conditions (e.g., dynamically). For example, if the number of users participating in a particular virtual experience 106 exceeds a threshold, the online virtual experience server 102 can execute one or more virtual experience engine functions previously executed by the client device 110.

[0106] For example, a user can play a virtual experience 106 on client device 110 and send control commands (e.g., user input such as right, left, up, down, user selection, or character position, and speed information) to online virtual experience server 102. After receiving control commands from client device 110, online virtual experience server 102 can send experience commands (e.g., position and speed information of characters participating in a team experience, or commands such as rendering commands, collision commands, etc.) to client device 110 based on the control commands. For example, online virtual experience server 102 can (e.g., using virtual experience engine 104) perform one or more logical operations on the control commands to generate experience commands for client device 110. In other cases, online virtual experience server 102 can pass one or more control commands from one client device 110 to other client devices participating in virtual experience 106 (e.g., from client device 110a to client device 110b). Client device 110 can use the experience commands and render the virtual experience to be displayed on client device 110's screen.

[0107] In some implementations, control instructions may refer to instructions that direct a user character's actions within a virtual experience. For example, control instructions may include user input for controlling actions within the experience, such as right, left, up, down, user selection, gyroscope position and orientation data, force sensor data, etc. Control instructions may include character position and velocity information. In some implementations, control instructions are sent directly to the online virtual experience server 102. In other implementations, control instructions may be sent from client device 110 to another client device (e.g., from client device 110b to client device 110n), where the other client device generates the experience instructions using a local virtual experience engine 104. Control instructions may include instructions for playing voice communication messages or other sounds from another user on an audio device (e.g., a speaker, headphones, etc.), such as voice communication or other sounds generated using audio spatialization techniques as described herein.

[0108] In some implementations, experience instructions can be instructions that enable client device 110 to render a virtual experience (e.g., a multi-participant virtual experience). Experience instructions may include one or more of user input (e.g., control instructions), character position and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.).

[0109] In some implementations, a character (or typically a virtual experience object) consists of components that are automatically linked together to help the user edit them, and one or more of these components can be selected by the user.

[0110] In some implementations, the character is implemented as a 3D model and includes a surface representation (also called skin or mesh) for drawing the character and a set of interconnected layered bones (also called skeleton or rig). This rig is used to animate the character and simulate its movement and actions. The 3D model can be represented as a data structure, and one or more parameters of the data structure can be modified to change various attributes of the character, such as size (height, width, perimeter, etc.); body type; movement style; number / type of body parts; proportions (e.g., shoulder-to-hip ratio); head size, etc.

[0111] One or more characters (also referred to herein as "avatars" or "models") may be associated with a user, who can control the characters to facilitate the user's interaction with the virtual experience 106.

[0112] In some implementations, a character may include components such as body parts (e.g., hair, arms, legs, etc.) and accessories (e.g., T-shirts, glasses, decorative images, tools, etc.). In some implementations, customizable body parts of the character include head type, body part type (arms, legs, torso, and hands), face shape, hairstyle, and skin type. In some implementations, customizable accessories include clothing (e.g., shirts, trousers, hats, shoes, glasses, etc.), weapons, or other tools.

[0113] In some implementations, for certain asset types (such as shirts, trousers, etc.), online virtual experience platforms can provide users with access to simplified 3D virtual object models, which are represented by a grid with a low polygon count (e.g., between about 20 and about 30 polygons).

[0114] In some implementations, the user can also control the character's proportions (e.g., height, width, or depth) or the proportions of the character's parts. In some implementations, the user can control the character's proportions (e.g., blocky, anatomical, etc.). Note that in some implementations, the character may not include a virtual character experience object (e.g., body parts), but the user can control the character (without a virtual character experience object) to facilitate user interaction with the virtual experience (e.g., in puzzle games, no rendered character game objects are shown, but the user can still control the character to control actions in the game).

[0115] In some implementations, components (e.g., body parts) can be primitive geometries such as blocks, cylinders, spheres, etc., or other primitive shapes such as wedges, tori, tubes, channels, etc. In some implementations, the creator module can publish user roles for other users of the online virtual experience server 102 to view or use. In some implementations, users can create, modify, or customize roles, other virtual experience objects, virtual experience 106, or virtual experience environments using I / O interfaces (e.g., developer interfaces) and with or without scripts (or with or without application programming interfaces (APIs)). Note that, for illustration, the role description is humanoid. Also note that roles can be of any form, such as vehicles, animals, inanimate objects, or other creative forms.

[0116] In some implementations, the online virtual experience server 102 may store user-created characters in data storage 120. In some implementations, the online virtual experience server 102 maintains a character directory and a virtual experience directory that can be presented to users. In some implementations, the virtual experience directory includes images of virtual experiences stored on the online virtual experience server 102. Additionally, users can select characters (e.g., characters created by the user or other users) from the character directory to participate in selected virtual experiences. The character directory includes images of characters stored on the online virtual experience server 102. In some implementations, one or more characters in the character directory may have been created or customized by the user. In some implementations, the selected character may have character settings that define one or more components of that character.

[0117] In some implementations, a user's role may include component configurations, wherein the configuration and appearance of the components (more generally, the appearance of the role) can be defined by role settings. In some implementations, the role settings of a user's role may be at least partially selected by the user. In other implementations, a user may select a role with default role settings or role settings selected by other users. For example, a user may select a default role from a directory of roles with predefined role settings, and the user may further customize the default role by changing some role settings (e.g., adding a shirt with a custom logo). Role settings can be associated with specific roles via an online virtual experience server 102.

[0118] In some embodiments, client devices 110 may each include computing devices (e.g., personal computers (PCs)), mobile devices (e.g., laptops, mobile phones, smartphones, tablets, or netbooks), network-connected televisions, game consoles, etc. In some embodiments, client devices 110 may also be referred to as "user devices." In some embodiments, one or more client devices 110 may connect to the online virtual experience server 102 at any given time. It should be noted that the number of client devices 110 is provided for illustrative purposes. In some embodiments, any number of client devices 110 may be used.

[0119] In some implementations, each client device 110 may include an instance of a virtual experience application 112. In one implementation, the virtual experience application 112 may allow a user to use and interact with an online virtual experience server 102, such as controlling virtual characters in a virtual experience hosted by the online virtual experience server 102, or viewing or uploading content such as virtual experience 106, images, video items, web pages, documents, etc. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, render, or navigate content served by a web server (e.g., virtual characters in a virtual environment). In another example, the virtual experience application may be a native application (e.g., a mobile application, app, virtual experience program, or game program) that is installed and executed locally on the client device 110 and allows the user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content (e.g., a web page, a media viewer) to the user. In implementations, the virtual experience application may also include an embedded media player (e.g., a Flash® or HTML5 player) embedded in a web page.

[0120] According to various aspects of this disclosure, a virtual experience application can be an online virtual experience server application used by users to build, create, edit, upload content to the online virtual experience server 102, and interact with the online virtual experience server 102 (e.g., participate in a virtual experience 106 hosted by the online virtual experience server 102). Thus, the virtual experience application can be provided to the client device 110 by the online virtual experience server 102. In another example, the virtual experience application can be an application downloaded from a server.

[0121] In some implementations, each developer device 130 may include an instance of a virtual experience application 132. In one implementation, the virtual experience application 132 may allow developer users to use and interact with an online virtual experience server 102, such as controlling virtual characters in a virtual experience hosted by the online virtual experience server 102, or viewing or uploading content such as virtual experience 106, images, video items, web pages, documents, etc. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, render, or navigate content served by a web server (e.g., virtual characters in a virtual environment). In another example, the virtual experience application may be a native application (e.g., a mobile application, app, virtual experience program, or game program) that is installed and executed locally on the developer device 130 and allows the user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content (e.g., a web page, a media viewer) to the user. In implementations, the virtual experience application may also include an embedded media player (e.g., a Flash® or HTML5 player) embedded in a web page.

[0122] According to various aspects of this disclosure, virtual experience application 132 can be an online virtual experience server application for users to build, create, edit, upload content to online virtual experience server 102, and interact with online virtual experience server 102 (e.g., providing and / or participating in virtual experiences 106 hosted by online virtual experience server 102). Thus, the virtual experience application can be provided to client device 130 by online virtual experience server 102. In another example, game application 132 can be an application downloaded from the server. Virtual experience application 132 can be used to interact with online virtual experience server 102 and gain access to user credentials, user currency, etc., of one or more virtual experiences 106 developed, hosted, or provided by virtual experience developers.

[0123] In some implementations, users can log in to the online virtual experience server 102 through a virtual experience application. Users can access their accounts by providing user account information (e.g., username and password), where the user account is associated with one or more roles that can be used to participate in one or more virtual experiences 106 on the online virtual experience server 102. In some implementations, using appropriate credentials, virtual experience developers can gain access to virtual objects within the virtual experience, such as in-platform currencies (e.g., virtual currency) owned or associated with other users, avatars, special abilities, and accessories.

[0124] Typically, if appropriate, functions described in one embodiment as being performed by the online virtual experience server 102 can also be performed by the client device 110 or the server in other embodiments. Furthermore, functions belonging to a particular component can be performed by different components or multiple components operating together. The online virtual experience server 102 can also be accessed as a service provided to other systems or devices through a suitable application programming interface (API), and is therefore not limited to use on a website.

[0125] Figure 2 - Workflow for adapting beard accessories Figure 2 An example workflow 200 for adapting a beard accessory to an avatar's head according to some implementations is shown.

[0126] Accessories can include inner and outer cages that enclose deformers. For catalog pattern avatars (e.g., officially generated avatars), their cages (at least at their vertices) match a fixed topology. Catalog pattern avatars may be created for a specific platform and follow that platform's official pattern. These requirements must be met for an avatar to be uploaded to the platform's official avatar database. If these criteria are not met, the avatar will not be able to be purchased, sold, or used in all experiences on the platform. Here, "pattern" refers to the avatar specification (e.g., R15 has 15 mesh parts, etc.). Such cages use a standard UV layout. Here, U and V can represent axes of a two-dimensional (2D) texture, which are used when projecting between a 3D model surface and a 2D image for texture mapping. Other assets may allow arbitrary cages and UV layouts. In some implementations, the specification requires avatars to be pre-skinned and include skeleton rigging. Some implementations may specify that pre-existing skinning is not required for functionality that generates skin from a rigid (unskinned) avatar.

[0127] To provide sufficient information, head / body assets include an outer cage that wraps the deformer. Cages of catalog pattern avatars (at least at the vertices of such cages) match a fixed topology. For example, such cages might use a standard UV layout. Other assets may allow arbitrary cages and arbitrary UV topologies. Some head / body assets may be pre-skinned and may include skeleton rigging. The number and names of joints on body assets can also be predefined. The number and names of joints on the head can be arbitrary. For the head, these joints can be controlled and driven by a facial action coding system (FACS).

[0128] Figure 2A graphic of the beard accessory 202 is shown. This graphic is associated with the initial beard accessory information 204. The initial beard accessory information 204 includes the beard accessory's rendered mesh and its wrapper deformer cage. However, the initial beard accessory information 204 does not include any skinning / binding information. This information is required when fitting the accessory to the avatar's head.

[0129] Figure 2 A graphic of the avatar head 208 is also shown. The graphic of the avatar head 208 is associated with the initialized avatar head / body information 210. Although in relation to the avatar head... Figure 2 The embodiment shown depicts a graphic of the avatar head 208, but other embodiments may also operate in the context of the entire avatar or other body parts of the avatar. Initializing avatar head / body information 210 includes the avatar head / body rendering mesh, the avatar head / body wrapper deformer cage, and the avatar head / body skinning / binding information.

[0130] Initial beard accessory information 204 and initial head / body information 210 are provided to the wrap deformer application 206. The wrap deformer application 206 analyzes the initial head / body information 210 and performs an automatic skinning migration operation 212.

[0131] Based on the initial beard accessory information 204 and the automatic skinning migration operation 212, an alternative mesh 214 for the beard accessory is generated. The alternative mesh 214 for the beard accessory includes a rendered mesh and a wrapper deformer cage (i.e., from the initial beard accessory information 204). However, the alternative mesh for the beard accessory 214 additionally includes skinning / binding information based on the automatic skinning migration operation 212.

[0132] Once an alternative mesh for the beard accessory 214 is generated, the beard accessory 214 can be applied and automatically skinned onto various avatars. For example, Figure 2 The beard applied and automatically skinned onto the first avatar 216 is shown. Figure 2 The beard shown is automatically skinned onto the first avatar 216 and can be obtained from the initial avatar head / body information 210. The avatar head 216 is shown in two poses—a neutral pose (left) and a smiling pose (right), showing that when the avatar head is animated, the accessory (beard) can be animated using an alternative mesh 214 of the beard accessory.

[0133] Figure 2The illustration also shows a beard applied and automatically skinned onto a second avatar 218 (left) and the same avatar's head in different poses (right), where the beard is automatically animated to match the different poses. Although not shown, for the beard applied and automatically skinned onto the second avatar 218, the second avatar may use additional avatar head / body information. These depictions of the beard illustrate how the techniques provided in this disclosure can improve the quality of applied accessories.

[0134] Figure 3 -Examples of beard accessories Figure 3 Examples 300 are shown illustrating the application of different types of beard ornaments to different avatars according to some embodiments. For example, Figure 3 An avatar named Blocky is shown, with different beards applied—Blocky skinned beard 302 applied to Blocky and Roxie skinned beard 304 applied to Blocky. Figure 3 It also shows an avatar named Roxie, with Blocky skinned beard 306 applied to Roxie and Roxie skinned beard 308 applied to Blocky.

[0135] therefore, Figure 3 This demonstrates how accessories, when skinned to fit the avatar to which they are attached, look natural and animate with the avatar (e.g., rotation or other movements of the avatar's head, expressions formed by different facial movements, etc.). This is the case, for example, with Blocky skinned beard 302 applied to Blocky and Roxie skinned beard 308 applied to Blocky, and the beards fit appropriately (correctly attached to the face without gaps, moves correctly with the face, and are sized to cover the appropriate portion of the face, etc.) to fit the underlying avatar. In contrast, the beards in Roxie skinned beard 304 and Roxie skinned beard 306 applied to Blocky look less desirable because the beards are adapted for different avatars. For example, because the Blocky skinned beard is designed for male avatars while the Roxie skinned beard is designed for female avatars, there may be adaptation differences. Objects created for the original avatar always look better than those on the migration target. While they may still function correctly, they may not achieve the optimal aesthetic effect.

[0136] Figure 4 - More examples of fur-trimmed beard accessories Figure 4 Other examples 400 are shown, according to some embodiments, of applying different types of beard ornaments to different avatars. For example, Figure 4 The first avatar head 402 is shown. The avatar head 402 includes various head accessories, such as hair, a mustache, and a beard. Figure 4The second incarnation head 404 is also shown. Although the shape of the second incarnation head 404 is significantly different from that of the first incarnation head 402, the second incarnation head 404 has head accessories similar to those of the first incarnation head 402.

[0137] Figure 4 The results of the underlying technique proposed in this paper are shown. For example, the implementation can extend the wrap deformer to automatically migrate the skin from the head / body to the accessory. An alternative wrap deformer state machine is extended with a new skin migration stage. This technique uses a cage-based geometric search to find the nearest position on the head for each accessory vertex.

[0138] This approach significantly improves performance by using Kd-trees in the search. Kd-trees, or k-dimensional trees, are data structures whose performance can be improved in several ways. Kd-trees divide the record set into smaller groups based on proximity in space. This allows for efficient searching of points within a given radius or bounding box, known as range queries. Kd-trees can also perform efficient k-nearest neighbor (KNN) queries, which are useful in applications such as image recognition and recommender systems.

[0139] A Kd-tree can be divided into two sets: tree nodes and buckets. Frequently reused tree nodes can be cached to speed up searches, while buckets can be organized in external memory for efficient access. Other memory optimization techniques include adding write and read caches to convert random access into sequential access, and interleaving tree building and searching to reduce redundant accesses.

[0140] The process also includes performing basic skinning, involving the influence of weights and bones on the sampling process. To integrate these techniques into the existing MeshPart structure, the application programming interface (API) used for rendering meshes can be extended to hold alternative meshes for rendering. The technique also processes hidden surface resurfacing (HSR) data and computes alternative face-visibility data.

[0141] Figure 5 - Stages of the accessories migration production line Figure 5 Existing and new stages of an accessory migration pipeline 500 according to some embodiments are shown. For example, an existing stage 502 may exist in the accessory migration pipeline. For example, existing stage 502 may begin with invalidation, where it is determined that an accessory needs to be migrated.

[0142] The pipeline can continue executing the cage acquisition operation, followed by cage acquisition, the solve operation, the solve operation, and updating the dynamic model. Once these existing stages 502 are completed, block 524 determines whether to migrate the skin. If not, block 524 is followed by block 526, and the accessory migration pipeline is complete.

[0143] If skinning migration is to be performed, a new skinning migration phase (according to the implementation described herein) begins after block 524, starting from block 504. In block 504, the mesh acquisition operation of block 506 is activated. The mesh acquisition operation of block 506 is activated by asynchronously acquiring the render mesh 514, for example, from a download operation or cache retrieval.

[0144] Block 506 can be followed by block 508, where the migration phase implemented by migration operation 510 occurs. Migration operation 510 is implemented through asynchronous application of skinning migration technique 516. Migration operation 510 is followed by block 512, which includes dynamic model (DM) update operation 522. Update DM operation 522 may include asynchronously storing / caching the new mesh with skinning. After update DM 522 is completed, the new skinning migration is complete, and the updated model can be used to display the avatar with the applied accessories and to animate the avatar.

[0145] Figure 6 - Matching the accessory grid to the head grid Figure 6 Example 600 is shown, which uses a cage to match an accessory mesh to a head mesh according to some implementations. Figure 6 Examples include an accessory grid 602 for the avatar's head. The accessory grid 602 for the head includes various hair sections that can cover the avatar's head. For example, the accessory grid 602 for the head includes hair, eyebrows, sideburns, a mustache, and a beard. However, these are just examples; other accessory grids may include fewer or more sections.

[0146] In addition to the head accessory mesh 602, there is a corresponding accessory inner cage 606. A geometric search operation 604 can be used to associate the head accessory mesh 602 with the corresponding accessory inner cage 606. There is also a corresponding head outer cage 610. To establish the correspondence between the accessory inner cage 606 and the head outer cage 610, a UV matching operation 608 is performed. The head outer cage 610 corresponds to the head mesh 614. Another geometric search 612 can be used to find the correspondence between the head outer cage 610 and the head mesh 614.

[0147] Figure 7 - Allows skin migration mask / area Figure 7Figure 700 illustrates examples of permitted skin transfer masks and / or regions according to some embodiments. For example, various different skin transfer masks and / or regions may be shown in Figure 700. For instance, a global texture map 702 may be provided. Such a texture map can facilitate skin transfer. However, skin transfer may work in conjunction with portions of the global texture map 702. For example, there may be a head portion 704, a lower torso and leg portion 706, and / or an upper torso and arm portion 708.

[0148] Figure 8 - Use barycentric coordinates when migrating the mesh Figure 8 An example of using barycenter coordinates when migrating grid 800 is shown according to some implementations. For example, Figure 8 A head mesh with a covering cage 804 is shown. Accessory vertices 802 may exist mapped to the cage to identify the nearest point 806 on the head mesh, wherein the nearest point 806 on the head mesh is selected as the point on the head mesh closest to the mapped accessory vertex 802. Such a nearest point is used for centroid sampling 808. For example, centroid sampling 808 may include three vertices a, b, and c.

[0149] Each vertex is associated with the influence of the first four bones, which together provide up to 12 individual bone weights. These weights allow point mapping using centroid coordinates. Such centroid sampling 808 produces high-quality results for associating the accessory vertex 802 mapped to cage 804 with the nearest point 806 on the head mesh.

[0150] Figure 9 -Example pipeline for storing / using the grid Figure 9 A pipeline 900 for storing and using meshes according to some embodiments is shown. Such a pipeline illustrates some problems that may arise with some embodiments. For example, the pipeline may include two main parts: a wrapper deformer frame 902 and a rendering frame 904. The wrapper deformer frame 902 deforms the object before rendering. The rendering frame 904 acquires object information and renders the object graphically.

[0151] For example, the wrap deformer framework 902 can begin with invalidation 912. Invalidation 912 causes the mesh of a given object to be updated before rendering using the wrap deformer framework 902. For example, there could be a wrap deformer phase 906 triggered by invalidation 912. After the wrap deformer phase 906 is the render mesh fetch operation 908. The render mesh fetch operation 908 involves finding meshes for one or more accessories, heads, bodies, etc. These meshes are retrieved from MeshContentProvider 916. MeshContentProvider 916, in turn, retrieves meshes from the content distribution network (CDN) 914.

[0152] Once the meshes are retrieved, they can be passed to the skin migration stage 910. The skin migration stage 910 migrates the skin to the appropriate MeshPart instance 918. The MeshPart instance 918 may include alternative FileMeshData 920 received from the skin migration stage 910.

[0153] Another invalidation 922 may exist, which causes the MeshPart instance 918 to be sent to the rendering framework 904, specifically FastCluster 924, which is the initial part of the avatar rendering system that performs accelerated clustering. FastCluster 924 performs its tasks and provides its results to GeometryGenerator 926. GeometryGenerator 926 provides its output to fetchResources 928. fetchResources 928 requests information from MeshContentProvider 932, which in turn requests information from CDN 930.

[0154] Figure 9 This explains the issue of using an internal cache held by the MeshPart instance instead of the mesh. Figure 9In the example, such alternative meshes are not reflected in the network or saved with the scene. Therefore, it is valuable to modify the `fetchResources` method associated with `GeometryGenerator` to find and use alternative meshes (if available). This technique is somewhat similar to how edited CSG meshes are handled. In this example, cached edited meshes are held internally as a `PartOperator` instance. These CSG meshes are reflected in the network and saved with the scene. The same `fetchResources` method of `GeometryGenerator` retrieves and uses these cached meshes on the `PartOperator` instance.

[0155] Figure 10 -Example pipeline for storing / using the grid Figure 10 An example pipeline 1000 for storing and using a grid according to some implementations is shown. Figure 10 and Figure 9 Similarly, but including the adaptation of managing alternative meshes. For example, the pipeline may include two main parts: a wrapper deformer frame 1002 and a rendering frame 1004.

[0156] For example, the wrap deformer framework 1002 can begin with invalidation 1012. Invalidation 1012 causes the mesh of a given object to be updated before rendering using the wrap deformer framework 1002. For example, there may be a wrap deformer phase 1006 triggered by invalidation 1012. After the wrap deformer phase 1006 is the operation of fetching the render mesh 1008. Fetching the render mesh 1008 involves finding meshes for one or more accessories, heads, bodies, etc. These meshes are retrieved from MeshContentProvider 1016. Mesh content provider 1016 in turn retrieves meshes from content delivery network (CDN) 1014.

[0157] Once the meshes are retrieved, they can be passed to the skin migration stage 1010, which migrates the skin to the appropriate MeshPart instance 1018, which includes the alternative FileMeshData 1020 received from the skin migration stage 1010.

[0158] There is another invalidation 1022, which causes the MeshPart instance 1018 to be sent to the rendering framework 1004, specifically FastCluster 1030, which is the initial part of the rendering system (such as...). Figure 9 The FastCluster 1030 performs its tasks and provides the results to the GeometryGenerator 1032 (as described above). Figure 9 (As mentioned above). GeometryGenerator 1032 provides its output to fetchResources 1034. fetchResources 1034 requests information from MeshContentProvider 1028.

[0159] However, Figure 10 It demonstrates the ability and organization to manage alternative grids, which is consistent with Figure 9 Different. Figure 10 In the skinning migration phase 1010, the skinning FileMeshData 1024 is also provided to ContentProvider 1026. ContentProvider 1026 includes a data repository for storing temporary files, such as a cache. ContentProvider 1026 interacts with MeshContentProvider 1028. ContentProvider 1026 provides information to MeshContentProvider 1028, which may have a least recently used (LRU) cache (or other suitable cache). The LRU cache provides data to fetchResources 1034. However, if an alternative mesh is not yet stored in the LRU cache, MeshContentProvider 1028 can access the alternative mesh from the data repository provided by ContentProvider 1026 for storing temporary files. Therefore, Figure 10 This demonstrates how ContentProvider 1026 and MeshContentProvider 1028 can work together to successfully allow access to the alternative mesh.

[0160] therefore, Figure 10 The use of a mesh content cache is illustrated, which stores auto-skinned accessory meshes for at least one of adaptation or animation. The mesh content cache can be coupled to a file repository. Figure 10 Some of the implementations shown determine that the auto-skinned accessory mesh does not exist in the mesh content cache. In response to determining that the auto-skinned accessory mesh does not exist in the mesh content cache, the auto-skinned accessory mesh is retrieved from the file repository, the retrieved auto-skinned accessory mesh is stored in the mesh content cache, and after storage, the auto-skinned accessory mesh is accessed from the mesh content cache.

[0161] Therefore, some implementations involve adding API options to existing application programming interfaces (APIs), such as MeshPart instances, which allow mesh part instances to hold an alternative Mesh-ID and an associated hash value (uniquely identifying the alternative Mesh-ID). Then, if an alternative Mesh-ID exists in the MeshPart instance, this alternative Mesh-ID is accessed during the rendering phase and replaced with the original Mesh-ID. The newly edited / skinned mesh itself is provided to and stored in the content provider (CP), and accessed during the rendering phase via the MeshContentProvider (MCP) using the new alternative Mesh-ID.

[0162] The skinning migration phase provides a new FileMeshData object to the CP. The CP caches the mesh to local temporary storage (e.g., disk storage) and returns a new Content-ID. This new Content-ID can be provided as a new alternative Mesh-ID to the MeshPart instance. Then, when the rendering phase (i.e., FastCluster or GeometryGenerator) attempts to access the mesh via the MCP using this new alternative Mesh-ID, the MCP determines that the mesh is not present in its Least Recently Used (LRU) cache (memory cache) (or any type of cache). The MCP then requests the CP to provide the mesh to the MCP.

[0163] The CP then reads the alternative mesh from its temporary disk storage and provides it to the MCP. The MCP then puts the alternative mesh into its LRU cache for fast access in the future. The MCP then provides the alternative mesh to the rendering stage. If the MCP subsequently evicts the mesh from its LRU cache due to memory limitations, and the rendering stage subsequently requests the mesh again, the mesh is pulled from the temporary disk storage by the CP.

[0164] The advantage of this framework for storing editable / skinned meshes is that it allows the mesh to be evicted from memory when necessary. This approach maintains a low overall memory usage for the implementation. However, it still ensures that the mesh is available for subsequent use in the rendering stage when requested.

[0165] Figure 11 - Enable / disable accessory adaptation function Figure 11 Examples of enabling and disabling functions when adapting accessories to avatar 1100 are shown according to some implementations. For example, Figure 11 An image of an avatar with a beard and mustache is shown. A pane can exist to the right of the avatar image. This pane can provide the user with information about the avatar and compatible features.

[0166] To enable these automatic skinning migration options for users, the implementation adds some new parameters to the WrapLayer instance (object). The WrapLayer object defines the inner and outer surfaces of the 3D accessory, as well as other properties associated with the layered accessory. Therefore, this instance (WrapLayer) controls the parameters of the accessory (any accessory bound to the target). Thus, this component guides the automatic skinning migration process.

[0167] The new WrapLayer parameter includes AutoSkin, which is an enumeration (i.e., a variable with a list of items). This new enumeration parameter determines whether the automatic skinning option is enabled. This parameter further determines whether, if enabled, any existing skinning found on the accessory is allowed to be overridden. In some implementations, the AutoSkin enumeration selection is Disabled, Enabled (Preserve), and Enabled (Override).

[0168] In some implementations, AutoSkin may default to Disabled. Therefore, users may need to explicitly enable AutoSkin migration for a given geometry (i.e., an selective incorporation method). If the Enabled (Preserve) option is selected, AutoSkin migration only takes effect if the associated accessory has no existing skinning information associated with it. If Enabled (Override) is selected, any existing skinning on the accessory is ignored (i.e., discarded). AutoSkin migration then builds a new skinning for that asset.

[0169] Another additional parameter is AutoSkinZone (also an enumeration variable). This AutoSkinZone parameter defines the avatar parts from which migration is allowed. Examples of values ​​for this enumeration include All (the default), Head, Body, Lower Torso and Legs, Upper Torso and Arms, Feet, and Hands. For example, if the user selects All, skin migration from any part of the avatar is allowed. All is the default value.

[0170] If the user selects Head instead, skinning is migrated only from the head mesh. The Body option allows migration from any body part, except the head itself. Other enumeration options, self-explanatory, restrict migration to only those parts of the avatar. Note that this AutoSkinZone enumeration parameter is merely one way to identify the source of skinning migration. AutoSkinZone can be replaced by another mechanism that allows the user to explicitly specify the areas or parts where skinning migration is permitted.

[0171] Some implementations may utilize more parameters, including possible smoothing parameters. However, the two parameters discussed here (AutoSkin and AutoSkinZone) control whether automatic skin migration is enabled. These parameters also control whether, when automatic skin migration is enabled, automatic skin migration is allowed to overlay any existing skin found on the associated mesh.

[0172] For example, in Figure 11 The pane shown may contain a descriptor 1102 indicating the avatar name, such as RoxieMaleShortBeard. There may also be a properties window 1104, which lists Properties-WrapLayer RoxieMaleShortBeard. There may also be a performance control 1106 for enabling and disabling automatic skinning. For example, performance control 1106 includes AutoSkin and AutoSkinZone, as described above. Performance control 1106 displays a drop-down menu where the value of AutoSkin is selected from Disabled, Enabled (Override), and Enabled (Preserve) as Enabled (Override).

[0173] Although Figure 11 An example user interface for managing the automatic skinning migration function is shown, but Figure 11 This is just an example, and other user interfaces can be used for this purpose. There could also be user interfaces that allow users to make other user interface choices. For example, a user could modify AutoSkinZone to influence which parts of the avatar participate in automatic skinning. Users could also manage other aspects and functions, such as rigid automatic skinning.

[0174] Figure 12 -Repair Avatar Figure 12A problematic avatar and a corrected (restored) avatar 1200 according to some embodiments are illustrated. For example, a first avatar 1202 and a second avatar 1204 each present an avatar of the same female character, wearing a sweater and trousers. However, the first avatar 1202 includes visible imperfections because gaps and other irregularities exist in the trousers worn by the first avatar 1202 (the avatar body is connected to the trousers accessory). However, the technique described herein fills the gaps in the trousers of the second avatar 1204, thereby producing a corrected avatar (an avatar whose clothing or other accessories are properly rendered). Therefore, the technique described herein solves these problems.

[0175] Figure 12 The correction performed is an example of Hidden Surface Removal (HSR). HSR defines additional data to be associated with the accessory mesh. For example, this additional information may include a per-polygon face visibility flag (e.g., true / false). This information may be held by the BaseWrap instance and provided to the GeometryGenerator to determine which polygons to render. Because the implementation is rendering a new alternative mesh, the order of the polygon visibility flags may not match.

[0176] Therefore, for automatically skinned meshes, some implementations can calculate and use an index mapping between the original polygons and the new polygons. Some implementations use this mapping to form a new face visibility vector. This new vector can be stored as mesh data along with the alternative mesh. If this vector exists in the mesh data of the alternative mesh, the GeometryGenerator can be modified to find and use this face visibility vector.

[0177] Figure 13 - Adapt accessories to the avatar Figure 13 An example 1300 of adapting accessories to an avatar according to some embodiments is shown. For example, an avatar 1302 without accessories may exist. Such an avatar can be modified by adding accessories, as shown in an avatar 1304 with accessories. The avatar 1304 with accessories can also be successfully animated to show an avatar 1306 with a pose. Adapted accessories also allow the avatar's head to display different facial expressions. For example, the avatar's head may be an avatar head 1308 yawning with eyes closed. The avatar's head may also be an avatar head 1310 with a calm expression.

[0178] Figure 14 -The problematic incarnation and the repaired incarnation Figure 14Examples 1400 are shown, according to some embodiments, of avatars with problematic accessories and avatars with repaired accessories. For example, there are three avatars 1402 and three avatars 1404. For example, the first avatar in each group could be a goblin monster, the second avatar could be a blocky avatar, and the third avatar could be a robot. In the first three avatars 1402, the avatars all lean back and extend their arms. In the latter three avatars 1404, the avatars stand in a more conventional posture.

[0179] therefore, Figure 14 Three different avatars are shown, each in the same animated pose, with different layered clothing adapters applied and automatically skinned. These avatars have unskinned, rigid-type bodies. However, as... Figure 14 As shown, the automatically skinned clothing components accurately follow the avatar's movements. As described below, some embodiments can utilize technology to perform rigid automatic skinning. This functionality allows even rigid avatars to utilize automatic skinning migration technology.

[0180] Figure 15 - Adaptation and calibration of clothing components Figure 15 Example 1500 of a costume component adapted to an avatar and for adjusting the costume component, according to some embodiments, is shown. For example, Figure 15 A set of clothing components (i.e., top and bottom) and the corresponding avatar 1502 are shown.

[0181] Figure 15 This illustrates how clothing parts are adapted to the avatar in two poses, 1504 and 1506. The clothing parts are adapted to pose 1504. Therefore, pose 1504 demonstrates an efficient way to adapt clothing to the avatar. In pose 1506, the clothing parts are also adapted to the avatar, but the avatar assumes a different pose. The change in pose from 1504 to 1506 illustrates that adjusting accessories as the avatar moves can contribute to the avatar's animation.

[0182] Figure 15 Rigid auto-skinning migration techniques are illustrated. These techniques allow auto-skinning to be applied to unskinned rigid body parts, such as parts of an R15 body. Most avatar bodies in a virtual environment are likely rigid bodies, such as avatars with an R15 body. It can be helpful to identify whether a body part target is rigid or non-rigid.

[0183] However, rigid body parts still have an associated skeleton and / or bones in the game engine. Rigid auto-skinning technology can perform an existing correspondence search until a body part is found. If the nearest body part is already skinned, the vertices at that location (e.g., vertices of the relevant mesh corresponding to the body part) can be sampled as described in this disclosure. If the nearest body part is rigid, a weight of 1.0 or 100% (or other weights corresponding to rigid body parts) can be assigned to that vertex.

[0184] Figure 16 - Smooth the mesh Figure 16 An example 1600 of a smoothed avatar mesh according to some implementations is shown. When using rigid autoskinning, the skinning weights at body joints may abruptly change. Such abrupt changes may occur because the implementation only sets a weight of 100% (or other preset value) for vertices associated with the rigid body.

[0185] Therefore, to smooth abrupt changes in the skin at joints, implementations can apply a smoothing step. In such a smoothing step, the smoothing process iterates over each rigid vertex and takes the average weight of that vertex with its neighboring vertices. Such smoothing iterations can be performed multiple times. For example, some implementations perform the above smoothing iterations three times. However, other implementations may perform more or fewer smoothing iterations.

[0186] For example, Figure 16 The various stages of smoothing are shown. Figure 16 The smoothing process is shown starting with the rigid skin at 1610. Next, a smoothing step occurs at 1620. This shows that the original rigid shape subsequently becomes less angular and blocky. Then, after three steps, the smoothed skin is achieved at 1630. This shows that the rigid skin at 1610 is already quite smooth at 1630. While more smoothing iterations can be performed, this is not necessary for obtaining good results; usually, three smoothing steps are sufficient to achieve satisfactory results.

[0187] Figure 17 - Sample incarnations with and without smoothing Figure 17 Example avatars 1700, smoothed and unsmoothed according to some implementations, are shown. In 1702, the avatar is unsmoothed and has poor visual quality. In 1704, the avatar is smoothed, resulting in higher visual quality.

[0188] Figure 18 -Example results using adaptation techniques Figure 18An example of an avatar using an adaptation technique is shown in some implementations. For example, in 1802, there are a total of eight avatars, including a blocky avatar, a first bald avatar, a second bald avatar in the front row (from left to right), and a goblin, a minotaur, a blocky avatar, a one-eyed avatar, and a third bald avatar in the back row (from left to right).

[0189] exist Figure 18 In this game, each avatar is equipped with the same set of accessories—a dark top, jeans, and sneakers. Each of these accessories is adapted to a corresponding avatar with a different body and head. Therefore, Figure 18 This demonstrates that while a wide variety of avatars exist, and not all of them necessarily include skinning information, the dark plaid blouse and jeans fit each avatar well. Furthermore, each avatar has a similar pose, with their right hand resting on their hip and their left arm extended. Each avatar leans to the left. Additionally, the technique preserves some differences between avatars, as some avatars have different undershirts and different shoes under their plaid blouses. Moreover, these avatars can be animated, with accessories moving accordingly with the avatar's body and head.

[0190] Figure 19 - Other example results avatars using adaptation techniques Figure 19 Examples 1900 of other embodiments using adaptation techniques according to some implementations are shown. Figure 19 It is shown in 1902 with Figure 18 Similar incarnations, but wearing different accessories and in different poses. For example, in Figure 19 In the middle, the incarnations place their hands behind their heads, legs together, facing the left side of the incarnation. Figure 18 The clothing parts shown are different. Figure 19 Each avatar was dressed in a light-colored jacket and light-colored pants.

[0191] Figure 20 - Other example results avatars using adaptation techniques Figure 20 The use of adaptation technology according to some implementations is illustrated. other Example 2000. Figure 20 It shows the relationship with Figures 18 to 19 Similar incarnations, but still wearing different accessories and still in different poses. For example, in Figure 20 In the middle, the figure extends its arms and leans to the left. (And...) Figures 18 to 19 The clothing parts shown are different. Figure 20 Each avatar was paired with a tie-dye shirt and a mini skirt.

[0192] Figure 21 - Rigid automatic skinning and existing asset skinning Figure 21 Examples 2100 of rigid auto-skinning and existing asset skinning according to some implementations are shown. For example, an avatar 2102 can be constructed using a rigid auto-skinning method. Another avatar 2104 can be constructed using existing asset skinning (pre-skinning). Figure 21 The use of the rigid automatic skinning method in China shows that the quality of Avatar 2102 is higher than that of Avatar 2104.

[0193] Figure 22 - Other rigid automatic skinning and existing asset skinning Figure 22 Other examples 2200 of rigid auto-skinning and existing asset skinning according to some implementations are shown. An avatar 2202 is constructed using the rigid auto-skinning method. Another avatar 2204 is constructed using existing asset skinning (pre-skinning). Figure 22 The use of the rigid automatic skinning method in China shows that the quality of Avatar 2202 is higher than that of Avatar 2204.

[0194] Figure 23 - Other rigid automatic skinning and existing asset skinning Figure 23 Other examples 2300 of rigid auto-skinning and existing asset skinning according to some implementations are shown. An avatar 2302 is constructed using the rigid auto-skinning method. Another avatar 2304 is constructed using existing asset skinning (pre-skinning). Figure 23 The use of the rigid automatic skinning method in China shows that the quality of Avatar 2302 is higher than that of Avatar 2304.

[0195] Figure 24 - Perform automatic skinning migration Figure 24 This is a flowchart illustrating a computer implementation method 2400 for performing automatic skinning migration of a 3D avatar according to some embodiments. Method 2400 may begin at block 2402.

[0196] In box 2402, skinning information for the avatar body is obtained. For example, a virtual environment may provide such skinning information as part of a mesh deformer framework that works with the rendering framework. Skinning information may include at least one of the following: weights, bone effects, rigging information, joints, other features of the avatar body, and combinations thereof. Box 2402 may be followed by box 2404 or box 2406.

[0197] In box 2404, automatic skinning can be performed. This type of automatic skinning involves obtaining a rigid avatar and inferring the skin through rigid automatic skinning. Automatic skinning is an optional step because the avatar may have pre-existing skinning information, and that pre-existing skinning information is sufficient. Other aspects of automatic skinning are discussed in... Figure 25 The method shown is presented in 2500. Box 2404 can be followed by box 2406.

[0198] In box 2406, skinning information can be transferred to an accessory to be fitted onto the avatar's body. The accessory can be from at least one of the following: body hair, clothing components, and combinations thereof. For example, this transfer could be part of a wrap deformer cage. Such a transfer operates when the accessory initially has a render mesh and / or wrap deformer cage but lacks skinning and / or rigging information. By transferring the skinning information, the skinning information provides the accessory with information that facilitates its use with the avatar during rendering. Box 2406 can be followed by box 2408.

[0199] In box 2408, an auto-skinned accessory mesh is generated. In this step, the accessory can supplement a previously existing rendered mesh and / or wrap deformer cage with previously migrated skinning information. This operation generates an auto-skinned accessory mesh. Such a mesh can be run as an alternative mesh to facilitate the effective use of the accessory in conjunction with an avatar. Generating an auto-skinned accessory mesh may include performing a geometric correspondence search that uses the accessory wrap deformer cage and the avatar wrap deformer cage to find the correspondence between the vertices of the original accessory mesh and the nearest corresponding position on the avatar mesh surface to which the accessory is to be attached. The above generation may also include performing centroid sampling of the avatar skinning information on at least one vertex of the original accessory mesh. Box 2408 may be followed by box 2410.

[0200] In box 2410, accessories can be fitted to the avatar body. Such fitting can be at least partially based on an autoskinned accessory mesh. The autoskinned accessory mesh facilitates fitting by helping to establish correspondences between the avatar and the accessory, which guide the fitting of the accessory to the avatar. Fitting accessories may include attaching the accessory to the avatar body using an autoskinned accessory mesh by providing skinning information to the wrapper deformer frame. Box 2410 may be followed by box 2412.

[0201] In box 2412, the avatar body can be animated. Animation can be performed after adaptation is complete. During such animation, the adapted accessories are animated in correspondence with the animated avatar body. This animation can be at least partially based on an auto-skinned accessory mesh. This can improve animation quality because using an auto-skinned accessory mesh makes the accessories fit better from the start. It can also improve the animation because the auto-skinned accessory mesh provides information to help the accessories move successfully along with the avatar body as it moves. Animating accessories can include rendering the accessories based on the avatar body's motion using an auto-skinned accessory mesh.

[0202] The motivation behind the described automatic skinning migration relates to solving at least two problems. One problem is that manually skinning accessories to make them naturally animated is both difficult and time-consuming. In some implementations, this problem is solved by eliminating the pre-skinning requirement for user / content creator communities that can create head and body-related accessories. Dynamic accessories may only need to include the geometry of the accessory itself and the wrapper deformer cage, without any further information. Creators no longer need to create skins when building head and body-related accessories, making the creation process for such accessories much easier.

[0203] Another problem is that any pre-skinned accessory doesn't work well on avatars that don't use the exact same skin and skeletal binding designed specifically for that accessory. Traditional accessory skinning techniques assume knowledge of the avatar's basic structure and the associated skeletal binding of the avatar to which the accessory is to be attached. Because of this assumption, any accessory pre-skinned for a specific avatar type (e.g., the Blocky avatar head) will not function properly if applied to other avatars with different skinning and / or skeletal binding structures.

[0204] Two characters can have two different underlying binding structures. These can differ in the number of joints, joint names, and joint hierarchy. If an accessory is constructed using one character's binding structure, these differences may prevent it from working with the binding structure of another character.

[0205] For dynamic heads, the implementation does not assume that a single skeletal rigging structure is used on every dynamic head. Therefore, the rigging of different dynamic heads may vary greatly in terms of the number of bones, bone effects, and even bone names. Because the implementation described herein eliminates the pre-skinning requirement, any accessory can be adapted to any dynamic avatar, regardless of the avatar's skinning or skeletal rigging structure, while ensuring that the accessory is correctly animated along with the avatar (body and / or head).

[0206] The automatic skinning migration option, driven by the various implementations described herein, is a useful approach that allows any accessory to fit and move correctly on any skinned avatar, regardless of the avatar's underlying skin or skeletal rigging structure. Furthermore, by eliminating the pre-skinning requirement for accessories, the described implementations make accessory creation easier for members of the creator community. Dynamic accessories only need to include their specific geometry (rendered geometry) and an associated wrapper deformer cage. The ultimate result of this change is the ability to successfully build more dynamic head and body-related accessories and make them available in the virtual environment's catalog.

[0207] Some implementations add these techniques as extensions to existing wrap deformer frameworks. This approach offers several advantages without adversely affecting the performance or functionality of the existing framework. Automatic skinning migration technology, implemented as an extension to existing wrap deformer frameworks (as a user-optional method), automatically migrates skinning information (such as skinning weights, bone effects, and associated skeletal bindings) from the avatar's head or body to which the accessory is attached to the accessory mesh using various search and projection techniques. Thus, the accessory itself does not require pre-skinning.

[0208] Provided the accessory has a suitable wrap deformer cage associated with it, the accessory can be correctly fitted and accurately deformed for a variety of dynamic avatars and heads. Existing wrap deformers perform the task of fitting the accessory to the head, while the automatic skinning transfer technique described in this paper allows the accessory to deform correctly and effectively with the avatar's body and facial expression animation.

[0209] Some implementations provide automatic skinning migration as an additional processing stage beyond the existing stages of the current wrap deformer framework. In some implementations, the additional migration stage is activated only if certain conditions are met. These conditions may include, for example: whether there is actually skinning on the source avatar head or body mesh; whether skinning migration is enabled on the WrapTarget instance via parameter settings such as AutoSkin and / or AutoSkinZone; whether there is no existing valid skinned mesh previously created through this process and located in a cache designated for managing skinned meshes, etc. In some implementations, the additional processing stage may occur only after the wrap deformer adaptation stage is completed and applied. If the conditions are met, the additional migration stage is activated.

[0210] There are several additional technical aspects to ensure that the automatic skinning migration functionality works effectively in the game engine and real-time experience. The first aspect is the performance of the skinning migration technology (speed, computational resource utilization, etc.). As with any in-game process, it's helpful to perform the skinning migration process as quickly as possible. Automatic skinning migration is activated during avatar loading, so any delays will be noticeable to the in-game user at that time. Automatic skinning migration is unlikely to occur after the initial character loading. The implementation needs to ensure that the migration process is as fast as possible, but within acceptable limits. For example, depending on the specific use case and / or implementation, there may be a maximum timeframe that the migration can take.

[0211] Furthermore, the automatic skinning migration can be separated as a background asynchronous task. While this task is in progress, the avatar remains active and dynamic within the experience. The only visible artifact the user might see during this time is that attached accessories may not deform correctly with the avatar (i.e., they may be static). Once the migration is complete, the attached accessories then begin to deform correctly. However, as mentioned earlier, preferably, this migration process should be performed as quickly as possible and ideally unnoticed by the user within the experience.

[0212] In some implementations, the automatic skinning migration phase takes a fairly short time (e.g., approximately 30 milliseconds for the example beard accessory). For clothing-type accessories or other simpler accessories, the phase may take even less time. Therefore, this performance is acceptable for most gaming environments.

[0213] Another aspect of some implementations is the quality of new / updated skinning on dynamic accessories. Any automated technique, including skinning migration, can be complex and may look incorrect (e.g., unnatural) or fail to achieve the quality expected of the in-game experience. Given the many different types of accessories (short hair, long hair, beards, skirts, pants, shirts, eyebrows, hats, etc.), designing a migration technique that works for any accessory type and situation can be quite difficult.

[0214] Automatic skinning migration can demonstrate acceptable quality results for the currently tested accessory types (hair, eyebrows, beard, shirt, trousers, etc.). In some cases, the quality of automatically skinned clothing adapters appears to be better than that of pre-skinned ones built into these accessories. In some implementations, quality can be managed and ensured by performing a series of quality tests using multiple accessories and avatars.

[0215] Various other implementations may include a variety of additional improvements, extensions, and optimizations. Some implementations include more aggressive geometric search optimizations, potentially involving pre-computed and cached information, as well as improved techniques. Other implementations include various cloud and CDN caching options.

[0216] Other implementations include more options, choices, and parameters for skinning migration techniques, which are used to fine-tune the adaptation performance for specific accessory types and conditions. Other implementations include the possibility of intelligently caching skinning meshes on a CDN so that future skinnings of the same accessory and head / body combination do not need to be recalculated. Some implementations also include improvements to skinning migration techniques so that unskinned R15 block type avatars can still use this automatic skinning migration method.

[0217] Another aspect of the implementation is asynchronous thread saturation. Automatic skinning migration adds new tasks to the background asynchronous thread pool. Therefore, the available threads may become more saturated, potentially slowing down the actual wrapper deformer phase. This can be particularly noticeable in implementations with limited available computational resources. To help mitigate potential thread saturation, some implementations modify the asynchronous thread pool with high-priority and low-priority queues.

[0218] The existing wrapper deformer stage and other existing asynchronous tasks still use the default / high-priority asynchronous task queue, while the new skinning migration stage is placed on the new low-priority asynchronous task queue. This ensures that other asynchronous tasks complete before the low-priority skinning migration stage is activated.

[0219] Another aspect of the implementation is avoiding memory bloat. Because the implementation internally caches the mesh to handle skinned meshes, it may add more memory to the executable. The implementation tracks this memory increase but also relies on the memory eviction limits of the MeshContentProvider to help manage internal memory. Some implementations apply novel and / or explicit eviction methods to the MeshContentProvider to force the eviction of original FileMeshData structures covered by newly skinned meshes. This approach can help mitigate any memory increase caused by internally caching these skinned meshes.

[0220] Another aspect of the implementation is to provide reverse skinning. In some cases, it may be advantageous to apply reverse skinning from accessories to the body. This method can be used for clothing components that will be applied to an unskinned body after skinning. Components can be constructed as modularly and universally as possible so that this technology can be implemented based on the techniques and disclosures provided for other implementations.

[0221] Figure 25 -Execute rigid automatic skinning Figure 25 This is a flowchart illustrating a computer implementation method 2500 for rigidly auto-skinning an avatar according to some embodiments. Method 2500 may begin at block 2502. Method 2500 may be used for rigid avatars (e.g., lacking all or part of the pre-skinning).

[0222] In box 2502, the body part is labeled as skinned or rigid. This determination establishes how a given body part is treated when performing rigid automatic skinning. Box 2502 may be followed by box 2504.

[0223] In box 2504, a correspondence search is performed to identify the nearest body part. Box 2504 can be followed by box 2506.

[0224] In box 2506, determine whether the given nearest body part is skinned or rigid. If the nearest body part is skinned, box 2508 follows box 2506. If the nearest body part is rigid, box 2510 follows box 2506.

[0225] In box 2508, skinning is sampled for this body part. This skinning can be sampled because previous operations in the method determined that the body part has skinning. The method can be repeated for other body parts after box 2508 to ensure successful skinning for the rigid avatar.

[0226] In box 2510, predetermined weights can be assigned to body parts. For example, if a body part is rigid, the predetermined weight of the vertices associated with that body part could be 1.0 or 100%. Predetermined weights (if 1.0 or 100%) can cause abrupt changes in skin weights between body joints. Therefore, performing smoothing can be helpful. Thus, box 2510 can be followed by box 2512.

[0227] In box 2512, smoothing is applied. For example, smoothing may involve iterating over each rigid vertex and taking the average weight of such a vertex with its neighboring vertices. Performing such smoothing iterations multiple times may also be appropriate. Some implementations may repeat the smoothing process three times, but other numbers of repetitions are also possible. After box 2512, the method may be repeated for other body parts to ensure successful skinning of the rigid avatar.

[0228] Figure 26 -Example computing device Figure 26 This is a block diagram of an example computing device 2600 according to some implementation methods.

[0229] Figure 26 This is a block diagram of an example computing device 2600 that can be used to implement one or more features described herein. In one example, device 2600 can be used to implement a computer device (e.g., Figure 1 (102 and / or 110), and performs the appropriate method described herein. The computing device 2600 can be any suitable computer system, server, or other electronic or hardware device. For example, the computing device 2600 can be a mainframe computer, desktop computer, workstation, portable computer, or electronic device (portable device, mobile device, cellular phone, smartphone, tablet computer, television, set-top box, personal digital assistant (PDA), media player, gaming device, wearable device, etc.). In some embodiments, the device 2600 includes a processor 2602, a memory 2604, an input / output (I / O) interface 2606, and an audio / video input / output device 2614.

[0230] Processor 2602 may be one or more processors and / or processing circuits to execute program code and control the basic operations of device 2600. "Processor" includes any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. A processor may include a system with a general-purpose central processing unit (CPU), multiple processing units, dedicated circuitry for implementing functions, or other systems. Processing is not limited to a specific geographical location or time. For example, a processor may perform its functions in a "real-time," "offline," or "batch processing mode." Parts of the processing may be executed by different (or identical) processing systems at different times and locations. A computer may be any processor that communicates with memory.

[0231] Memory 2604 is typically disposed in device 2600 for access by processor 2602 and can be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. Memory 2604 is suitable for storing instructions for execution by the processor and is decoupled from and / or integrated with processor 2602. Memory 2604 may store software operated by processor 2602 on server device 2600, including operating system 2608, virtual experience application 2610, autoskinning and migration and rigid autoskinning application 2612, and other applications (not shown). In some embodiments, application 2610 and / or application 2612 may include enabling processor 2602 to perform (or control) the functions described herein (e.g., regarding...). Figure 24 and Figure 25 Instructions (described in part or all of the methods).

[0232] For example, application 2610 may include automatic skinning migration and rigid automatic skinning application 2612, as described herein, which can perform automatic skinning migration and rigid automatic skinning within an online virtual experience server (e.g., 102). Software elements in memory 2604 may optionally be stored on any other suitable storage location or computer-readable medium. Furthermore, memory 2604 (and / or other connected storage devices) may store instructions and data used in the features described herein. Memory 2604 and any other type of memory (disk, optical disk, magnetic tape, or other tangible media) may be considered as "memory" or "storage device".

[0233] I / O interface 2606 can provide functionality that enables server device 2600 to interface with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data storage 120), and input / output devices can communicate through interface 2606. In some embodiments, the I / O interface can connect to interface devices including input devices (keyboard, pointing device, touch screen, microphone, camera, scanner, etc.) and / or output devices (display device, speaker device, printer, motor, etc.).

[0234] Audio / video input / output device 2614 may be an input and display device that includes a user input device (e.g., a mouse, etc.) that can be used to receive user input, a display device (e.g., a screen, a monitor, etc.) that can be used to provide graphical and / or visual output, and / or a combination of input and display devices.

[0235] For ease of explanation, Figure 26 A box is shown for each of the software blocks of processor 2602, memory 2604, I / O interface 2606, and operating system 2608 and virtual experience application 2610. These boxes may represent one or more processors or processing circuits, operating systems, memory, I / O interfaces, applications, and / or software engines. In other embodiments, device 2600 may not have all the components shown and / or may have other elements, including elements that replace those shown herein or other types of elements besides those shown herein. Although online virtual experience server 102 is described as performing the operations described in some embodiments herein, online virtual experience server 102, or any suitable component or combination of components of a similar system, or any suitable one or more processors associated with such a system, may perform the described operations.

[0236] User equipment may also implement and / or be used with the functions described herein. Example user equipment may be a computer device including components similar to device 2600, such as processor 2602, memory 2604, and I / O interface 2606. Operating systems, software, and applications suitable for the client device may be provided in memory and used by the processor. The I / O interface for the client device may be connected to network communication devices and input and output devices, such as a microphone for capturing sound, a camera for capturing images or video, a mouse for capturing user input, a gesture device for recognizing user gestures, a touchscreen for detecting user input, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices. For example, a display device within audio / video input / output device 2614 may be connected to (or included in) device 2600 to display pre-processed and post-processed images as described herein, wherein such a display device may include any suitable display device, such as an LCD, LED, or plasma display screen, CRT, television, monitor, touchscreen, 3D display screen, projector, or other visual display device. Some implementations may provide an audio output device, such as a synthesized voice output or text reading.

[0237] One or more methods described herein (e.g., method 2400 and / or method 2500) can be implemented by computer program instructions or code executable on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry) and can be stored on a computer program product including a non-transitory computer-readable medium (e.g., a storage medium), such as magnetic, optical, electromagnetic, or semiconductor storage media, including semiconductor or solid-state memories, magnetic tape, removable computer floppy disks, random access memory (RAM), read-only memory (ROM), flash memory, rigid disks, optical disks, solid-state storage drives, etc. Program instructions can also be contained in and provided as electronic signals, for example, in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.) or a combination of hardware and software. Example hardware can be a programmable processor (e.g., a field-programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), etc. One or more methods may be executed as part of or a component of an application running on the system, or as an application or software that runs with other applications and the operating system.

[0238] One or more methods described herein can run in standalone programs that can run on any type of computing device, programs that run on a web browser, or mobile applications (“apps”) that run on mobile computing devices (e.g., mobile phones, smartphones, tablets, wearable devices (watches, armbands, jewelry, headwear, goggles, glasses, etc.), laptops, etc.). In one example, a client / server architecture can be used, whereby the mobile computing device (as a client device) sends user input data to a server device and receives final output data from the server for output (e.g., for display). In another example, all computations can be performed within a mobile application (and / or other applications) on the mobile computing device. In yet another example, computations can be split between the mobile computing device and one or more server devices.

[0239] While specific embodiments have been described herein, these embodiments are for illustrative purposes only and not for limitation. The concepts illustrated in the examples can be applied to other examples and embodiments.

[0240] The functional blocks, operations, features, methods, devices, and systems described in this disclosure can be integrated or divided into different combinations of systems, devices, and functional blocks known to those skilled in the art. Routines for a particular implementation can be implemented using any suitable programming language and programming techniques. Different programming techniques can be employed, such as procedural or object-oriented. The routine can be executed on a single processing device or multiple processors. Although steps, operations, or calculations are presented in a specific order, this order can be varied in different specific implementations. In some implementations, multiple steps or operations shown as being performed sequentially in this specification can be performed simultaneously.

Claims

1. A computer-based method for adapting accessories to an avatar's body in a three-dimensional (3D) environment, the computer-based method comprising: Obtain the skin information of the avatar's body; The skin information is transferred to the accessories to be adapted to the avatar's body; An automatic skinning accessory mesh is generated, based at least in part on the skinning information and the original accessory mesh. The accessories are adapted to the avatar body, at least in part, based on the automatic skinning accessory mesh; as well as After the adaptation, the avatar body is animated, wherein the adapted accessories are animated in correspondence with the animated avatar body, based at least in part on the auto-skinned accessory mesh.

2. The computer implementation method according to claim 1, wherein, The skinning information includes at least one from a group consisting of: weights, skeletal effects, binding information, joints, other features of the avatar body, and combinations thereof.

3. The computer implementation method according to claim 1, wherein, The accessories to be adapted to the avatar's body include at least one from a group consisting of: body hair, clothing components, and combinations thereof.

4. The computer implementation method according to claim 1, wherein, The skin information of the avatar body is incomplete, and the computer implementation method further includes: Perform automatic skinning technology to generate derived skinning information for the avatar's body; and The automatic skinning accessory mesh is generated based on the derived skinning information of the avatar body.

5. The computer implementation method according to claim 4, wherein, Performing the automatic skinning technology to generate the derived skinning information for the avatar body includes: Each body part of the avatar is marked as either a skinned body part or a rigid body part; Perform a correspondence search to identify one or more body parts within a threshold distance of the accessory; For each of the one or more identified body parts... In response to the identified body part being the skinned body part, skinning sampling is performed on the corresponding vertex of the avatar body at the spatial location of the nearest identified body part; and In response to the identified body part being the rigid body part, original weights are assigned to the corresponding vertices of the avatar body at the location of the identified body part; and Based on the associated weights of the rigid vertices of the avatar body, an automatic skinning smoothing operation is applied to the rigid vertices of the avatar body.

6. The computer implementation method according to claim 1, wherein, Generating the automatic skinning accessory mesh includes supplementing the original accessory mesh and the accessory wrapping deformer (WD) cage associated with the accessory using information from the avatar mesh, the avatar wrapping deformer (WD) cage, and the skinning information of the avatar body.

7. The computer implementation method according to claim 6, wherein, Generating the automatic skinning accessory mesh includes: performing a geometric correspondence search, which uses the accessory wrapper deformer cage and the avatar wrapper deformer cage to find the correspondence between the vertices of the original accessory mesh and the nearest corresponding position on the surface of the avatar mesh to which the accessory is to be attached.

8. The computer implementation method according to claim 7, wherein, Generating the auto-skinned accessory mesh further includes performing centroid sampling of the avatar skinning information on at least one vertex of the original accessory mesh to generate the auto-skinned accessory mesh.

9. The computer implementation method according to claim 6, wherein, Adapting the accessory also includes attaching the accessory to the avatar body by providing skinning information to the wrapper deformer frame using the auto-skinning accessory mesh.

10. The computer implementation method according to claim 6, wherein, Animating the avatar body to which the accessory is adapted includes: using the auto-skinned accessory mesh, rendering the animation of the accessory based on the motion of the avatar body.

11. A non-transitory computer-readable medium storing instructions thereon, the instructions being responsive to execution by a processing device to cause the processing device to perform an operation, the operation comprising: Obtain skin information of the avatar body in a three-dimensional (3D) environment; The skin information is transferred to the accessories to be adapted to the avatar's body; An automatic skinning accessory mesh is generated, based at least in part on the skinning information and the original accessory mesh. The accessories are adapted to the avatar body, at least in part, based on the automatic skinning accessory mesh; as well as After the adaptation, the avatar body is animated, wherein the adapted accessories are animated at least in part based on the auto-skinned accessory mesh corresponding to the animated avatar body.

12. The non-transitory computer-readable medium according to claim 11, wherein, The skin information of the avatar body is incomplete, and the operation further includes: Perform automatic skinning technology to generate derived skinning information for the avatar's body; and The automatic skinning accessory mesh is generated based on the derived skinning information of the avatar body.

13. The non-transitory computer-readable medium according to claim 11, wherein, Generating the automatic skinning accessory mesh includes supplementing the original accessory mesh and the accessory wrapping deformer (WD) cage associated with the accessory using information from the avatar mesh, the avatar wrapping deformer (WD) cage, and the skinning information of the avatar body.

14. The non-transitory computer-readable medium according to claim 13, wherein, Generating the automatic skinning accessory mesh includes: performing a geometric correspondence search, which uses the accessory wrapper deformer cage and the avatar wrapper deformer cage to find the correspondence between the vertices of the original accessory mesh and the nearest corresponding position on the surface of the avatar mesh to which the accessory is to be attached.

15. The non-transitory computer-readable medium according to claim 13, wherein, Adapting the accessory also includes attaching the accessory to the avatar body by providing skinning information to the wrapper deformer frame using the auto-skinning accessory mesh.

16. A system comprising: Computing devices, including: A memory, on which instructions are stored; and A processing device coupled to the memory, the processing device being configured to access the memory and execute the instructions, wherein the instructions cause the processing device to perform an operation, the operation including: Obtain skin information of the avatar body in a three-dimensional (3D) environment; The skin information is transferred to the accessories to be adapted to the avatar's body; An automatic skinning accessory mesh is generated, based at least in part on the skinning information and the original accessory mesh. At least in part based on the automatic skinning accessory mesh, the accessory is adapted to the avatar's body; and After the adaptation, the avatar body is animated, wherein the adapted accessories are animated at least in part based on the auto-skinned accessory mesh corresponding to the animated avatar body.

17. The system of claim 16, further comprising a mesh content cache coupled to the computing device, the mesh content cache being used to store the auto-skinning accessory mesh for at least one of the adaptation or the animation, and the mesh content cache being coupled to a file repository, wherein, The operation also includes: It was determined that the auto-skinned accessory mesh did not exist in the mesh content cache; In response to determining that the auto-skinned accessory mesh does not exist in the mesh content cache, the auto-skinned accessory mesh is retrieved from the file repository; The retrieved auto-skinning accessory mesh is stored in the mesh content cache; and After the storage, the auto-skinned accessory mesh is accessed from the mesh content cache.

18. The system according to claim 16, wherein, The skin information of the avatar body is incomplete, and the operation further includes: Perform automatic skinning technology to generate derived skinning information for the avatar's body; and The automatic skinning accessory mesh is generated based on the derived skinning information of the avatar body.

19. The system according to claim 16, wherein, Generating the automatic skinning accessory mesh includes supplementing the original accessory mesh and the accessory wrapping deformer (WD) cage associated with the accessory using information from the avatar mesh, the avatar wrapping deformer (WD) cage, and the skinning information of the avatar body.

20. The system according to claim 16, wherein, Adapting the accessory also includes attaching the accessory to the avatar body by providing skinning information to the wrapper deformer frame using the auto-skinning accessory mesh.

Images