Adapting simulated character interaction to different forms and interaction scenes

Abstract

Some embodiments relate to methods, systems, and computer readable media for adapting simulated character interactions to different forms and interaction scenarios. The system accesses a graph representing control policies of motion of a simulated character in a virtual environment. The graph is encoded and processed by a graph neural network to generate a potential embedding of the graph. And determining a fixed-length potential vector according to the potential embedding. The vector is input to a feed-forward neural network to generate a control signal for the action of the character. The action of the role is continuously perfected by enhancing a learning loop by iteratively adjusting the graph based on evaluating the action of the simulated role via a reward function, such that the control strategy adapts to different role morphologies and / or interaction scenarios.

Description

TECHNICAL FIELD

[0001] Embodiments relate generally to the field of character simulation in virtual environments. More specifically, embodiments relate to methods and systems for accessing and processing motion control policies to adapt the motion and interaction of simulated characters to different morphologies and interaction scenarios. BACKGROUND

[0002] Simulated environments and virtual character interaction are a focus for electronic game and virtual experience developers. A continuing focus in the field is to enhance the realism and adaptability of character animation. Existing approaches often rely on pre-defined motion capture data or hand-crafted animations, which limits the versatility and adaptability of characters in virtual environments. These approaches face challenges in translating specific motions of a character to a variety of different character morphologies and interaction scenarios.

[0003] A significant limitation of current systems is their difficulty in generalizing motion control policies to different character types and interactions. The lack of a robust framework to automatically adapt control policies to different character morphologies and interaction contexts presents a significant barrier. Existing approaches often require manual adjustments and lack the ability to dynamically respond to changing virtual scenarios. Furthermore, these approaches can struggle to achieve nuanced and contextually appropriate character interactions, particularly in complex virtual environments where characters must perform a variety of actions.

[0004] Accordingly, there is a need in the field for methods and systems to facilitate character animation that can seamlessly adapt to different character morphologies and dynamically respond to the complex details of various virtual interactions. Addressing these challenges would improve the field of character simulation in virtual environments and ensure a more immersive and flexible experience for users of such virtual environments.

[0005] The background description provided herein is intended to generally present the context of the present disclosure. Work of the presently named inventors, to the extent the work is described in this background section, as well as aspects of the description that can not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure. SUMMARY

[0006] Embodiments relate to adapting simulated character interaction to different morphologies and interaction scenarios. In some embodiments, a system accesses a graph representing a control policy for motion of a simulated character in a virtual environment. The graph is encoded and processed by a graph neural network to generate a latent embedding of the graph. A fixed-length latent vector is determined from the latent embedding. The vector is sent as input to a feedforward neural network, which generates a control signal for an action of the character. Through a reinforcement learning loop, the action of the character is continually refined to adapt the control policy to different character morphologies and / or interaction scenarios by iteratively adjusting the graph based on evaluating the action of the simulated character via a reward function.

[0007] For example, in some implementations, a computer-implemented method includes accessing a graph representing a motion control policy for a simulated agent in a virtual environment, the graph including a plurality of nodes, the nodes including a body of the simulated agent and features of the virtual environment. The method includes encoding information of the graph, the information including one or more of: a position, a velocity, or a direction of each node, and the information including relational information of pairs of nodes connected by edges. The encoded information is processed by a graph neural network to generate one or more latent embeddings of the nodes of the graph. The method includes determining a fixed-length latent vector from the one or more latent embeddings, sending the fixed-length latent vector as input to a feedforward neural network to output a control signal for the simulated agent. The method includes applying the control signal to the body of the simulated agent in the virtual environment to provide an action of the simulated agent, and evaluating the action of the simulated agent by a reward function to assess a motion quality of the action and an interaction quality of the action. The method includes adapting the control policy by reinforcement learning by iteratively adjusting the graph based on the evaluation via the reward function, wherein the adaptation generalizes to one or both of: different agent morphologies in the virtual environment and different interactions.

[0008] Various implementations and examples of the method are described. For example, in some implementations, determining the fixed-length latent vector includes discarding a portion of the one or more latent embeddings to remove information related to a subset of the nodes. In some implementations, the graph neural network includes one or more graph attention network (GAT) layers. In some implementations, the feedforward neural network includes a multi-layer perceptron (MLP) including a series of fully connected layers. In some implementations, the virtual environment includes at least one additional agent as part of the motion control policy, the simulated agent interacting with the at least one additional agent. In some implementations, the nodes of the graph represent a plurality of internal points on the simulated agent and a plurality of external points external to the simulated agent. In some implementations, the edges of the graph represent nearest neighbors of each node determined by a K-nearest neighbor (KNN) algorithm. In some implementations, the encoding further includes calculating an angular velocity of each node. In some implementations, the graph is dynamically updated based on interactions of the agent in the simulated environment. In some implementations, the graph is dynamically updated based on changing positions of target agents or objects in the simulated environment. In some implementations, discarding the portion of the latent embedding includes selectively compressing information specific to one or more of the nodes in the graph.

[0009] In some implementations, a system includes one or more processors coupled to memory having software instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations include accessing a graph representing a motion control policy for a simulated agent in a virtual environment, the graph including a plurality of nodes including a body of the simulated agent and features of the virtual environment. The operations include encoding information of the graph, the information including one or more of: a position, a velocity, or a direction of each node, the information including relational information of pairs of nodes connected by edges, processing the encoded information through a graph neural network to generate one or more latent embeddings of nodes of the graph. The operations include discarding a portion of the one or more latent embeddings to remove information related to a subset of the nodes to obtain a fixed-length latent vector, and sending the fixed-length latent vector as input to a feedforward neural network to output a control signal for the simulated agent. The operations apply the control signal to the body of the simulated agent in the virtual environment to provide an action of the simulated agent, and evaluate the action of the simulated agent through a reward function to assess a motion quality of the action and an interaction quality of the action. The operations include adapting the control policy through reinforcement learning by iteratively adjusting the graph based on the evaluation via the reward function, wherein the adaptation generalizes to one or both of: different role morphologies in the virtual environment and different interactions.

[0010] Various implementations and examples of the system are described. For example, in some implementations, the reward function includes one or more of: a joint angle match, a joint velocity match, an end effector position match, or a root state match. In some implementations, the reward function includes a product of the joint match term and the interaction match term. In some implementations, the control signal determines one or both of: an angle or a torque of a joint in the simulated agent. In some implementations, the instructions further cause the system to perform operations including: randomly sampling to introduce additional nodes at a training phase. In some implementations, the graph neural network includes one or more graph attention network (GAT) layers, and the feedforward neural network includes a multilayer perceptron (MLP) including a series of fully connected layers. In some implementations, the operations of adapting the control policy include transfer learning for additional simulated agents, wherein the simulated agents perform a role involving a synchronized action. In some implementations, the output of the feedforward neural network is used to generate a database of motion patterns for one or both of: a plurality of different role morphologies, and a plurality of different interaction scenarios.

[0011] In some embodiments, a non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform operations. The operations include accessing a graph representing a motion control policy for a simulated agent in a virtual environment, the graph including a plurality of nodes including a body of the simulated agent and features of the virtual environment. The operations include encoding information of the graph, the information including one or more of: a position, a velocity, or a direction of each node, the information including relational information of pairs of nodes connected by edges. The operations include processing the encoded information through a graph neural network to generate one or more latent embeddings of nodes of the graph, modifying a portion of the latent embeddings to obtain a fixed-length latent vector, and sending the fixed-length latent vector as input to a feed-forward neural network to output a control signal for the simulated agent. The operations include applying the control signal to the body of the simulated agent in the virtual environment to provide an action of the simulated agent, evaluating the action of the simulated agent through a reward function to assess a motion quality of the action and an interaction quality of the action, and adapting the control policy through reinforcement learning by iteratively adjusting the graph based on the evaluation via the reward function, wherein the adaptation generalizes to one or both of: different agent morphologies in the virtual environment and different interactions.

[0012] Some embodiments can include a system including a processor and a memory coupled to the processor. The memory can have stored thereon instructions that, when executed by the processor, cause the processor to perform operations including one or more of the features of the above-described methods and / or computer-readable media. Some embodiments can include a computer-implemented method including one or more of the operations performed by the processor of the system and / or through the above-described computer-readable media. Some embodiments include a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor, cause the processor to perform operations that are the same as or similar to the features of the above-described methods and / or systems. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a diagram of an example system architecture for adapting simulated agent interactions to different morphologies and interaction scenarios, in accordance with some embodiments.

[0014] Figure 2 is a flowchart illustrating a method of adapting simulated agent interactions to different morphologies and interaction scenarios, in accordance with some embodiments.

[0015] Figure 3 is a diagram illustrating a method of generating and applying control signals for simulated agent motion from a control policy, in accordance with some embodiments.

[0016] Figure 4A is a diagram illustrating an example of a simulated agent in a virtual environment, in accordance with some embodiments.

[0017] Figure 4B FIG. 1 is a diagram illustrating an example of reinforcement learning of multi-agent interaction, according to some embodiments.

[0018] Figure 5 FIG. 2 is a block diagram illustrating an example computing device, according to some embodiments. DETAILED DESCRIPTION

[0019] In the following detailed description, reference is made to the accompanying drawings that form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments can be used, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. Aspects of the disclosure, as described throughout this document, can be arranged, substituted, combined, separated, and designed in numerous different configurations, all of which are contemplated herein.

[0020] Reference throughout this specification to “some embodiments,” “an embodiment,” “exemplary embodiment,” or similar language means that a described embodiment can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Further, appearances of such phrases in various places in the specification do not necessarily all refer to the same embodiment. Additionally, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the purview of one of ordinary skill in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not

[0021] One or more embodiments described herein relate to adapting simulated agent interaction to different morphologies and / or interaction scenarios. In some embodiments, a general-purpose neural network pipeline is described that is capable of processing any spatial representation of a scenario. Spatial information is processed by layers of a graph neural network (GNN), e.g., a graph attention network, and is projected into a latent space. The latent embedding can be modified to obtain a latent vector; for example, an amount of redundant information in the latent space can be discarded, resulting in a fixed-size latent variable that can capture the properties of the desired motion. This fixed-size latent variable can be processed by a feed-forward neural network, e.g., an MLP network. This network architecture allows the neural network to learn a single generalized policy that can be applied to various agent shapes and morphologies, and / or that can be used to represent different physical interaction scenarios between an agent and the surrounding environment in a virtual environment.

[0022] In some examples, a multi-agent reinforcement learning process is provided in which a control policy is trained to produce control signals for a simulated character. At each control step, the policy can receive input describing the state of the simulated character and observations of the reference future poses. The policy outputs joint angles for computing torques for each joint of the character. A reward function is used to measure the quality of motion and the quality of interaction of the character.

[0023] In one or more implementations, the above-described techniques enable a single policy to control a character to interact with its surroundings given a plurality of spatial locations that the character can attend to. In some implementations, the character and local environment state can be represented using a graph structure in which graph nodes are located on body parts of the character and other objects (e.g., other characters or other objects) in the character’s surroundings. In some implementations, edges of the graph are constructed by computing the K nearest neighbors of each node and connecting the neighboring nodes.

[0024] In one or more implementations, the interaction graph can be viewed as a kind of observation, which can include, for example, the positions, velocities, orientations, and angular velocities of all graph nodes and the differences in information in the node pairs of all edges. The number of nodes and edges outside of the controlled character is not fixed but variable. In some implementations, a GNN-based neural network architecture is used that can directly process graph structures and handle a variable number of external nodes (outside of the character) by maintaining a fixed-length latent specification after encoding and processing the graph. The latent variable is fed into a series of fully connected layers to output actions.

[0025] In some implementations, the above-described techniques can train a policy for a single character, e.g., that does not depend on a particular number of environmental objects interacting with the character. A single-character policy can be used to train a generalized policy for various different interactions of the character with other objects and / or to provide different character morphologies. For example, during training of a single character, a plurality of nodes can be randomly sampled around different body parts of the character in addition to existing nodes. Then, the interaction graph reward and the motion imitation reward are optimized such that the single-character policy is able to attend to different morphologies and / or regional interactions. The pre-trained model can then be used as a base policy for further multi-character interaction training.

[0026] In some implementations, to maintain the quality of motion and the quality of interaction, the reward function can be a product of two parts: a joint matching term and an interaction matching term. The joint matching reward term measures the matching of joint angles, joint velocities, end-effector positions, and root state between the simulated character and the reference character. The interaction matching term measures the difference in edges in the graph, where edges connect graph nodes outside of the character.

[0027] Advantages can include providing a reinforcement learning formulation that enables a training control policy to perceive a changing external environment state around a character or other object, enabling the character to perform various degrees of realistic physics motion and interaction. The training control policy can generalize to adapt to different environments and / or different character morphologies. The above features facilitate implementing a general simulation and training pipeline for physics simulation character control, e.g., where a user can set up and run a training job for a general simulation motion imitation controller with only a small amount of input (e.g., changing parameters). The above techniques can provide realistic physics simulation and interaction for a variety of different types of characters in a virtual experience, thereby enhancing the immersive experience for a user.

[0028] The above features provide a number of technical advantages over prior techniques for providing control policies for characters in virtual environments. For example, the above features provide a technical advantage that enables a reduction in the use of computing resources (e.g., computer memory, processor usage and time, network traffic bandwidth, display processing and memory usage, etc.) in various of the above embodiments. For example, prior control policies are fixed to a particular environment and cannot generalize to unseen environments due to a lack of perception of the environment around the controlled character; thus, different control policies need to be specifically trained and used for different environments. In contrast, the above control policies can adapt to a variable number of nodes (e.g., external nodes) and / or reduce the number of nodes (e.g., after discarding nodes), and thus can be used as a generalized control policy. This enables a single control policy to be trained and used in a variety of environments with different numbers of objects, thereby saving computing resources that would otherwise be used to train and store multiple control policies.

[0029] Another example technical advantage is that the above features allow virtual environments with digital characters, e.g., avatars, to animate the avatars using control policies such that these control policies can generalize to any type of character and any type of interaction, i.e., any interaction with any element, object, or character in the scene. For example, this includes generalizing control policies to characters with any skeletal morphology or representation, or any character that a user of the virtual environment can create. For example, this includes generalizing control policies to interactions with any geometry in the virtual environment, not just avatars or objects. This enables a single control policy to be trained and used for a variety of characters, thereby saving computing resources that would otherwise be used to train and store multiple control policies for multiple types of characters.

[0030] Figure 1 is a diagram of an example system architecture that can be used to adapt simulated character interactions to different morphologies and interaction scenarios, in accordance with some embodiments. Figure 1The same reference numbers can be used in different drawings to identify the same or similar elements. Characters following a reference number, for example, "110," indicate that the element has the same or similar structure as the element with that particular reference number. Reference numbers without a following character, for example, "110," indicate that the element has any or all of the possibilities of the element(s) in the drawing bearing that reference number (e.g., "110" in the text refers to reference numbers "110a," "110b," and / or "110n" in the drawings).

[0031] System architecture 100 (also referred to herein as "the system") includes an online virtual experience server 102, a data store 120, client devices 110a, 110b, and 110n (generally referred to herein as "client devices 110"), and developer devices 130a and 130n (generally referred to herein as "developer devices 130"). Virtual experience server 102, data store 120, client devices 110, and developer devices 130 are coupled by a network 122. In some embodiments, client devices 110 and developer devices 130 can refer to the same device or the same type of device.

[0032] Online virtual experience server 102 can include a virtual experience engine 104, one or more virtual experiences 106, a graphics engine 108, and the like. In some embodiments, graphics engine 108 can be a system, application, or module that allows online virtual experience server 102 to provide graphics and animation functionality. In some embodiments, graphics engine 108 can perform one or more operations described below in connection with the flowcharts shown. Figure 2 Client devices 110 can include a virtual experience application 112 and an input / output (I / O) interface 114 (e.g., input / output devices). The input / output devices can include one or more of a microphone, a speaker, headphones, a display device, a mouse, a keyboard, a game controller, a touchscreen, a virtual reality console, and the like.

[0033] Developer devices 130 can include a virtual experience application 132 and an input / output (I / O) interface 134 (e.g., input / output devices). The input / output devices can include one or more of a microphone, a speaker, headphones, a display device, a mouse, a keyboard, a game controller, a touchscreen, a virtual reality console, and the like.

[0034] System architecture 100 is provided for illustration. In different embodiments, system architecture 100 can include the same, fewer, more, or different elements configured in the same or different manners as shown. Figure 1

[0035] ​In some implementations, the network 122 can include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN), a wired network (e.g., an Ethernet network, a fiber optic network, or the like), a wireless network (e.g., an 802.11 network, a WiMax network, or the like), a router, a hub, a switch, a server computer, or a combination thereof. In some implementations, the network 122 can include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN), a wired network (e.g., an Ethernet network, a fiber optic network, or the like), a wireless network (e.g., an 802.11 network, a WiMax network, or the like), a router, a hub, a switch, a server computer, or a combination thereof.

[0036] In some implementations, the data store 120 can be a non-transitory computer- readable memory (e.g., a random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data. The data store 120 can also include multiple storage components (e.g., multiple drives or multiple databases) that can also span multiple computing devices (e.g., multiple server computers). In some implementations, the data store 120 can include a cloud-based storage.

[0037] In some implementations, the online virtual experience server 102 can include a server (e.g., a cloud computing system, a rack server, a server computer, a cluster of physical servers, or the like) having one or more computing devices. In some implementations, the online virtual experience server 102 can be a standalone system, can include multiple servers, or can be part of another system or server.

[0038] In some implementations, the online virtual experience server 102 can include one or more computing devices (e.g., a rack server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, or the like), a data store (e.g., a hard disk, a memory, a database), a network, software components, and / or hardware components that can be used to perform operations on the online virtual experience server 102 and provide users with access to the online virtual experience server 102. The online virtual experience server 102 can also include a website (e.g., a web page) or an application backend software that can be used to provide users with access to content provided by the online virtual experience server 102. For example, a user can access the online virtual experience server 102 using the virtual experience application 112 on the client device 110.

[0039] In some embodiments, virtual experience session data is generated by the online virtual experience server 102, the virtual experience application 112, and / or the virtual experience application 132 and stored in the data store 120. With the permission of the virtual experience participants, the virtual experience session data can include associated metadata, such as a virtual experience identifier; device data associated with the participants; demographic information of the participants; a virtual experience session identifier; chat logs; session start time, session end time, and session duration for each participant; relative locations of participant avatars in the virtual experience environment; purchases made by the participants in the virtual experience; accessories used by the participants; and the like.

[0040] In some embodiments, the online virtual experience server 102 can be a social network that provides connectivity between users or a user-generated content system that allows users (e.g., end users or consumers) to communicate with other users on the online virtual experience server 102, where the communication can include voice chat (e.g., synchronous and / or asynchronous voice communication), video chat (e.g., synchronous and / or asynchronous video communication), or text chat (e.g., one-to-one and / or many-to-many synchronous and / or asynchronous text-based communication). Some or all of the user communication records can be stored in the data store 120 or the virtual experience 106. The data store 120 can be used to store chat logs (text, audio, images, etc.) exchanged between the participants.

[0041] In some embodiments of the present disclosure, a “user” can be represented as a single individual. However, other embodiments of the present disclosure encompass a “user” as an entity controlled by a group of users or automated sources (e.g., a creative user). For example, a group of individual users that collectively act as a community or group in a user-generated content system can be considered a “user.”

[0042] In some implementations, the online virtual experience server 102 can be a virtual game server. For example, the game server can provide a single-player or multi-player game to a community of users that can access the“system” herein including the online game server 102, the data store 120, and the client device 110, and / or can interact with the virtual experience using the client device 110 over the network 122. In some implementations, the virtual experience (including a virtual realm or world, a virtual game, other computer-simulated environment) can be a two-dimensional (2D) virtual experience, a three-dimensional (3D) virtual experience (e.g., a user-generated 3D virtual experience), a virtual reality (VR) experience, or an augmented reality (AR) experience. In some implementations, a user can participate in an interaction (e.g., a game) with other users. In some implementations, other users of the virtual experience can experience the virtual experience in real-time.

[0043] In some implementations, virtual experience participation can refer to an interaction of one or more participants in a virtual experience (e.g., 106) using a client device (e.g., 110), or rendered on a display or other output device (e.g., 114) of the client device 110. For example, virtual experience participation can include an interaction with one or more participants in a virtual experience, or rendered on a display of a client device.

[0044] In some implementations, the virtual experience 106 can include an electronic file that can be executed or loaded using software, firmware, or hardware for presenting virtual experience content (e.g., a digital media item) to a physical entity. In some implementations, the virtual experience application 112 can be executed and render the virtual experience 106 in conjunction with the virtual experience engine 104. In some implementations, a virtual experience 106 can have a set of common rules or common goals, and the environment of the virtual experience 106 shares the set of common rules or common goals. In some implementations, different virtual experiences can have different rules or goals from each other.

[0045] In some implementations, a virtual experience can have one or more environments (also referred to herein as“virtual experience environments” or“virtual environments”) in which multiple environments can be linked. An example of an environment can be a three-dimensional (3D) environment. One or more environments of a virtual experience 106 can be collectively referred to herein as a“world” or“virtual experience world” or“game world” or“virtual world” or“virtual space” or“universe.” An example of a world can be a 3D world of a virtual experience 106. For example, a user can build a virtual environment that is linked to another virtual environment created by another user. Characters of a virtual experience can cross virtual boundaries into adjacent virtual environments.

[0046] It should be noted that a 3D environment or 3D world uses graphics that use a three- dimensional representation of geometry data representing virtual experience content (or at least present virtual experience content as 3D content whether or not a 3D representation of geometry data is used). A 2D environment or 2D world uses graphics that use a two-dimensional representation of geometry data representing virtual experience content.

[0047] In some implementations, an online virtual experience server 102 can host one or more virtual experiences 106 and can allow users to interact with virtual experiences 106 using virtual experience applications 112 of client devices 110. Users of an online virtual experience server 102 can play virtual experiences 106, create virtual experiences 106, interact with virtual experiences 106, or build virtual experiences 106, communicate with other users, and / or create and build objects of virtual experiences 106 (e.g., also referred to herein as“items” or“virtual experience objects” or“virtual experience items”).

[0048] For example, in generating a user-generated virtual item, a user can create a character, a character’s accessory, one or more virtual environments for an interactive virtual experience, or a structure for use in a virtual experience 106, among others. In some embodiments, a user can buy, sell, or trade virtual experience objects with other users of the online virtual experience server 102, such as in-platform currency (e.g., virtual currency). In some embodiments, the online virtual experience server 102 can transmit virtual experience content to a virtual experience application (e.g., 112). In some embodiments, virtual experience content (also referred to herein as “content”) can refer to any data or software instructions associated with the online virtual experience server 102 or a virtual experience application (e.g., virtual experience objects, virtual experiences, user information, videos, images, commands, media items, etc.). In some embodiments, a virtual experience object (also referred to herein as “item” or “object” or “virtual object” or “virtual experience item”) can refer to an object used, created, shared, or otherwise depicted in a virtual experience application 106 of the online virtual experience server 102 or a virtual experience application 112 of a client device 110. For example, a virtual experience object can include a component, a model, a character, an accessory, a tool, a weapon, a costume, a building, a vehicle, a currency, a plant population, an animal population, a component of the above (e.g., a window of a building), among others.

[0049] It should be noted that the online virtual experience server 102 hosting virtual experiences 106 is provided for illustration. In some embodiments, the online virtual experience server 102 can host one or more media items, which can include communication messages from one user to one or more other users. With user permission and explicit user consent, the online virtual experience server 102 can analyze chat log data to improve the virtual experience platform. Media items can include, but are not limited to, digital videos, digital movies, digital photographs, digital music, audio content, melodies, website content, social media updates, e-books, e-magazines, digital newspapers, digital audiobooks, electronic journals, weblogs, real simple syndication (RSS) feeds, electronic comic books, software applications, among others. In some embodiments, a media item can be an electronic file that can be executed or loaded using software, firmware, or hardware for presenting the digital media item to an entity.

[0050] In some implementations, a virtual experience 106 can be associated with a particular user or a particular group of users (e.g., a private virtual experience), or can be generally available to users with access to the online virtual experience server 102 (e.g., a public virtual experience). In some implementations, where the online virtual experience server 102 associates one or more virtual experiences 106 with a particular user or group of users, the online virtual experience server 102 can use user account information (e.g., user account identifiers such as a username and password) to associate a particular user with a virtual experience 106.

[0051] In some implementations, the online virtual experience server 102 or the client device 110 can include a virtual experience engine 104 or a virtual experience application 112. In some implementations, the virtual experience engine 104 can be used for development or execution of a virtual experience 106. For example, the virtual experience engine 104 can include a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, a scripting functionality, an animation engine, an artificial intelligence engine, a networking functionality, a streaming functionality, a storage management functionality, a threading functionality, a scene graph functionality, or animation video support, among other functionalities. The components of the virtual experience engine 104 can generate commands (e.g., rendering commands, collision commands, physics commands, etc.) that help compute and render a virtual experience. In some implementations, the virtual experience application 112 of the client device 110 can work independently and / or in coordination with the virtual experience engine 104 of the online virtual experience server 102, respectively.

[0052] In some implementations, both the online virtual experience server 102 and the client devices 110 can execute virtual experience engines (104 and 112, respectively). The online virtual experience server 102 using the virtual experience engine 104 can perform some or all of the virtual experience engine functionality (e.g., generating physics commands, rendering commands, etc.) or offload some or all of the virtual experience engine functionality to the virtual experience engine 104 of the client devices 110. In some implementations, the ratio between the virtual experience engine functionality executed on the online virtual experience server 102 and the virtual experience engine functionality executed on the client devices 110 can vary for each virtual experience 106. For example, the virtual experience engine 104 of the online virtual experience server 102 can be used to generate physics commands in the event of a collision between at least two virtual experience objects, while additional virtual experience engine functionality (e.g., generating rendering commands) can be offloaded to the client devices 110. In some implementations, the ratio of virtual experience engine functionality executed on the online virtual experience server 102 and the client devices 110 can change (e.g., dynamically) based on virtual experience conditions. For example, if the number of users participating in a particular virtual experience 106 exceeds a threshold number, the online virtual experience server 102 can execute one or more virtual experience engine functions previously executed by the client devices 110.

[0053] For example, a user can play a virtual experience 106 on a client device 110 and can send control instructions (e.g., user inputs such as right, left, up, down, user selection, or character position, and velocity information, etc.) to the online virtual experience server 102. After receiving the control instructions from the client device 110, the online virtual experience server 102 can send experience instructions (e.g., position and velocity information for a character participating in a team experience, or commands such as rendering commands, collision commands, etc.) to the client device 110 based on the control instructions. For example, the online virtual experience server 102 can perform one or more logical operations on the control instructions (e.g., using the virtual experience engine 104) to generate experience instructions for the client device 110. In other cases, the online virtual experience server 102 can pass one or more control instructions from one client device 110 to other client devices participating in the virtual experience 106 (e.g., from client device 110a to client device 110b). The client devices 110 can use the experience instructions and render the virtual experience for presentation on the display of the client device 110.

[0054] In some implementations, the control instructions can refer to instructions that instruct the user character’s actions in the virtual experience. For example, the control instructions can include user inputs for controlling actions in the experience, such as right, left, up, down, user selections, gyroscope position and orientation data, force sensor data, etc. The control instructions can include character position and velocity information. In some implementations, the control instructions are sent directly to the online virtual experience server 102. In other implementations, the control instructions can be sent from the client device 110 to another client device (e.g., from client device 110b to client device 110n), where the other client device generates experience instructions using the local virtual experience engine 104. The control instructions can include instructions for playing voice communication messages or other sounds from another user on an audio device (e.g., a speaker, headphones, etc.), such as voice communications or other sounds generated using audio spatialization techniques as described herein.

[0055] In some implementations, the experience instructions can refer to instructions that enable the client device 110 to render a virtual experience (e.g., a multi-participant virtual experience). The experience instructions can include one or more of user inputs (e.g., control instructions), character position and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.).

[0056] In some implementations, a character (or generally a virtual experience object) is composed of components that are automatically connected together to assist the user in editing, one or more of which can be selected by the user.

[0057] In some implementations, a character is implemented as a 3D model and includes a surface representation (also referred to as a skin or mesh) for rendering the character and a set of interconnected hierarchical bones (also referred to as a skeleton or rig). The rig can be used to animate the character and simulate the character’s movements and actions. The 3D model can be represented as a data structure, and one or more parameters of the data structure can be modified to change various attributes of the character, such as size (height, width, circumference, etc.); body type; movement style; number / type of body parts; proportions (e.g., shoulder-to-hip ratio); head size, etc.

[0058] One or more characters (also referred to herein as “avatars” or “models”) can be associated with a user, where the user can control the character to facilitate the user’s interaction with the virtual experience 106.

[0059] In some embodiments, a character can include components such as body parts (e.g., hair, arms, legs, etc.) and accessories (e.g., T-shirt, glasses, decorative images, tools, etc.). In some embodiments, customizable body parts of a character include head type, body part type (arms, legs, torso, and hands), face shape, hair style, and skin type, among others. In some embodiments, customizable accessories include clothing (e.g., shirt, pants, hat, shoes, glasses, etc.), weapons, or other tools.

[0060] In some embodiments, for some asset types (e.g., shirts, pants, etc.), the online virtual experience platform can provide users with access to simplified 3D virtual object models represented by meshes with a low number of polygons (e.g., between about 20 and about 30 polygons).

[0061] In some embodiments, a user can also control the scale (e.g., height, width, or depth) of a character or the scale of components of a character. In some embodiments, a user can control the scale (e.g., blockiness, anatomical structure, etc.) of a character. Note that in some embodiments, a character can not include a character virtual experience object (e.g., body part, etc.), but a user can control the character (without a character virtual experience object) to facilitate user interaction with a virtual experience (e.g., a puzzle game in which no character game object is rendered, but a user can still control the character to control actions in the game).

[0062] In some embodiments, a component (e.g., a body part) can be a primitive geometric shape such as a block, a cylinder, a sphere, or other primitive shape such as a wedge, a torus, a tube, a channel, etc. In some embodiments, a creator module can publish a user’s character for viewing or use by other users of the online virtual experience server 102. In some embodiments, a user can create, modify, or customize a character, other virtual experience objects, virtual experiences 106, or virtual experience environments using an I / O interface (e.g., a developer interface) and with or without scripts (or with or without application programming interfaces (APIs)). Note that a character is described as having a humanoid form for illustration. Also note that a character can be in any form such as a vehicle, an animal, a living or non-living object, or other creative form.

[0063] In some embodiments, the online virtual experience server 102 can store user-created avatars in the data store 120. In some embodiments, the online virtual experience server 102 maintains a catalog of avatars and a catalog of virtual experiences that can be presented to users. In some embodiments, the catalog of virtual experiences includes images of virtual experiences stored on the online virtual experience server 102. Additionally, a user can select an avatar (e.g., an avatar created by the user or another user) from the catalog of avatars to participate in a selected virtual experience. The catalog of avatars includes images of avatars stored on the online virtual experience server 102. In some embodiments, one or more avatars in the catalog of avatars can have been created or customized by a user. In some embodiments, a selected avatar can have avatar settings that define one or more components of the avatar.

[0064] In some embodiments, a user's avatar can include a configuration of components, where the configuration and appearance of the components (and more generally, the appearance of the avatar) can be defined by avatar settings. In some embodiments, a user's avatar settings can be at least partially selected by the user. In other embodiments, a user can select an avatar with default avatar settings or avatar settings selected by another user. For example, a user can select a default avatar from a catalog of avatars with predefined avatar settings, and the user can further customize the default avatar by changing some of the avatar settings (e.g., adding a shirt with a customized logo). Avatar settings can be associated with a particular avatar by the online virtual experience server 102.

[0065] In some embodiments, the client devices 110 can each include a computing device (e.g., a personal computer (PC), a mobile device (e.g., a laptop computer, a mobile phone, a smart phone, a tablet computer, or a netbook computer), a network-connected television, a gaming console, etc. In some embodiments, the client devices 110 can also be referred to as "user devices." In some embodiments, one or more client devices 110 can be connected to the online virtual experience server 102 at any given moment. It should be noted that the number of client devices 110 is provided for illustration. In some embodiments, any number of client devices 110 can be used.

[0066] In some implementations, each client device 110 can include an instance of a virtual experience application 112, respectively. In one implementation, the virtual experience application 112 can allow a user to use and interact with the online virtual experience server 102, such as controlling a virtual character in a virtual experience hosted by the online virtual experience server 102, or viewing or uploading content, such as virtual experiences 106, images, video items, webpages, documents, etc. In one example, the virtual experience application can be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, render, or navigate content (e.g., virtual characters in a virtual environment, etc.) served by a web server. In another example, the virtual experience application can be a native application (e.g., a mobile application, an app, a virtual experience program, or a game program) that is installed and executed locally at the client device 110 and allows a user to interact with the online virtual experience server 102. The virtual experience application can render, display, or present content (e.g., webpages, media viewers) to a user. In implementations, the virtual experience application can also include an embedded media player (e.g., Flash® or HTML5 player) embedded in a webpage.

[0067] According to aspects of the disclosure, the virtual experience application can be an online virtual experience server application for a user to build, create, edit, upload content to, and interact with the online virtual experience server 102 (e.g., participate in virtual experiences 106 hosted by the online virtual experience server 102). As such, the virtual experience application can be provided by the online virtual experience server 102 to the client device 110. In another example, the virtual experience application can be an application downloaded from a server.

[0068] ​In some implementations, each developer device 130 can include an instance of a virtual experience application 132, respectively. In one implementation, the virtual experience application 132 can allow a developer user to use and interact with the online virtual experience server 102, such as controlling a virtual character in a virtual experience hosted by the online virtual experience server 102, or viewing or uploading content, such as virtual experiences 106, images, video items, web pages, documents, etc. In one example, the virtual experience application can be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, render, or navigate content (e.g., virtual characters in a virtual environment, etc.) served by a web server. In another example, the virtual experience application can be a native application (e.g., a mobile application, an app, a virtual experience program, or a game program) that is installed and executed natively on the developer device 130 and allows a user to interact with the online virtual experience server 102. The virtual experience application can render, display, or present content (e.g., web pages, media viewers) to the user. In implementations, the virtual experience application can also include an embedded media player (e.g., a Flash® or HTML5 player) embedded in a web page.

[0069] According to aspects of the disclosure, the virtual experience application 132 can be an online virtual experience server application for users to build, create, edit content uploaded to the online virtual experience server 102 and interact with the online virtual experience server 102 (e.g., provide and / or participate in virtual experiences 106 hosted by the online virtual experience server 102). As such, the virtual experience application can be provided by the online virtual experience server 102 to the client device 130. In another example, the virtual experience application 132 can be an application downloaded from a server. The virtual experience application 132 can be used to interact with the online virtual experience server 102 and gain access to user credentials, user currency, etc. of one or more virtual experiences 106 developed, hosted, or provided by a virtual experience developer.

[0070] In some implementations, a user can log in to the online virtual experience server 102 through the virtual experience application. The user can access a user account by providing user account information (e.g., a username and a password), where the user account is associated with one or more characters that can be used to participate in one or more virtual experiences 106 of the online virtual experience server 102. In some implementations, with proper credentials, a virtual experience developer can gain access to virtual experience virtual objects, such as in-platform currency (e.g., virtual currency), avatars, special abilities, accessories owned or associated with other users.

[0071] ​Generally, functionality described in one implementation as being performed by the online virtual experience server 102 can also be performed by the client device 110 or a server in other implementations, if appropriate. Moreover, functionality attributed to a particular component can be performed by different components or multiple components operating together. The online virtual experience server 102 can also be accessed as a service provided to other systems or devices through a suitable application programming interface (API) and thus is not limited to use in a website.

[0072] Figure 2 Methods are shown for adapting simulated character interactions to different morphologies and interaction scenarios. In various embodiments, Figure 2 The blocks shown, and described below, can be performed by Figure 1 any of the elements shown.

[0073] At block 202, the system accesses a graph representing a motion control policy for a simulated character in a virtual environment (e.g., a simulated environment, a virtual space, etc.). In some implementations, the graph can be considered a base control policy. In some implementations, the graph is a spatial graph, and the graph includes a plurality of nodes, each node corresponding to a component of the simulated character’s anatomy or an element in the character’s environment. In some implementations, links can be established between particular nodes to indicate simulated character anatomy components, such as body parts (e.g., limbs or portions thereof, such as upper arms, lower arms, thighs, etc.). In some implementations, the nodes contain data related to the character or environment, such as, for example, the position, velocity, and orientation of the respective character component or environmental element.

[0074] Characters have a particular morphology, and different characters can have different morphologies. Morphologies can be characterized by the bones of the nodes or links. In some examples, characters with similar body parts but different limb lengths can have different morphologies. In other examples, characters with different numbers of arms, heads, or other body parts have different morphologies.

[0075] Edges in the graph define relationships between the nodes. For example, an edge can indicate a connection between particular nodes. These relationships can indicate articulation between anatomy components, such as limb joints and limbs, or interaction elements between the character and environmental objects. The edges can identify and distinguish interactions in the virtual environment. For example, a graph edge can connect two adjacent joints in a representation of a particular limb of the character. Graph edges can also connect any two nodes in the graph that can not be adjacent. Nodes in the graph can also be located “outside” of the character, in which case they do not represent a joint on the character’s body. Thus, graph edges can be used to represent any type of connection between nodes in the graph.

[0076] As used herein, an "interaction" refers to a dynamic interaction between a simulated character and an element in a virtual environment (e.g., another character or object external to the simulated character). This encompasses a range of activities, such as a character's movement relative to its own body parts or external characters or objects. For example, a movement of a hand of an avatar relative to a shoulder or a manipulation of an object in a virtual space can be considered different interactions.

[0077] The graph structure representing the control policy serves as an integrated framework encapsulating these interactions. Each node in the graph corresponds to a key feature of the simulated character's body and the virtual environment, forming a representation of the mirror character's morphology and any external objects of the character's interactions. The edges in the graph represent connections between these nodes, encapsulating relational information and spatial dependencies.

[0078] As used herein, an "interaction scenario" refers to control policy information about different contexts and elements that affect how a simulated character performs any given interaction. For example, this can include particular conditions, objects, or entities involved in the interaction. For example, an interaction scenario for a character picking up a cup from a table can change when the character encounters different objects, such as a ball, trophy, pen, or cell phone. For another example, the same character can pick up the same ball differently if it is in a rainstorm or there are various other external objects in the path the character must take. In this way, the interaction scenario represents a customized instance of engagement, reflecting the adaptability and flexibility of the control policy in navigating different virtual environments and scenarios and reacting appropriately to various nuances.

[0079] In some implementations, the system generates a graph data structure (or other representation) representing the graph by training a neural network on different motion examples of a simulated character. For example, each motion of the character can be represented as a different interaction graph. The system then generates a graph representing the neural network trained for a particular motion interaction. This graph is a motion control policy that, when executed on the character, is used to reproduce the motion interaction. The system then accesses the graph representing the motion control policy.

[0080] In some implementations, the system accesses the graph by retrieving a graph data structure from a storage medium. In some implementations, the graph data structure models the state of the character and related environmental entities through a network of nodes and edges. Nodes are assigned to particular points indicative of the character's body and the environment, while edges represent connections defining relationships therebetween.

[0081] In some implementations, the nodes can embody position data as well as other potential motion data related to parts of the character’s body (e.g., the character’s joints and / or limbs) and environmental objects that can interact with the character. In some examples, the respective nodes can represent a head, a neck, each hand, each wrist, each forearm, each upper arm, each thigh, each lower leg, each foot, etc. (see Figure 4A and 4B some examples). Edges connect these nodes to map out possible interactions in the graph framework. The graph is used as an input to a motion control policy that is capable of rendering interactions of the character with the surrounding environment. In some implementations, the system retrieves the graph through a computational method such as a query or read operation that extracts the graph from its storage location and places it in a working memory for processing.

[0082] In some implementations, the simulated character represents an anthropomorphic avatar with a skeletal structure. In some implementations, the skeletal structure in such an avatar can be configured in terms of the number and arrangement of bones. This configuration allows the system to adapt to various morphologies of anthropomorphic avatars in different implementations. Other forms of avatars can also be used, such as characters representing animals or objects.

[0083] In some implementations, the avatar can operate in a virtual environment in which the avatar can interact with other entities (e.g., other avatars, virtual objects, etc.). In some implementations, the motion control policy generates signals that direct the movement of the avatar’s joints and limbs. These signals are based on the avatar’s skeletal structure and affect the avatar’s positioning, rotation, and joint activity in the virtual environment.

[0084] In some implementations, the motion control policy is designed for versatility across different avatars. The motion control policy is developed through training on various skeletal structures, which enables the policy to manage the animation of a range of anthropomorphic avatars. This versatility enables the use of a single policy to animate different forms of anthropomorphic avatars in a virtual environment.

[0085] In some embodiments, the virtual environment includes at least one additional character as part of the motion control policy, and the simulated character is programmed to interact with the at least one additional character. In some embodiments, a defined algorithm controls the interaction between the simulated character and the additional character, where the policy of the simulated character is adapted according to the prescribed dynamics of the environment and the actions of the other characters. In some embodiments, the system can utilize the motion control policy to detect and respond to the presence and activity of other characters. The policy is used to govern the navigation functions and object manipulation of the simulated character, as well as interactions with other characters. In some embodiments, the simulated character is programmed to modify its responses according to predefined interaction protocols, which can include a range of cooperative and independent activities. For example, one predefined interaction protocol can relate to self-interaction, where the simulated character is programmed to interact with itself, e.g., moving its hands to scratch its nose or slap its body. Another example of an interaction protocol can be interaction with an object, e.g., catching a ball. Another example of an interaction protocol can be interaction with another avatar, e.g., the simulated character dancing with another simulated character.

[0086] In some embodiments, the motion control policy includes spatial and temporal information related to the position and motion of the character in the environment. The algorithm computes the position data and predicts potential interactions. In some embodiments, this allows the simulated character to adjust its behavior in real time.

[0087] In some embodiments, the graph nodes represent a plurality of internal points on the simulated character and a plurality of external points external to the simulated character. In some embodiments, the internal nodes can represent specific points on the simulated character, such as joints and / or center of mass.

[0088] In some embodiments, the graph integrates external nodes representing specific points in the surrounding virtual environment external to the character nodes and / or edges. In some embodiments, at least a subset of the external nodes are related to locations where the simulated character is expected to interact with other characters or objects. For example, one such location can be a space in the virtual environment close to the head of the character and where a ball is moving towards it, or a space in the virtual environment where both characters are trying to grab a single object (e.g., close to the hands of both characters). In some embodiments, external nodes can be randomly sampled, e.g., nodes around different body parts of the character. In some embodiments, the use of external nodes causes the control policy to focus on certain locations of the joints to external nodes, thereby causing the control policy to focus on different interactions.

[0089] In some embodiments, the selection and representation of nodes in the graph is based on the location of the nodes on the simulated character or in the virtual environment. The graph is structured to include character nodes and environment-related external nodes.

[0090] In some embodiments, the edges of the graph represent the nearest neighbors of each node, which can be determined according to a K-Nearest Neighbor (hereinafter “KNN”) algorithm or similar algorithm. The algorithm determines the edges by identifying a predefined number of nearest nodes for each node based on a spatial proximity metric. For example, in some embodiments, four nearest nodes can be used, or in other various embodiments, other numbers of nearest nodes can be used. In some embodiments, the KNN algorithm handles multidimensional data by considering the position and orientation of the nodes. It can calculate proximity using a predetermined distance metric, which can include standard metrics such as Euclidean distance or application-specific metrics. These metrics guide the KNN algorithm to form a graph that reflects the spatial organization and relationships relevant to the simulation task.

[0091] In some embodiments, the graph structure is dynamically updated by a KNN algorithm that recomputes the node connections in response to the motion and interactions of the characters in the virtual environment. This ultimately generates a graph that continuously reflects the current state of the characters relative to the virtual environment, including the positions and interaction states of the characters.

[0092] In some embodiments, the graph is dynamically updated based on the interactions of the characters in the virtual environment. For example, the graph can be updated in response to interactions that occur in the virtual environment. These updates adjust the properties and relationships of the nodes in the graph, which correspond to the body parts of the characters and elements in the surrounding environment.

[0093] In some embodiments, the update mechanism changes the graph by adding, removing, or modifying nodes and edges. Node properties such as position, velocity, or orientation are changed as needed by the actions of the characters or interactions with objects and other characters. In some embodiments, the system utilizes one or more algorithms to update the graph by detecting changes in the simulation environment. These algorithms track relevant data points such as position shifts or velocity changes to maintain an up-to-date representation of the interactions in the graph structure.

[0094] In some implementations, the graph is dynamically updated based on changing positions of target agents or objects in the virtual environment. For example, the system can track and reflect changes in positions and motions of agents or objects in the virtual environment using a graph update mechanism. The system adjusts graph elements (including nodes and edges) according to the motions to ensure that the graph remains up-to-date with respect to the state of the virtual environment. In some implementations, the dynamic update of the graph depends in part on changing positions of target agents or objects in the simulated environment. As the graph is updated, the motion control policy continues to respond to spatial contexts and position changes of the target agents or objects. In some implementations, the system takes into account real-time motions and positions of the target agents or objects, triggering adjustments to the graph structure to accommodate the changing interaction dynamics. In one example, if a simulated agent attempts to grab a ball object (i.e., a target object) moving in the air within the environment according to simulated physical properties of the ball in flight, the graph can be dynamically adjusted in real-time as the position or trajectory of the ball changes to reflect this changing spatial relationship.

[0095] In some implementations, the system allows simulated agents to modify their behavior according to changes in the virtual environment. Dynamic updates to the graph can support these modifications, enabling agents to adjust their motion strategies and align their actions with updated environmental conditions. In some implementations, the system dynamically updates the graph by triggering updates based on specific events. In one example, an avatar in a game environment can enable new motion control capabilities depending on whether a zero-gravity condition is enabled in a particular virtual space. In another example, an avatar can be subject to a new motion control policy based on different simulated environmental conditions (e.g., rain, wind, or ice). The system can operate periodically or continuously to assess and implement changes within the virtual environment and / or the agents. In some implementations, the system implements updates by recalibrating the nodes and edges of the graph as needed according to virtual environment motions and interactions.

[0096] At block 204, the system encodes information for the nodes and edges of the graph. Encoding information for the graph involves collecting and representing specific data attributes for each node. These nodes correspond to features of simulated agents and environmental elements. In some implementations, the system captures position data for each node in the graph to define spatial positions of the elements in the simulated space. In some implementations, the system encodes velocity data for each node, thereby capturing motions of all elements in the simulation over time.

[0097] In some implementations, directional details (including directional orientations for each node) are encoded to define directions of the elements in the simulated environment. Angular velocities are encoded for each node, specifying rates of change of direction, which are relevant to actions involving rotation.

[0098] In some implementations, the system encodes relationship data for pairs of connected nodes in the graph. This includes information on how the nodes interact with each other based on their connections, which are depicted by edges. These edges constitute the structure of the simulated character’s body and define relationships with environmental elements.

[0099] In some implementations, the encoding includes computing angular velocities for each node. In some implementations, the system encodes angular velocities for each node to quantify rotational dynamics of the simulated character’s motion and interactions with objects and other characters in the virtual environment.

[0100] At block 206, the system processes the encoded graph information through a neural network based on a graph neural network (hereinafter “GNN”) architecture. In some implementations, the neural network can be a graph attention network (hereinafter “GAT”) and / or include GAT layers to generate latent embeddings for graph nodes. In some implementations, this operation involves an attention mechanism applied to node features, including, for example, position, velocity, orientation, and relationship information for pairs of connected nodes. In this context, an attention mechanism refers to a computational technique that selectively focuses on particular aspects of node features, enabling the neural network to assign different levels of importance to different elements in the graph representation.

[0101] “Relationship information” refers to details of the connections or relationships between pairs of nodes in the graph. In some implementations, the system captures not only the intrinsic properties of each node, such as its position, velocity, or orientation, but also the inter-node relationships defined by the edges connecting the nodes. These relationships can encompass various spatial and contextual aspects that can be used to provide the system elements with information about how different parts of the simulated character’s body or the virtual environment interact. For example, for a humanoid avatar represented by the graph, relationship information can include, for example, the spatial proximity of one joint to another, the relative orientation between body parts, or dynamic interactions between nodes representing different elements of the character’s anatomy.

[0102] In some implementations, the attention mechanism of the neural network is calibrated to assign weights to node features during processing, thereby assigning importance to each feature based on the graph structure. These weights are dynamically determined based on the graph structure, allowing the neural network to assign different levels of importance to each feature. Different weights allow the neural network to selectively process input features when creating latent embeddings. For example, in some implementations, the weights of the neural network are automatically learned using well-established machine learning optimization algorithms, such as gradient descent. For reinforcement learning tasks, algorithms such as Proximal Policy Optimization (“PPO”) can be used.

[0103] After processing the encoded graph information by the GNN, a reduced dimensional representation of the input data in the form of a latent embedding is obtained. For example, a graph that can be represented as an N-dimensional vector can be processed by the GNN into an M-dimensional vector, i.e., a latent embedding, where M is less than N. In some implementations, the structure of this embedding is compatible with subsequent operations of the policy network, allowing further processing to output control signals that simulate the motion of the agent.

[0104] In some implementations, the system uses one or more other types of graph neural networks (“GNNs”) instead of or in addition to GAT to encode the graph structure. This graph structure corresponds to a motion policy for a simulated agent in a virtual environment. The GNN leverages node connections that are associated with elements of the agent’s body and environmental objects. Such connections enable the processing of spatial features such as positions, orientations, and velocities through the network.

[0105] In some implementations, the GNN can be used for various graph structures that vary in size and complexity. This configuration allows for representing agents with different morphologies and simulating a variety of different interaction scenarios. The system leverages the GNN to process and encode relational data that reflects dynamic interactions between the agent and elements in the environment. In some implementations, the GNN is designed to support generalization to a variety of interaction scenarios. The GNN layers manage the necessary computational load associated with the agent-environment interactions. In some implementations, this includes more complex learning tasks that are required to perform high-dimensional policy training operations.

[0106] At block 208, the system determines a fixed-length latent vector that represents a modified structure compared to the structure of one or more latent embeddings. For example, the system can modify the latent embedding to obtain the fixed-length latent vector. For example, in the example of Figure 2 In the example of, the system discards a portion of the latent embedding to obtain the fixed-length latent vector. The discarding operation can involve deleting data corresponding to certain nodes in the graph representation.

[0107] In some implementations, during the discarding operation, a predetermined algorithm or set of rules identifies and removes the embeddings of nodes that are deemed unnecessary for maintaining the functionality of the motion control policy. The criteria for discarding the node embeddings can be predefined and can be based on various characteristics, such as the relative influence or spatial relationship of the nodes in the graph structure. “Relative influence” here refers to the importance or impact of a node in the overall graph, with nodes having greater influence being more likely to be retained. “Spatial relationship” here refers to the positions and connections between nodes, where the discarding algorithm can prioritize nodes that contribute important information based on their spatial proximity or specific relationships in the graph.

[0108] The output of this block is a latent vector of fixed dimensionality that contains condensed information from the original graph. This vector is formed in a way that only the embeddings necessary to generate the core motion control signal are retained, while extraneous data that is not necessary for the core motion control functionality is discarded. By discarding a certain amount of non-essential or redundant information in the latent space, a fixed-size latent vector can be obtained that can capture the properties of the desired motion. This discarding can decouple the learning of the neural network from the structure of the graph, enabling the policy to generalize to different character morphologies and different interactions with the character’s own nodes and / or other object nodes in the virtual environment.

[0109] In some implementations, discarding a portion of the latent embedding involves selectively compressing information specific to one or more nodes of the graph. This can include identifying and removing nodes that represent information that is redundant or non-essential to the graph-based representation of the control policy. In some implementations, the system utilizes selective compression to reduce the dimensionality of the interaction graph by deleting certain nodes and their associated attributes (e.g., which can include position, orientation, or velocity data).

[0110] In some implementations, data representing external nodes (e.g., external to the character) is discarded in the embedding space, while internal character node data is retained in the latent embedding. This enables the internal nodes of the character to include information based on the (prior) presence of external nodes. Thus, the graph attention layer can “squeeze” additional information related to external nodes into their neighboring internal nodes, thereby enabling the policy to know which nodes it should focus on and where interactions can occur.

[0111] In some implementations, the system utilizes heuristic methods to determine which nodes to discard during the selective compression process. These heuristic methods are used to evaluate the relevance of each node to the desired outcome of the interaction and identify nodes that have minimal impact on the outcome as candidates for deletion. In some examples, nodes that are furthest from the interaction or that exceed a certain distance can be deemed less relevant and can be discarded. In some implementations, random nodes of the graph can be discarded, enabling the neural network to learn and adapt based on these random discards and new graph structures, as described below. In some implementations, the system randomly discards nodes so that the neural network trains on a variety of graph structures. This operation of randomly discarding nodes of the graph can be performed to make the learned policy robust to different graph structures.

[0112] At block 210, the system sends the fixed-length latent vector as input to a feedforward neural network (hereinafter “FNN”) to generate control signals for the simulated agent. The FNN structure includes an input layer, hidden layers, and an output layer composed of nodes. Each node is associated with a set of weights and biases that are applied to the input data as part of the transformation process. One example of an FNN that can be used is a multilayer perceptron (“MLP”). In various implementations, other neural network architectures can be utilized, such as convolutional neural networks applied to graphs (i.e., graph convolutional networks).

[0113] In some implementations, during operation, the latent vector is processed sequentially through the hidden layers. In each layer, the vector undergoes a weighted sum followed by an activation function, thereby introducing nonlinearity. Commonly used activation functions include sigmoid, hyperbolic tangent, or rectified linear unit (ReLU). The output of the hidden layers is fed forward to the output layer, which produces the control signals. These control signals correspond to specific actions or motions required by the joints or actuators of the simulated agent in the virtual environment.

[0114] The design of the FNN, in terms of the number and configuration of layers and nodes, the selection of activation functions, and the learning rate, is capable of mapping the high-dimensional data represented by the latent vector to actionable control signals suitable for a variety of simulated agent morphologies and interaction scenarios. In some implementations, the parameters of the network are determined during a training phase and are fixed prior to this operation. The output of this block includes the joint angles or torques necessary to guide the motion of the agent in accordance with the intended interaction objectives.

[0115] In some implementations, the FNN comprises a series of fully connected layers. These layers are used to map the latent embeddings to control signals for the simulated agent in the virtual environment. Each layer in the FNN transforms the input features in a hierarchical manner to produce an output vector representing the control signals.

[0116] In some implementations, the FNN receives information encoded in the form of fixed-length latent vectors from previous layers (possibly graph-based processing layers, such as GAT layers). The fully connected layers of the FNN process these latent vectors and adjust the weights and biases of the network to generate the output.

[0117] In some implementations, the adaptability of the FNN is provided through the ability to modify the number and density of neurons in the fully connected layers. This tunability can cater to different levels of complexity in the interaction scenarios as well as differences in agent morphologies. In some implementations, the control signals output by the FNN are applied to the simulated agent to achieve motion and interaction in the virtual environment. These signals typically correspond to specifications such as joint angles, velocities, and torques required for the animation of the agent.

[0118] In some embodiments, the control signals determine one or both of the joint angles and joint torques of the simulated character. These pieces of information can be used to manipulate the joint movements of the simulated character. Specifically, in some embodiments, the joint angles and torques are required for each joint movement. In some embodiments, this makes the motion in the virtual environment conform to the specified motion parameters. These signals direct the physical actions of the character, including object manipulation and character-character interactions, that are determined from the input from the virtual environment and the objectives encoded in the control policy. In some embodiments, the FNN architecture outputs a vector of control signals for each joint in the simulated character body. This architecture is capable of producing specific joint movement commands that are processed and applied to the character model.

[0119] At block 212, the system applies the control signals to the simulated character body. Applying the control signals to the simulated character body in the virtual environment includes converting the output of the FNN into physical commands that control the motion of the character. These control signals are within the constraints of the character physics simulation model, indicating joint angles and positions corresponding to body segment movements.

[0120] In some embodiments, each control signal corresponds to a motion parameter, such as a joint torque or a position setpoint. The virtual environment processes these parameters and applies them to the character model, which can be composed of different numbers of joint segments or joints. As the simulation progresses, these motions are executed according to the physical laws programmed in the simulation, including collision detection and response.

[0121] In some embodiments, the virtual environment collects data about the execution of these control signals, including spatial and temporal accuracy relative to expected results. In some embodiments, the virtual environment provides real-time feedback according to the motion dynamics of the character and interactions with other entities and objects in the simulated space.

[0122] At block 214, the system evaluates the simulated agent’s action through a reward function. The reward function quantitatively assesses the motion quality and interaction quality of the simulated agent’s performance of the action. The reward function operates by comparing the simulated action to one or more reference models. In this context, the reference models serve as benchmarks or standards for gauging the motion and interaction quality of the simulated agent’s action. They are determined based on predefined criteria or models that contain desired characteristics or behaviors. During the evaluation process, the reward function systematically compares the action performed by the simulated agent to the attributes specified by the reference models. This comparative analysis forms the basis for quantifying the motion and interaction quality of the simulated agent, providing measurable indicators for reinforcement learning adaptation. For example, the motion quality can be assessed by examining the degree of imitation of the simulated agent’s motion to a reference motion, e.g., by examining specific metrics such as joint angles, joint velocities, end-effector positions, and accuracy of the agent’s root (e.g., a reference point in the simulated agent’s skeletal structure, such as the pelvis or the base of the spine) state. By comparing these metrics, the agent’s motion can be made to conform to the target reference motion specified in the reference model.

[0123] In some embodiments, the interaction quality is measured by examining the simulated interaction of the agent with one or more objects against the reference interaction specified in the reference model between the objects. This aspect of the reward function examines the discrepancies between the edges connecting the agent’s nodes to one or more nodes of other objects (e.g., nodes belonging to entities in the environment or one or more nodes belonging to other agents) in the agent’s interaction graph.

[0124] In some embodiments, the reward function includes one or more of the following: joint angle matching, joint velocity matching, end-effector position matching, and root state matching. The joint angle matching term can be used to measure the joint angles of the simulated agent relative to the reference model. The joint velocity matching term can be used to measure the velocity of the simulated agent’s joints compared to a predetermined velocity profile. The end-effector position matching term can be used to assess the accuracy of the end-effector position, which is the point on the agent used for interaction with the environment. The root state matching term can be used to evaluate the position and orientation of the core part of the simulated agent’s body, which in some agent morphologies can be, for example, the pelvis or torso.

[0125] In some embodiments, the reward function is the product of the joint matching term and the interaction matching term. In some embodiments, the joint matching term can be a different type of matching term. In some embodiments, the joint matching term can be used as a “motion matching term” or imitation reward, meaning that it penalizes deviations from the reference motion. In some embodiments, the reward term can be explicitly defined, e.g., by training a motion discriminator, and implicitly learned.

[0126] In some implementations, the joint matching term quantifies the alignment of the simulated character's joints with the reference motion, which can include, for example, the character's joint angles and velocities, the end effector's position, and the root state. In some implementations, the interaction matching term in the reward function evaluates the character's interactions in the virtual environment. This term computes the difference between the information of pairs of nodes on the edges of an interaction graph, where the edges connect nodes outside the character to other elements in the environment. In some implementations, the interaction matching term rewards the minimization of the distance between nodes inside the character and nodes outside the environment, e.g., rewards the movement of the inside nodes towards the outside nodes. In some implementations, the reward function uses the joint matching term and the interaction matching term to evaluate the motion imitation and the environmental interactions. The control policy driven by this combined reward function adapts to produce outputs that take into account both precise motion replication and interaction dynamics.

[0127] The following are examples of overall reward functions according to one or more implementations. For the internal motion quality, the following reward function equations are related to the following, respectively: joint state matching (r joint ), end effector position matching (r ee ), and root joint state matching (r root ):

[0128]

[0129] r ee = exp(k3∑||x sim -x ref ||)

[0130] r root = exp(k4[||p sim -p ref ||+||q sim -q ref ||])

[0131] For the external interaction quality, the following reward function equations are related to the similarity of the external edges between the simulation and the reference:

[0132] r edge = exp(k5[∑w||∈ sim -∈ ref ||])

[0133] In these equations, q is the joint direction, is the angular velocity, x is the center of mass position, p is the position, ∈ is the edge position attribute (e.g., based on the difference in position between connected nodes), w is the weight of the term, k1 to k5 are the sensitivities of the terms, and “sim” refers to the simulated graph and “ref” refers to the reference graph. In some implementations, the interaction reward function can be determined by measuring the distance between the positions and velocities of the reference graph and the simulated graph using the weighted edges, and determining the interaction reward function based on the measured distance, e.g., to reward minimization of the distance between the internal nodes and the external nodes, and / or to reward movement of the internal nodes towards the external nodes.

[0134] At block 216, the system adapts the control policy through reinforcement learning by iteratively adjusting the graph based on the evaluation process via the reward function. This adaptation can generalize to one or both of different character morphologies and different interaction scenarios in the virtual environment. The adaptation phase involves refining the control policy through reinforcement learning, thereby providing adaptability and generalization of the motion control system. Through the learning process via the reward function, the control policy is iteratively adjusted, refined, and / or optimized based on the evaluation of the simulated character actions.

[0135] To adapt the control policy, the system employs reinforcement learning, i.e., a paradigm in which the system learns through trial and error, continually adjusting its behavior to maximize a defined reward. This iterative optimization aims to account for different character morphologies and various interaction scenarios in the virtual environment. The system actively learns from the interactions of the simulated characters and refines the control signals based on the specified reward function, ensuring that the adapted control policy is not limited to specific character types or interactions, but extends its applicability to a broader range.

[0136] In some implementations, the process of adapting the control policy to different interaction scenarios involves utilizing reinforcement learning to adapt the control policy to various situational variations within a broader class of interactions. For example, the simulated character can be responsible for an interaction scenario of picking up an object from a surface. This general type of interaction can be instantiated differently depending on the specific object involved. The control policy, driven by reinforcement learning, adapts to account for the subtle differences associated with different objects. In this way, the control policy is trained to dynamically adjust the same picking up action of the character to accommodate scenarios with different objects, such as a cup, a ball, a trophy, a pen, or a phone. Each object can introduce unique spatial considerations and characteristics that affect the execution of the interaction. Through the reinforcement learning-driven adaptation process, the control policy gains the ability to generalize its approach to a multitude of interaction scenarios within a specific interaction type.

[0137] In some implementations, the system utilizes reinforcement learning to adapt control policies to a variety of different character morphologies, enabling the control policies to generalize to different anatomies of the character. For example, consider a control policy designed for a humanoid avatar with a particular skeletal structure. Through reinforcement learning, the policy can be adapted to different character morphologies by learning representations of motion that are broadly applicable to motions outside of the particular skeletal configuration. The reinforcement learning mechanism enables the control policy to identify patterns, features, and characteristics of motion that are applicable to a range of morphological variations, and then apply them to a particular different morphology. For example, a control policy can be trained to adapt its representations of motion to humanoid avatars with different limb lengths, joint positions, or body proportions. The adaptation process ensures that a single control policy can be adapted to different avatars with different morphologies, while taking into account and applying information in the control policy that is relevant to the broader interactions to be implemented.

[0138] In some implementations, the system utilizes random sampling techniques to insert additional nodes during the training phase. In some implementations, the training phase utilizes variable node layouts related to different regions of the character to develop control policies that are generalizable to a range of virtual environment interactions and character morphologies. In some implementations, additional nodes are randomly placed on the character and / or in the virtual environment during training. This random placement alters the character-environment spatial relationships in the graph representation. The addition of such nodes enables the control policy to dynamically adapt to changes in the graph structure.

[0139] In some implementations, the introduction of randomly sampled nodes can diversify the dataset available to the reinforcement learning. This randomization affects the graph representation at each iteration of training. It influences the neural network processing of the graph structure, focusing on important parts of the motion control policy. The system can develop control policies that are applicable to different graph arrangements, without being limited to a particular training configuration.

[0140] In some implementations, the system utilizes transfer learning to adapt control policies to other simulated characters. The transfer learning approach enables the system to utilize an initial model trained on a particular character or task as a starting point for training other characters. In some implementations, the system preserves certain network layers and parameters in the original model, while adjusting other parameters to reflect the structure and interaction properties of the new character. In this way, the policy can be targeted to different characters.

[0141] In some implementations, during transfer learning, the latent embedding in the simulated character model is modified to cater to the requirements of a different character morphology or interaction type. This adjustment involves the system re-encoding the latent embedding based on the new character specification, such as the configuration of joint or body part proportions.

[0142] In some implementations, the system applies transfer learning to the control policy, adapting to different morphologies and interactions without needing to retrain completely from scratch. The system preserves the motion and interaction features encoded in the policy while making necessary adjustments to cater to the new simulated agents.

[0143] In some implementations, the system synchronizes the motions of simulated agents in a virtual environment. The agents perform a collaborative task according to a control policy. The policy regulates the motions based on temporal and spatial data relevant to each agent’s role in the task. The motions can include coordinated velocities, matching motions, or continuous activities. For example, for complex coordinated motions, such as a complex dance, the motions can be broken down into smaller steps that are executed in a specific order between the interacting agents.

[0144] In some implementations, the reward functions in the reinforcement learning framework are constructed to address synchronization. These functions measure the accuracy of the temporal and motion alignment between different agents. The policy refines its output by optimizing against these reward functions, which are designed to evaluate the alignment of the agent motions with predefined sequences of motions. In some implementations, a prediction mechanism in the control policy aligns the agent interactions with the upcoming task state. The policy adjusts according to temporal shifts in the task environment, enabling the agents to remain coordinated across different scenarios without considering potential intentions or interests.

[0145] In some implementations, the FNN output is used to generate a database of motion patterns for one or both of: a set of different agent morphologies, and a set of different interaction scenarios. In some implementations, creating the database with the FNN output involves capturing and storing motion patterns associated with different agent morphologies. The database represents a repository of predefined sequences of motions tailored for specific agent types, such as agent morphologies such as specific skeletal types. In some implementations, the database includes multiple interaction scenarios. The FNN output facilitates compiling motion patterns corresponding to a variety of different virtual interactions.

[0146] Figure 3 FIG. 1 is a diagram illustrating a method of generating and applying control signals for simulated agent motions according to a control policy, according to some implementations.

[0147] Figure 3The interaction graph 302 shown on the left is a visual representation of the control policy that simulates the motion of the avatar. This graph has been trained for a specific motion of the simulated avatar. The graph 302 encapsulates the dynamics of the motion of the simulated avatar in the virtual environment. The graph is composed of a set of nodes and edges. The graph models the physical characteristics of the simulated avatar as well as the elements present in the virtual environment. The nodes correspond to descriptive points, including internal aspects of the avatar and external entities in the environment, forming a comprehensive network that encapsulates the avatar’s interactions. The edges represent the connections between these nodes, representing the relationships and dependencies required to define the behavior of the simulated avatar.

[0148] The GAT layers 304 (denoted by arrows labeled “GAT layers”) represent the system that processes the interaction graph 302 with graph attention network layers (and / or other GNNs). The GAT layers 304 receive the interaction graph 302 and provide latent embeddings with fewer dimensions than the graph 302. The GAT layers 304 are responsible for processing the information encoded in the interaction graph. These layers implement a specialized neural network architecture designed for graph-based data, enabling the system to discern the importance of different nodes in the graph and determine their priorities. The GAT layers facilitate the generation of latent embeddings by weighting the contributions of neighboring nodes.

[0149] The graph structure 306 represents the latent embeddings generated by applying the GAT layers 304. The top portion 307 of the graph is labeled “latent embeddings,” representing the encoded information that encapsulates the essential characteristics of the simulated avatar’s body and the virtual environment. The bottom portion 309 is labeled “dropout,” representing the removal of a subset of information from the graph nodes through a dropout process. This selective preservation and discarding of information is performed so that the system generates fixed-length latent vectors.

[0150] After generating the fixed-length latent vectors through the latent embedding and dropout stages, the information is then fed to the FNN 308 for further processing. The FNN, which represents a series of fully connected layers, is responsible for converting the encoded latent information into actionable control signals for the simulated avatar. This conversion is based on the FNN’s ability to learn complex patterns and relationships in the encoded data. The action stage 310 (denoted by arrows labeled “actions”) represents the output of the FNN in the form of control signals that guide the motion and interactions of the simulated avatar in the virtual environment. The action stage 310 includes the system applying these control signals to the simulated avatar’s body in the virtual environment.

[0151] Figure 4Ais a diagram illustrating an example of a simulated character in a virtual environment according to some embodiments. The simulated character is depicted as a polygonal model that depicts a humanoid avatar 402 with a basic morphology that contains distinguishable features such as a head, arms, legs, and feet, as well as joints between these features. Internal nodes 404 in the character body are visually represented, corresponding to specific points on the avatar that describe the motion and interaction of the avatar, and can represent features (e.g., a head or hand) and / or joints between features (e.g., an elbow, shoulder, knee, ankle, etc.). Internal links 405 in the character body are also visually represented, can connect two nodes 404, and can correspond to features of the character, such as limbs (e.g., upper arm, lower arm, thigh, etc.).

[0152] Edges 406 in the diagram represent relational connections between nodes 404, revealing the internal structure of the character. These edges 406 can be determined through a K-Nearest Neighbor (KNN) algorithm, and highlight direct adjacency relationships between nodes in the character body. The KNN algorithm is selected to help capture the spatial proximity of nodes, i.e., factors that represent real and contextually relevant interactions in the character body. The techniques described herein can use connections between nearest neighbor nodes (as shown by edges 406 in this example) to preserve spatial relationships, resulting in coherent and reasonable character motion interactions.

[0153] External nodes 408 in the diagram are examples of one or more nodes external to the character body that can be added to the graph to support external interactions with elements in the surrounding environment described herein. The external nodes 408 in this example can be a ball or other object external to the character body in the environment, or a specific spatial region that has been determined to potentially provide interaction for the character, e.g., a space that other objects can move into. Other external nodes can be located at different regions around the character.

[0154] Figure 4B is a diagram illustrating an example of reinforcement learning for multi-character interaction according to some embodiments. Specifically, this schematic depicts the results of reinforcement learning for multi-character interaction in a virtual environment. This visual representation shows two humanoid avatars, each designed with a polygonal model. Both avatars are facing forward in the simulated environment. Note that the first avatar (the avatar closer to the right edge of the diagram) has its hand extended backward to touch the hand of the second avatar (the avatar closer to the left edge of the diagram), while the second avatar has its hand extended slightly forward to touch the hand of the first avatar. This represents a specific interaction scenario that is achieved by adapting the respective control policies of the avatars. In some examples, external nodes are placed between the hands of the avatars as potential interaction nodes.

[0155] As described above, the control strategies of both avatars were iteratively optimized based on the received reward signals. Through this process, the avatars learned to coordinate their actions synchronously, thereby achieving the desired hand-holding interaction. The adaptability of the control strategies is emphasized, indicating their ability to generalize to different character forms and interaction scenarios within the virtual environment.

[0156] Various example implementations of one or more features described herein may also include, or alternatively include, one or more of the following.

[0157] Machine learning models (e.g., machine learning controllers) manage interactions between roles and one or more other roles or objects through control policies implemented via neural networks. In some implementations, roles may be represented by graphs (e.g., graph data structures).

[0158] Machine learning models (such as machine learning controllers) can implement control strategies through neural networks to adapt to the interactions between management roles and one or more other roles or objects.

[0159] In some implementations, the more-dimensional graph representing the role is compressed into a simpler, less-dimensional representation in the embedding space, specific to the behavior created for the control policy. Since these morphologies can all use the same (low-dimensional) latent space, the control policy can control multiple different role morphologies to perform behaviors.

[0160] In some implementations, a method may include obtaining a controller that controls the motion of a reference graph representation of a character, trained based on a basic control policy to cause the character to follow a reference motion trajectory (e.g., a predetermined reference motion trajectory). The reference graph representation of the character is processed by a neural network into a latent space with a different dimension than the reference graph representation to recast the reference graph representation into a modified graph representation with one or more structures different from the reference graph representation, thereby adapting the basic policy to the modified policy. In some implementations, this enables a network learned on a first graph with N nodes to support runtime inference on a second graph with an arbitrary number of nodes; for example, the second graph may represent a character with a different morphology, different skeletal structure, and / or fewer or more nodes than the original graph / character. In some implementations, one or more different structures in the modified graph representation may include dropping and / or adding nodes.

[0161] In some embodiments, a method of providing a generalized control policy for interactions of an agent with other objects can include obtaining data indicative of a reference motion of a reference avatar, and / or obtaining (or constructing) and storing a reference graph of the reference motion that captures spatial relationships of the agent and external objects, where the reference graph is a set of spatial descriptors that encode relative information between vertices in the reference graph; initializing a simulation graph that simulates the motion; assigning weights to each edge of the reference graph and the simulation graph; measuring a distance between positions and velocities of the reference graph and the simulation graph using the weighted edges; determining an interaction reward function based on the measured distance, e.g., to reward minimization of distance between internal and external nodes, or to reward motion of internal nodes towards external nodes; and optimizing a combination of the interaction reward function and a motion reward function that rewards the simulation graph to mimic the motion of the reference graph.

[0162] Figure 5 is a block diagram of an example computing device 500 that can be used to implement one or more features described herein. In one example, the device 500 can be used to implement a computer device (e.g., the 102 and / or 110 of FIG. 1), and perform appropriate method embodiments described herein. The computing device 500 can be any suitable computer system, server, or other electronic or hardware device. For example, the computing device 500 can be a mainframe computer, a desktop computer, a workstation, a portable computer, or an electronic device (a portable device, a mobile device, a cellular phone, a smart phone, a tablet, a television, a television set-top box, a personal digital assistant (PDA), a media player, a gaming device, a wearable device, etc.). In some embodiments, the device 500 includes a processor 502, a memory 504, an input / output (I / O) interface 506, and an audio / visual input / output device 514. Figure 1

[0163] The processor 502 can be one or more processors and / or processing circuitry to execute program code and control the basic operations of the device 500. A “processor” includes any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. The processor can include a system with a central processing unit (CPU), multiple processing units, dedicated circuitry for providing functionality, or other systems. Processing need not be limited to a particular geographic location or have temporal limitations. For example, a processor can perform its functions in “real-time,” “offline,” in a “batch mode,” etc. Portions of processing can be performed at different times and at different locations, by different (or the same) processing systems. A computer can be any processor in communication with a memory.

[0164] ​Memory 504 is generally provided in device 500 for processor 502 to access, and can be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrical erasable read-only memory (EEPROM), flash memory, etc., suitable for storing instructions for execution by the processor and separate from and / or integral to the processor 502. The various engines, modules, instructions, machine learning models, software code, and other blocks used in the above-described functionality can be stored in memory 504 and / or other connected storage devices, such as database 512. Memory 504 can store software operated on by processor 502 on server device 500, including operating system 508, one or more applications 510, and database 512 that can store data used by components of device 500. In some implementations, applications 510 can include software applications, engines, machine learning models, etc., that include instructions that enable processor 502 to perform (or control) the functionality described herein (e.g., with respect to Figure 2 the described portions or all of the methods). For example, applications 510 can include a control policy module that implements one or more machine learning models used in the techniques described herein. Applications 510 can include a GNN, such as GAT layer 304 and FNN 308 (e.g., MLP) in Figure 3 Database 512 (and / or other connected storage) can store various data used in the above-described techniques, including graph representations, latent embeddings, latent vectors, training data, etc.

[0165] Elements of the software in memory 504 can optionally be stored on any other suitable storage location or computer-readable medium. Moreover, memory 504 (and / or other connected storage devices) can store instructions and data used in the features described herein. Memory 504 and any other type of memory (magnetic disk, optical disk, tape, or other tangible medium) can be considered a “memory” or “storage device.”

[0166] I / O interface 506 can provide functionality for enabling server device 500 to interface with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data storage 120), and input / output devices can communicate through interface 506. In some embodiments, the I / O interface can connect to interface devices that include input devices (keyboard, pointing device, touchscreen, microphone, camera, scanner, etc.) and / or output devices (display devices, speaker devices, printers, motors, etc.).

[0167] Audio / video input / output devices 514 can be a variety of devices, including user input devices (e.g., mouse, etc.) that can be used to receive user input, audio output devices (e.g., speakers), and display devices (e.g., screen, monitor, etc.) that can be used to provide graphical and / or visual output, and / or combined input and display devices.

[0168] For ease of explanation, Figure 5 One block is shown for each of processor 502, memory 504, I / O interface 506, and software blocks of operating system 508 and virtual experience application 510. These blocks can represent one or more processors or processing circuits, operating systems, memories, I / O interfaces, applications, and / or software engines. In other embodiments, device 500 can not have all of the components shown and / or can have other types of components instead of, or in addition to, those shown. While online virtual experience server 102 is described as performing operations as described in some embodiments herein, any suitable component or combination of components of online virtual experience server 102, client device 110, or similar system, or any suitable processor or processors associated with such a system, can perform the described operations.

[0169] Device 500 can be a server device or a client device. An example client device or user device can be a computer device that includes some similar components to device 500, such as processor 502, memory 504, and I / O interface 506. An operating system, software, and applications suitable for a client device can be provided in memory and used by the processor. The I / O interface for a client device can connect to network communication devices as well as input and output devices, such as a microphone for capturing sound, a camera for capturing images or video, a mouse for capturing user input, a gesture device for recognizing user gestures, a touch screen for detecting user input, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices. For example, a display device within audio / video input / output device 514 can be connected to (or included in) device 500 to display pre-processed and post-processed images as described herein, where such a display device can include any suitable display device, such as an LCD, LED, or plasma display screen, CRT, television, monitor, touch screen, 3-D display screen, projector, or other visual display device. Some implementations can provide an audio output device, such as a synthesized voice that speaks or reads text.

[0170] One or more methods described herein (e.g., method 600) can be implemented by computer program instructions or code that can be executed on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry), and can be stored on a computer program product including a non-transitory computer readable medium (e.g., a storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium (including, e.g., semiconductor or solid-state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), flash memory, a rigid disk, an optical disk, a solid-state drive, etc.). The program instructions can also be contained in, and provided as, an electronic signal, such as in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.) or a combination of hardware and software. Example hardware can be a programmable processor (e.g., a field-programmable gate array (FPGA), a complex programmable logic device), a general purpose processor, a graphics processor, an application specific integrated circuit (ASIC), etc. One or more methods can be executed as part of or in conjunction with an application running on a system, or as an application or software running with other applications and an operating system.

[0171] One or more of the methods described herein can run in a standalone program that can run on any type of computing device, a program that runs on a web browser, a mobile application ("app") that runs on a mobile computing device (e.g., a phone, a smartphone, a tablet, a wearable device (watch, armband, jewelry, headwear, eyewear, glasses, etc.), a laptop, etc.). In one example, a client / server architecture can be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives final output data from the server for output (e.g., for display). In another example, all computations can be performed within a mobile application (and / or other application) on the mobile computing device. In another example, computations can be split between the mobile computing device and one or more server devices.

[0172] While the specific implementations have been described in connection with specific embodiments thereof, these specific implementations are merely for illustration and not limitation. The concepts illustrated in the various examples can be applied to other examples and implementations.

[0173] The functional blocks, operations, features, methods, devices, and systems described in this disclosure can be integrated or divided into different combinations of systems, devices, and functional blocks known to those skilled in the art. The routines of the specific implementations can be implemented using any suitable programming language and programming techniques. Different programming techniques can be employed, e.g., procedural or object-oriented. The routines can be executed on a single processing device or multiple processors. Although the steps, blocks, operations, or computations can be presented in a particular order, this order can be changed in different particular implementations. In some implementations, multiple steps or operations shown as sequential in this specification can be performed at the same time.

Claims

1. A computer-implemented method comprising: accessing a graph representing a motion control policy for a simulated agent in a virtual environment, the graph comprising a plurality of nodes, the nodes comprising a body of the simulated agent and features of the virtual environment; encoding information of the graph, the information comprising one or more of: a position, a velocity, or a direction of each node, the information comprising relational information of pairs of nodes connected by edges; processing the encoded information through a graph neural network to generate one or more latent embeddings of the nodes of the graph; determining a fixed-length latent vector from the one or more latent embeddings; sending the fixed-length latent vector as input to a feedforward neural network to output a control signal for the simulated agent; applying the control signal to the body of the simulated agent in the virtual environment to provide an action of the simulated agent; evaluating the action of the simulated agent through a reward function to assess a motion quality of the action and an interaction quality of the action; and adapting the control policy through reinforcement learning by iteratively adjusting the graph based on the evaluation via the reward function, wherein the adaptation generalizes to one or both of: different agent morphologies in the virtual environment and different interactions. Determining the fixed-length latent vector comprises discarding a portion of the one or more latent embeddings to remove information related to a subset of the nodes.

2. The method of claim 1, wherein, The graph neural network comprises one or more graph attention network (GAT) layers.

3. The method of claim 1, wherein, The feedforward neural network comprises a multilayer perceptron (MLP) comprising a series of fully connected layers.

4. The method of claim 1, wherein, The virtual environment comprises at least one additional agent as part of the motion control policy, the simulated agent interacting with the at least one additional agent.

5. The method of claim 1, wherein, The nodes of the graph represent a plurality of internal points on the simulated agent and a plurality of external points external to the simulated agent.

6. The method of claim 1, wherein, The edges of the graph represent nearest neighbors of each node determined through a K-nearest neighbor (KNN) algorithm.

7. The method of claim 1, wherein, The encoding further comprises calculating an angular velocity of each node.

8. The method of claim 1, wherein, The graph is dynamically updated based on interactions of the agent in the simulated environment.

9. The method of claim 1, wherein, The graph is dynamically updated based on changing positions of target agents or objects in the simulated environment.

10. The method of claim 9, wherein, Discarding the portion of the latent embeddings comprises selectively compressing information specific to one or more of the nodes in the graph.

11. The method of claim 1, wherein, 12. A system comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising: accessing a graph representing a motion control policy for a simulated agent in a virtual environment, the graph comprising a plurality of nodes, the nodes comprising a body of the simulated agent and features of the virtual environment; encoding information of the graph, the information comprising one or more of: a position, a velocity, or a direction of each node, the information comprising relational information of pairs of nodes connected by edges; processing the encoded information through a graph neural network to generate one or more latent embeddings of the nodes of the graph; ​ ​ discarding a portion of the one or more latent embeddings to remove information related to the subset of the nodes to obtain a fixed-length latent vector; sending the fixed-length latent vector as input to a feedforward neural network to output a control signal for the simulated agent; applying the control signal to the body of the simulated agent in the virtual environment to provide an action of the simulated agent; evaluating the action of the simulated agent through a reward function to assess a motion quality of the action and an interaction quality of the action; and adapting the control policy through reinforcement learning by iteratively adjusting the graph based on the evaluation through the reward function, wherein the adaptation generalizes to one or both of: different agent morphologies and different interactions in the virtual environment.

13. The system of claim 12, wherein, The reward function includes one or more of: joint angle matching, joint velocity matching, end-effector position matching, or root state matching.

14. The system of claim 12, wherein, The reward function includes a product of a joint matching term and an interaction matching term.

15. The system of claim 12, wherein, The control signal determines one or both of: an angle or a torque of a joint in the simulated agent.

16. The system of claim 12, wherein, The instructions further cause the system to perform operations comprising: randomly sampling at a training phase to introduce additional nodes.

17. The method of claim 12, wherein: The graph neural network includes one or more graph attention network (GAT) layers; and The feedforward neural network includes a multilayer perceptron (MLP) that includes a series of fully connected layers.

18. The system of claim 12, wherein, The operations of adapting the control policy include transfer learning for additional simulated agents, wherein the simulated agents perform tasks involving synchronized actions.

19. The system of claim 12, wherein, An output of the feedforward neural network is used to generate a database of motion patterns for one or both of: a plurality of different agent morphologies, and a plurality of different interaction scenarios.

20. A non-transitory computer-readable medium containing instructions that, when executed by a processor, cause the processor to perform operations comprising: accessing a graph representing a motion control policy for a simulated agent in a virtual environment, the graph including a plurality of nodes including a body of the simulated agent and features of the virtual environment; encoding information of the graph, the information including one or more of: a position, a velocity, or a direction of each node, the information including relationship information of pairs of nodes connected by edges; processing the encoded information through a graph neural network to generate one or more latent embeddings of the nodes of the graph; modifying a portion of the latent embeddings to obtain a fixed-length latent vector; sending the fixed-length latent vector as input to a feedforward neural network to output a control signal for the simulated agent; applying the control signal to the body of the simulated agent in the virtual environment to provide an action of the simulated agent; evaluating the action of the simulated agent through a reward function to assess a motion quality of the action and an interaction quality of the action; and The control policy is adapted by reinforcement learning by iteratively adjusting the graph based on the evaluation via the reward function, wherein the adaptation generalizes to one or both of: different character morphologies and different interactions in the virtual environment.

Images