Adapting simulated character interactions to different forms and interaction scenarios

Abstract

To provide a character animation method or the like capable of seamlessly being adapted to different character forms and dynamically reacting to complexity of various interactions.SOLUTION: A method accesses a graph representing a control policy for movement of a character simulated in a virtual environment. The graph is encoded and processed through a graph neural network to generate a latent embedding for the graph. A latent vector of fixed length is determined from the latent embedding. The vector is input to a feed-forward neural network to generate control signals for the character's actions. Through a reinforcement learning loop, the character's movement iteratively adjusts the graph on the basis of an evaluation of the simulated character's actions via the reward function, and is continuously remodeled by adapting the control policy to different character forms and interaction scenarios.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] Implementations relate generally to the field of character simulation in virtual environments, and more particularly to methods and systems for accessing and processing motion control policies for applying simulated character movement and interaction to different morphologies and interaction scenarios. [Background technology]

[0002] Simulated environments and virtual character interactions are a significant source of attention for developers of video games and virtual experiences. A current focus in this area is on extending the realism and adaptability of character animation. Existing methodologies often rely on predefined motion capture data or hand-crafted animations, limiting the versatility and adaptability of characters within virtual environments. These approaches face challenges in translating specific character movements into a variety of different character forms and interaction scenarios.

[0003] The limitations of current systems become apparent when they attempt to generalize motion control policies across different character types and character interactions. The lack of a robust framework for automatically adapting control policies to various character morphologies and interaction contexts presents a significant obstacle. Existing approaches often require manual tuning and lack the ability to dynamically respond to changing virtual scenarios. Furthermore, these methodologies can struggle to achieve nuanced and contextually appropriate character interactions, especially in complex virtual environments where characters must perform a variety of actions.

[0004] Thus, there is a need in the art for methods and systems for smoothing character animation that can seamlessly adapt to different character morphologies and dynamically respond to the complexities of diverse virtual interactions. Addressing these challenges will improve the field of character simulation in virtual environments and ensure a more immersive and flexible experience for users of such virtual environments.

[0005] The background art discussion provided herein is generally intended to provide an overview of the context for the present disclosure. The work of the presently named inventors is not expressly or implicitly admitted as prior art to the present disclosure to the extent that it is within this background art section, and aspects of the present disclosure that, in some cases, may not qualify as prior art at the time of filing. Summary of the Invention [Means for solving the problem]

[0006] Implementations relate to adapting simulated character interactions to different morphologies and interaction scenarios. In some implementations, a system accesses a graph representing a control policy for the movement of a simulated character in a virtual environment. The graph is encoded and processed through a graph neural network to generate a latent embedding for the graph. A fixed-length latent vector is determined from the latent embedding. The vector is fed as an input into a feedforward neural network to generate control signals for the character's actions. Through a reinforcement learning loop, the character's motion is continuously refined by iteratively adjusting the graph based on an evaluation of the simulated character's actions via a reward function, adapting the control policy to different character morphologies and / or interaction scenarios.

[0007] For example, in some implementations, a computer-implemented method includes accessing a graph representing a motor control policy for a simulated character in a virtual environment, the graph including a plurality of nodes, the nodes including features of the simulated character's body and the virtual environment. The method includes encoding information about the graph, including one or more of a position, a velocity, or an orientation of each node, the information including relational information for pairs of nodes connected by edges. The encoded information is processed through a graph neural network to generate one or more latent embeddings for the nodes of the graph. The method includes determining fixed-length latent vectors from the one or more latent embeddings and sending the fixed-length latent vectors as inputs to a feedforward neural network to output control signals for the simulated character. The method includes applying the control signals to the body of the simulated character in the virtual environment to provide actions for the simulated character, and evaluating the actions of the simulated character via a reward function to assess the movement quality of the actions and the interaction quality of the actions. The method includes adapting a control policy through reinforcement learning by iteratively adjusting a graph based on evaluation via a reward function, wherein the adaptation generalizes to one or both of different character morphologies and different interactions within the virtual environment.

[0008] Various implementations and examples of the method are described. For example, in some implementations, determining a fixed-length latent vector includes discarding a portion of one or more latent embeddings to remove information about a subset of nodes. In some implementations, the graph neural network includes one or more graph attention network (GAT) layers. In some implementations, the feedforward neural network comprises a multi-layer perceptron (MLP) including a sequence of fully connected layers. In some implementations, the virtual environment includes at least one additional character with which the simulated character interacts as part of a motion control policy. In some implementations, the nodes of the graph represent multiple in-points on the simulated character and multiple out-points outside the simulated character. In some implementations, the edges of the graph represent nearest neighbors of each node as determined by a K-nearest neighbor (KNN) algorithm. In some implementations, the encoding further includes calculating angular velocities for each node. In some implementations, the graph is dynamically updated based on character interactions in the simulated environment. In some implementations, the graph is dynamically updated based on the changing location of the target character or object in the simulated environment. In some implementations, discarding portions of the potential embeddings includes selectively compressing information specific to one or more of the nodes in the graph.

[0009] In some implementations, a system includes one or more processors coupled to a memory having stored thereon software instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. These operations include accessing a graph representing a motion control policy for a simulated character in a virtual environment, the graph including a plurality of nodes, the nodes including features of the body of the simulated character and the virtual environment. These operations include encoding information about the graph including one or more of a position, a velocity, or an orientation of each node, the information including relational information for pairs of nodes connected by edges, and processing the encoded information through a graph neural network to generate one or more latent embeddings for the nodes of the graph. These operations include discarding a portion of the one or more latent embeddings to remove information about a subset of the nodes to obtain a fixed-length latent vector, and sending the fixed-length latent vector as an input to a feedforward neural network to output a control signal for the simulated character. These operations include adapting control signals to a body of a simulated character in a virtual environment to provide an action for the simulated character, and evaluating the action of the simulated character via a reward function to assess the motor quality of the action and the interaction quality of the action. These operations include adapting a control policy through reinforcement learning by iteratively adjusting a graph based on the evaluation via the reward function, where the adaptation generalizes to one or both of different character morphologies and different interactions within the virtual environment.

[0010] Various implementations and examples of the system are described. For example, in some implementations, the reward function includes one or more of joint angle matching, joint velocity matching, end-effector position matching, or root state matching. In some implementations, the reward function includes a product of a joint matching term and an interaction matching term. In some implementations, the control signal determines one or both of joint angles or torques in the simulated character. In some implementations, the instructions further cause the system to perform an operation including random sampling to introduce additional nodes during the training phase. In some implementations, the graph neural network includes one or more graph attention network (GAT) layers, and the feedforward neural network includes a multilayer perceptron (MLP) including a sequence of fully connected layers. In some implementations, the operation of adapting the control policy includes transfer learning for additional simulated characters, where the simulated characters perform tasks related to synchronized actions. In some implementations, the output of the feedforward neural network is utilized to generate a database of movement patterns for one or both of a plurality of different character forms and a plurality of different interaction scenarios.

[0011] In some implementations, a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor, cause the processor to perform operations. These operations include accessing a graph representing a motion control policy for a simulated character in a virtual environment, the graph including a plurality of nodes, the nodes including features of the simulated character's body and the virtual environment. These operations include encoding information about the graph including one or more of a position, a velocity, or an orientation of each node, the information including relational information for pairs of nodes connected by edges. These operations include processing the encoded information through a graph neural network to generate one or more latent embeddings for the nodes of the graph, modifying a portion of the latent embeddings to obtain fixed-length latent vectors, and sending the fixed-length latent vectors as inputs to a feedforward neural network to output control signals for the simulated character. These operations include adapting control signals to a body of a simulated character within a virtual environment to provide an action for the simulated character; evaluating the action of the simulated character via a reward function to assess the motor quality of the action and the interaction quality of the action; and adapting a control policy through reinforcement learning by iteratively adjusting a graph based on the evaluation via the reward function, wherein the adaptation generalizes to one or both of different character morphologies and different interactions within the virtual environment.

[0012] Some implementations may include a system including a processor and a memory coupled to the processor. The memory may have instructions stored thereon that, when executed by the processor, cause the processor to perform operations including one or more of the methods and / or computer-readable medium features described above. Some implementations may include a computer-implemented method including one or more of the operations performed by the processor and / or via the computer-readable storage medium of the systems described above. Some implementations include a non-transitory computer-readable medium having instructions stored thereon that, when executed by the processor, cause the processor to perform operations that may be the same as or similar to the methods and / or system features described above. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 illustrates an exemplary system architecture for adapting simulated character interactions to different forms and interaction scenarios, according to some implementations. [Figure 2] 1 is a flow diagram illustrating a method for adapting simulated character interactions to different forms and interaction scenarios, according to some implementations. [Figure 3] A diagram illustrating how control signals for simulated character movement are generated and applied from control policies according to some implementations. [Figure 4A] FIG. 1 illustrates an example of a simulated character in a virtual environment, according to some implementations. [Figure 4B] FIG. 1 illustrates an example of reinforcement learning for multi-character interactions, according to some implementations. [Figure 5] FIG. 1 is a block diagram illustrating an exemplary computing device, according to some implementations. DETAILED DESCRIPTION OF THE INVENTION

[0014] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, like numerals generally identify like elements unless the context dictates otherwise. The exemplary implementations described in the detailed description, drawings, and claims are not meant to be limiting. Other implementations may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. The aspects of the present disclosure, as generally described herein and illustrated in the drawings, can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are contemplated herein.

[0015] References herein to "some implementations," "implementations," "exemplary implementations," and the like indicate that the described implementations may include a particular feature, structure, or characteristic, but that all implementations may not necessarily include that particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same implementations. Furthermore, when a particular feature, structure, or characteristic is described with respect to an implementation, such feature, structure, or characteristic may also be achieved with respect to other implementations, whether or not explicitly described.

[0016] One or more implementations described herein relate to adapting simulated character interactions to different forms and / or interaction scenarios. Some implementations describe a general neural network pipeline capable of processing any spatial representation of a scene. The spatial information is processed by layers of a graph neural network (GNN) (e.g., a graph attention network) and cast into a latent space. The latent embedding can be modified to obtain latent vectors; for example, a certain amount of redundant information in the latent space can be discarded, resulting in fixed-size latent variables that can capture the desired movement properties. The fixed-size latent variables can be processed through a feedforward neural network (e.g., an MLP network). This network architecture allows the neural network to learn a single generalized policy that can be applied to a wide variety of character shapes and forms and / or used to represent different physical interaction scenarios between the character and the surrounding environment in a virtual environment.

[0017] In some examples, a multi-agent reinforcement learning process is provided in which a control policy is trained to generate control signals for a simulated character. At each control step, the policy may receive inputs of observations describing the state of the simulated character as well as reference future poses. The policy outputs joint angles that are used to calculate torques for each joint of the character. A reward function is used to measure both the character's movement quality and interaction quality.

[0018] In one or more implementations, the described techniques may enable a single policy to control a character's interaction with its surroundings, assuming a variable number of spatial locations that the character can attend. In some implementations, the character and local environmental state may be represented using a graph structure in which graph nodes are placed on the character's body parts and also on other objects in the environment surrounding the character (e.g., other characters or other objects). In some implementations, the edges of the graph are constructed by computing a number of nearby K-nearest neighbors for each node and connecting the nearby nodes.

[0019] In one or more implementations, the interaction graph can be considered as an observation, which may include, for example, the position, velocity, orientation, and angular velocity of every graph node, as well as the difference in information within node pairs of every edge. The number of nodes and edges outside the controlled character is not fixed but variable. In some implementations, a GNN-based neural network architecture is used, which can directly process the graph structure and deal with a variable number of external nodes (outside the character) by maintaining a fixed-length latent specification after encoding and processing the graph. The latent variables are fed into a series of fully connected layers to output actions.

[0020] In some implementations, the described techniques can train a policy for a single character; for example, such a policy does not depend on the specific number of environmental objects interacting with the character. The single-character policy can be used to train a generalized policy for a variety of different character interactions with other objects and / or to provide different character morphologies. For example, during training of a single character, multiple nodes surrounding different body parts of the character can be randomly sampled in addition to existing nodes. The interaction graph reward and motor imitation reward are then optimized so that the single-character policy can attend to different morphologies and / or region interactions. The pre-trained model can then be used as a base policy for further multi-character interaction training.

[0021] In some implementations, to preserve both movement quality and interaction quality, the reward function may be the product of two parts: a joint consistency term and an interaction consistency term. The joint consistency reward term measures the joint angle, joint velocity, end-effector position, and root state consistency between the simulated character and the reference character. The interaction consistency term measures edge differences in the graph, where edges connect graph nodes external to the character.

[0022] Advantages may include providing a reinforcement learning formulation that enables training control policies capable of controlling characters to perform various levels of realistic physical movement and interaction to recognize various external environmental conditions around a character or other object. The training control policies may be generalized to address different environments and / or different character morphologies. The described features may, for example, facilitate the implementation of a general simulation and training pipeline for physically simulated character control in which a user may set up and execute training missions for a general simulated movement-mimicking controller with few inputs, such as changes to parameters. The described techniques may provide realistic physics simulation and interaction for a variety of different types of characters in a virtual experience, thereby improving the immersive experience for the user.

[0023] The described features provide several technical advantages over previous techniques for providing control policies for characters in virtual environments. For example, the described features provide technical advantages that enable reduced computational resource usage (e.g., computer memory, processor usage and time, networking traffic bandwidth, display processing and memory utilization, etc.) in various described implementations. For example, previous control policies were fixed to specific environments and did not generalize to unseen environments due to the controlled character's lack of awareness of its surroundings. Therefore, different control policies were specifically trained and used for different environments. In contrast, the described control policies are adaptable to a variable number of nodes (e.g., external nodes) and / or a reduced number of nodes (e.g., after discarding nodes) and thus may be used as generalized control policies. This allows a single control policy to be trained and used in various environments having varying numbers of objects, thus saving computational resources that would otherwise be used in training and storing multiple control policies.

[0024] Another exemplary technical advantage is that the described features enable a virtual environment having digital characters, e.g., avatars, to use a control policy to animate the avatar in a way that the control policy can generalize to any kind of character and any kind of interaction, i.e., with any element, object, or character in the scene. For example, this includes generalizing the control policy to characters with any skeletal form or expression, or any character that a user of the virtual environment may be able to create. For example, this includes generalizing the control policy to interactions with any aspect of geometry in the virtual environment, not just avatars or objects. This allows a single control policy to be trained and used for a variety of characters, thus saving computational resources that would potentially be used in training and storing multiple control policies for multiple types of characters.

[0025] 1 is a diagram of an exemplary system architecture that can be used to adapt simulated character interactions to different forms and interaction scenarios, according to some implementations. FIG. 1 and other figures use similar reference numbers to identify similar elements. A letter after a reference number, such as "110," indicates that the text specifically refers to the element with that particular reference number. A reference number in text without a following letter, such as "110," refers to any or all of the elements in the figure with the reference number (e.g., "110" in the text refers to reference numbers "110a," "110b," and / or "110n" in the figures).

[0026] System architecture 100 (also referred to herein as the "system") includes an online virtual experience server 102, a data store 120, client devices 110a, 110b, and 110n (generally referred to herein as "client devices 110"), and developer devices 130a and 130n (generally referred to herein as "developer devices 130"). The virtual experience server 102, the data store 120, the client devices 110, and the developer devices 130 are coupled via a network 122. In some implementations, the client devices 110 and the developer devices 130 may refer to the same or the same type of device.

[0027] The online virtual experience server 102 may include, among other things, a virtual experience engine 104, one or more virtual experiences 106, and a graphics engine 108. In some implementations, the graphics engine 108 may be a system, application, or module that allows the online virtual experience server 102 to provide graphics and animation capabilities. In some implementations, the graphics engine 108 may perform one or more of the operations described below with respect to the flowchart shown in FIG. 2. The client device 110 may include a virtual experience application 112 and an input / output (I / O) interface 114 (e.g., input / output devices). The input / output devices may include one or more of a microphone, speakers, headphones, a display device, a mouse, a keyboard, a game controller, a touchscreen, a virtual reality console, etc.

[0028] The developer device 130 may include a virtual experience application 132 and an input / output (I / O) interface 134 (e.g., input / output devices). The input / output devices may include one or more of a microphone, speakers, headphones, a display device, a mouse, a keyboard, a game controller, a touchscreen, a virtual reality console, etc.

[0029] System architecture 100 is provided for illustrative purposes. In different implementations, system architecture 100 may include the same, fewer, more, or different elements, arranged in the same or different manner as shown in FIG.

[0030] In some implementations, the network 122 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network, a Wi-Fi network, or a wireless LAN (WLAN)), a cellular network (e.g., a 5G network, a Long Term Evolution (LTE) network, etc.), a router, a hub, a switch, a server computer, or a combination thereof.

[0031] In some implementations, data store 120 may be non-transitory computer-readable memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data. Data store 120 may include multiple storage components (e.g., multiple drives or multiple databases), which may also span multiple computing devices (e.g., multiple server computers). In some implementations, data store 120 may include cloud-based storage.

[0032] In some implementations, the online virtual experience server 102 may include a server having one or more computing devices (e.g., a cloud computing system, a rack-mounted server, a server computer, a cluster of physical servers, etc.) In some implementations, the online virtual experience server 102 may be a standalone system, may include multiple servers, or may be part of another system or server.

[0033] In some implementations, the online virtual experience server 102 may include one or more computing devices (such as a rack-mounted server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memory, databases), networks, software components, and / or hardware components that may be used to perform operations on the online virtual experience server 102 and provide users with access to the online virtual experience server 102. The online virtual experience server 102 may also include websites (e.g., web pages) or application backend software that may be used to provide users with access to content provided by the online virtual experience server 102. For example, a user may access the online virtual experience server 102 using a virtual experience application 112 on a client device 110.

[0034] In some implementations, virtual experience session data is generated via online virtual experience server 102, virtual experience application 112, and / or virtual experience application 132 and stored in data store 120. With permission from the virtual experience participants, the virtual experience session data may include associated metadata, such as a virtual experience identifier; device data associated with the participant; participant demographic information; a virtual experience session identifier; a chat transcript; session start time, session end time, and session duration for each participant; the relative location of participant avatars within the virtual experience environment; purchases made within the virtual experience by one or more participants; accessories utilized by participants, etc.

[0035] In some implementations, the online virtual experience server 102 may be a type of social network that provides connectivity between users or types of user-generated content systems, allowing users (e.g., end users or consumers) to communicate with other users on the online virtual experience server 102; the communications may include voice chat (e.g., synchronous and / or asynchronous voice communication), video chat (e.g., synchronous and / or asynchronous video communication), or text chat (e.g., 1:1 and / or N:N synchronous and / or asynchronous text-based communication). Records of some or all user communications may be stored in the data store 120 or within the virtual experience 106. The data store 120 may be utilized to store chat transcripts (text, audio, images, etc.) exchanged between participants.

[0036] In some implementations of the present disclosure, a "user" may be represented as a single individual. However, other implementations of the present disclosure encompass "users" (e.g., creating users) that are a set of users or entities controlled by an automated source. For example, a set of individual users organized as a community or group within a user-generated content system may be considered a "user."

[0037] In some implementations, the online virtual experience server 102 may be a virtual game server. For example, the game server may provide single-player or multiplayer games to a community of users who may access the “system” herein, including the online game server 102, the data store 120, and the client devices 110, and / or who may interact with the virtual experience using the client devices 110 via the network 122. In some implementations, the virtual experience (including a virtual realm or world, a virtual game, or other computer-simulated environment) may be, for example, a two-dimensional (2D) virtual experience, a three-dimensional (3D) virtual experience (e.g., a 3D user-generated virtual experience), a virtual reality (VR) experience, or an augmented reality (AR) experience. In some implementations, users may participate in interactions (e.g., gameplay) with other users. In some implementations, the virtual experience can be experienced in real time with other users of the virtual experience.

[0038] In some implementations, virtual experience engagement may refer to the interaction of one or more participants using a client device (e.g., 110) within a virtual experience (e.g., 106), or the presentation of the interaction on a display or other output device (e.g., 114) of the client device 110. For example, virtual experience engagement may include interaction with one or more participants within the virtual experience, or the presentation of the interaction on a display of the client device.

[0039] In some implementations, the virtual experience 106 may include electronic files that may be executed or loaded using software, firmware, or hardware configured to present virtual experience content (e.g., digital media items) to an entity. In some implementations, the virtual experience application 112 may execute in association with the virtual experience engine 104 to render the virtual experience 106. In some implementations, the virtual experiences 106 may have a common set of rules or a common goal, and the environments of the virtual experiences 106 share a common set of rules or a common goal. In some implementations, different virtual experiences may have different rules or goals from each other.

[0040] In some implementations, a virtual experience may have one or more environments (sometimes referred to herein as "virtual experience environments" or "virtual environments"), and multiple environments may be linked. An example of an environment may be a three-dimensional (3D) environment. One or more environments of a virtual experience 106 may be collectively referred to herein as a "world" or "virtual experience world" or "game world" or "virtual world" or "virtual space" or "universe." An example of a world may be the 3D world of the virtual experience 106. For example, a user may build a virtual environment that is linked to another virtual environment created by another user. Characters of the virtual experience may cross virtual boundaries and enter adjacent virtual environments.

[0041] It should be noted that a 3D environment or world uses graphics that use three-dimensional representations of geometric data that represent the virtual experiential content (or at least presents the virtual experiential content to appear as 3D content, regardless of whether 3D representations of geometric data are used), and a 2D environment or world uses graphics that use two-dimensional representations of geometric data that represent the virtual experiential content.

[0042] In some implementations, the online virtual experience server 102 may host one or more virtual experiences 106 and may allow users to interact with the virtual experiences 106 using virtual experience applications 112 on client devices 110. Users of the online virtual experience server 102 may play the virtual experiences 106, create the virtual experiences 106, interact with the virtual experiences 106, build the virtual experiences 106, communicate with other users, and / or create and build objects (e.g., also referred to as "items" or "virtual experience objects" or "virtual experience items") of the virtual experiences 106.

[0043] For example, in creating a user-generated virtual item, a user may create a character, decorations for a character, one or more virtual environments for an interactive virtual experience, or build structures used in the virtual experience 106, among other things. In some implementations, a user may buy, sell, or trade virtual experience objects, such as in-platform currency (e.g., virtual currency), with other users of the online virtual experience server 102. In some implementations, the online virtual experience server 102 may transmit virtual experience content to a virtual experience application (e.g., 112). In some implementations, virtual experience content (also referred to herein as "content") may refer to any data or software instructions (e.g., virtual experience objects, virtual experiences, user information, video, images, commands, media items, etc.) associated with the online virtual experience server 102 or the virtual experience application. In some implementations, a virtual experience object (e.g., also referred to herein as an “item” or “object” or “virtual object” or “virtual experience item”) may refer to an object used, created, shared, or possibly shown in a virtual experience application 106 on an online virtual experience server 102 or a virtual experience application 112 on a client device 110. For example, a virtual experience object may include parts, models, characters, accessories, tools, weapons, clothing, buildings, vehicles, currency, flora and fauna, components of the foregoing (e.g., windows of a building), etc.

[0044] Note that the online virtual experience server 102 hosting the virtual experience 106 is provided for illustrative purposes. In some implementations, the online virtual experience server 102 may host one or more media items, which may include communication messages from one user to one or more other users. With user permission and explicit user agreement, the online virtual experience server 102 may analyze chat transcript data to improve the virtual experience platform. Media items may include, but are not limited to, digital videos, digital movies, digital photos, digital music, audio content, melodies, website content, social media updates, e-books, e-magazines, digital newspapers, digital audiobooks, e-journals, web blogs, real simple syndication (RSS) feeds, e-comic books, software applications, and the like. In some implementations, the media items may be electronic files that can be executed or loaded using software, firmware, or hardware configured to present the digital media items to entities.

[0045] In some implementations, a virtual experience 106 may be associated with a particular user or a particular group of users (e.g., a private virtual experience), or may be made generally available to users with access to the online virtual experience server 102 (e.g., a public virtual experience). In some implementations, when the online virtual experience server 102 associates one or more virtual experiences 106 with a particular user or group of users, the online virtual experience server 102 may associate the particular user with the virtual experience 106 using user account information (e.g., a user account identifier such as a username and password).

[0046] In some implementations, the online virtual experience server 102 or the client device 110 may include a virtual experience engine 104 or a virtual experience application 112. In some implementations, the virtual experience engine 104 may be used for developing or executing the virtual experience 106. For example, the virtual experience engine 104 may include, among other features, a rendering engine (“renderer”) for 2D graphics, 3D graphics, VR graphics, or AR graphics, a physics engine, a collision detection engine (and collision response), a sound engine, scripting capabilities, an animation engine, an artificial intelligence engine, networking capabilities, streaming capabilities, memory management capabilities, threading capabilities, scene graph capabilities, or video support for movies. Components of the virtual experience engine 104 may generate commands (e.g., rendering commands, collision commands, physics commands, etc.) that help compute and render the virtual experience. In some implementations, the virtual experience applications 112 of the client devices 110 may each operate independently, in cooperation with the virtual experience engine 104 of the online virtual experience server 102, or a combination of both.

[0047] In some implementations, both the online virtual experience server 102 and the client device 110 may run a virtual experience engine (104 and 112, respectively). The online virtual experience server 102, using its virtual experience engine 104, may perform some or all virtual experience engine functions (e.g., generate physics commands, rendering commands, etc.) or may offload some or all virtual experience engine functions to the virtual experience engine 104 of the client device 110. In some implementations, each virtual experience 106 may have a different ratio between the virtual experience engine functions implemented on the online virtual experience server 102 and the virtual experience engine functions implemented on the client device 110. For example, the virtual experience engine 104 of the online virtual experience server 102 may be used to generate physics commands when there is a collision between at least two virtual experience objects, and additional virtual experience engine functions (e.g., generate rendering commands) may be offloaded to the client device 110. In some implementations, the ratio of virtual experience engine functions implemented on the online virtual experience server 102 and the client device 110 may be changed (e.g., dynamically) based on virtual experience engagement conditions. For example, if the number of users engaging with a particular virtual experience 106 exceeds a threshold number, the online virtual experience server 102 may implement one or more virtual experience engine functions previously implemented by the client device 110.

[0048] For example, a user may be playing a virtual experience 106 on a client device 110 and may send control instructions (e.g., user input such as right, left, up, down, user selection, or character position and velocity information) to the online virtual experience server 102. After receiving the control instructions from the client device 110, the online virtual experience server 102 may send experience instructions (e.g., rendering commands, collision commands, position and velocity information of characters participating in a group experience, etc.) to the client device 110 based on the control instructions. For example, the online virtual experience server 102 may perform one or more logical operations on the control instructions (e.g., using the virtual experience engine 104) to generate experience instructions for the client device 110. In other cases, the online virtual experience server 102 may pass one or more of the control instructions from one client device 110 to other client devices participating in the virtual experience 106 (e.g., from client device 110a to client device 110b). Client device 110 may use the experience instructions to render the virtual experience for presentation on the display of client device 110 .

[0049] In some implementations, control instructions may refer to instructions indicating actions of a user's character within a virtual experience. For example, control instructions may include user inputs for controlling actions within the experience, such as right, left, up, down, user selection, gyroscope position and orientation data, force sensor data, etc. The control instructions may include character position and velocity information. In some implementations, the control instructions are sent directly to the online virtual experience server 102. In other implementations, the control instructions may be sent from the client device 110 to another client device (e.g., from client device 110b to client device 110n), where the other client device generates the experience instructions using a local virtual experience engine 104. The control instructions may include instructions to play a voice communication message or other sound from another user on an audio device (e.g., speaker, headphones, etc.), e.g., a voice communication or other sound generated using the audio spatialization techniques described herein.

[0050] In some implementations, experience instructions may refer to instructions that enable client device 110 to render a virtual experience, such as a multi-participant virtual experience. The experience instructions may include one or more of user input (e.g., control instructions), character position and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.).

[0051] In some implementations, a character (or, generally, a virtual experience object) is constructed from components that can be selected by the user, one or more of which automatically join together to assist the user in editing.

[0052] In some implementations, a character is implemented as a 3D model, including a surface representation (also known as a skin or mesh) and a hierarchical set of interconnected bones (also known as a skeleton or rig) used to depict the character. The rig may be utilized to animate the character and simulate movements and actions by the character. The 3D model may be represented as a data structure, and one or more parameters of the data structure may be modified to change various traits of the character, such as dimensions (height, width, girth, etc.), body type, movement style, number / type of body parts, proportions (e.g., shoulder-to-waist ratio), head size, etc.

[0053] One or more characters (also referred to herein as "avatars" or "models") may be associated with a user, and the user may control the characters to facilitate the user's interaction with the virtual experience 106.

[0054] In some implementations, a character may include components such as body parts (e.g., hair, arms, legs, etc.) and accessories (e.g., t-shirts, glasses, decorative images, tools, etc.). In some implementations, customizable character body parts include, among others, head type, body part type (arms, legs, torso, and hands), face type, hair type, and skin type. In some implementations, customizable accessories include clothing (e.g., shirts, pants, hats, shoes, glasses, etc.), weapons, or other tools.

[0055] In some implementations, for some asset types, e.g., shirts, pants, etc., the online virtual experience platform may provide users with access to simplified 3D virtual object models represented by a low-polygon mesh, e.g., between approximately 20 polygons and approximately 30 polygons.

[0056] In some implementations, the user can also control the scale of the character (e.g., height, width, or depth) or the scale of components of the character. In some implementations, the user may control the proportions of the character (e.g., stocky, anatomical, etc.). Note that in some implementations, the character may not include character virtual experience objects (e.g., body parts, etc.), although the user may control the character (without the character virtual experience objects) to facilitate the user's interaction with the virtual experience (e.g., a puzzle game where there is no rendered character game object, but you still control the character to control in-game actions).

[0057] In some implementations, components such as body parts may be simple geometric shapes such as blocks, cylinders, spheres, or some other simple shape such as a wedge, ring, tube, channel, etc. In some implementations, a creator module may publish a user's character for viewing or use by other users of the online virtual experience server 102. In some implementations, creation, modification, or customization of a character, other virtual experience object, virtual experience 106, or virtual experience environment may be performed by a user using an I / O interface (e.g., a developer interface), with or without scripting (or with or without an application programming interface (API)). Note that for purposes of illustration, characters are described as having a human form. Note further that characters may have any form, such as a vehicle, an animal, an animate or inanimate object, or other creative form.

[0058] In some implementations, the online virtual experience server 102 may store characters created by users in the data store 120. In some implementations, the online virtual experience server 102 maintains a character catalog and a virtual experience catalog, which may be presented to the user. In some implementations, the virtual experience catalog includes images of virtual experiences stored on the online virtual experience server 102. In addition, the user may select a character (e.g., a character created by the user or another user) from the character catalog to participate in a selected virtual experience. The character catalog includes images of characters stored on the online virtual experience server 102. In some implementations, one or more of the characters in the character catalog may have been created or customized by the user. In some implementations, the selected character may have a character setting that defines one or more of the character's components.

[0059] In some implementations, a user's character may include a configuration of components, and the configuration and appearance of the components, or more generally, the character's appearance, may be defined by a character setting. In some implementations, a user's character's character setting may be chosen at least in part by the user. In other implementations, a user may choose a character with a default character setting or a character setting chosen by another user. For example, a user may choose a default character from a character catalog with predefined character settings, and the user may further customize the default character by changing some of the character settings (e.g., adding a shirt with a customized logo). A character setting may be associated with a particular character by the online virtual experience server 102.

[0060] In some implementations, the client devices 110 may each include a computing device such as a personal computer (PC), a mobile device (e.g., a laptop, a mobile phone, a smartphone, a tablet computer, or a netbook computer), a network-connected television, a game console, etc. In some implementations, the client devices 110 may also be referred to as "user devices." In some implementations, one or more client devices 110 may connect to the online virtual experience server 102 at any one time. It may be noted that the number of client devices 110 is provided by way of example. In some implementations, any number of client devices 110 may be used.

[0061] In some implementations, each client device 110 may include a respective instance of a virtual experience application 112. In one implementation, the virtual experience application 112 may allow a user to use and interact with the online virtual experience server 102, such as controlling a virtual character in a virtual experience hosted by the online virtual experience server 102, or to view and upload content such as images, video items, web pages, documents, etc. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, present, or navigate content (e.g., virtual characters in a virtual environment) served by a web server. In another example, the virtual experience application may be a native application (e.g., a mobile application, app, virtual experience program, or game program) that is installed and runs locally to the client device 110 and allows a user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content (e.g., a web page, media viewer) to the user. In implementations, the virtual experience application may include an embedded media player (e.g., a Flash or HTML5 player) that is embedded within a web page.

[0062] According to aspects of the present disclosure, the virtual experience application may be an online virtual experience server application through which a user builds, creates, edits, and uploads content to the online virtual experience server 102, as well as interacts with the online virtual experience server 102 (e.g., engages in a virtual experience 106 hosted by the online virtual experience server 102). Thus, the virtual experience application may be provided to the client device 110 by the online virtual experience server 102. In another example, the virtual experience application may be an application downloaded from a server.

[0063] In some implementations, each developer device 130 may include a respective instance of a virtual experience application 132. In one implementation, the virtual experience application 132 may allow a developer to use and interact with the online virtual experience server 102, such as controlling a virtual character in a virtual experience hosted by the online virtual experience server 102, or viewing or uploading content for the virtual experience 106, images, video items, web pages, documents, and the like. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, present, or navigate content (e.g., virtual characters in a virtual environment, etc.) served by a web server. In another example, the virtual experience application may be a native application (e.g., a mobile application, app, virtual experience program, or game program) that is installed and runs locally on the client device 130 and allows a user to interact with the online virtual experience server 102. The virtual experience application may render, display, or present content (e.g., a web page, media viewer) to a user. In one implementation, the virtual experience application may include an embedded media player (e.g., a Flash or HTML5 player) that is embedded within a web page.

[0064] According to aspects of the present disclosure, the virtual experience application 132 may be an online virtual experience server application through which a user builds, creates, edits, and uploads content to the online virtual experience server 102, and interacts with the online virtual experience server 102 (e.g., to provide and / or engage in virtual experiences 106 hosted by the online virtual experience server 102). Thus, the virtual experience application may be provided to the client device 130 by the online virtual experience server 102. In another example, the virtual experience application 132 may be an application downloaded from the server. The virtual experience application 132 may be configured to interact with the online virtual experience server 102 and obtain access to user credentials, user currency, etc., for one or more virtual experiences 106 developed, hosted, or provided by a virtual experience developer.

[0065] In some implementations, a user may log in to the online virtual experience server 102 through a virtual experience application. A user can access a user account by providing user account information (e.g., a username and password), and the user account is associated with one or more characters that can be used to participate in one or more virtual experiences 106 on the online virtual experience server 102. In some implementations, with the appropriate credentials, a virtual experience developer may gain access to virtual experience virtual objects, such as in-platform currency (e.g., virtual currency), avatars, special abilities, accessories, etc., owned by or associated with other users.

[0066] In general, functionality described in one implementation as being performed by the online virtual experience server 102 may be performed by the client device 110 or, in other implementations, by the server, where appropriate. Additionally, functionality attributed to a particular component may be performed by different or multiple components operating together. The online virtual experience server 102 may be accessed as a service offered to other systems or devices through a suitable application programming interface (API) and, therefore, is not limited to use within a website.

[0067] 2 illustrates how simulated character interactions can be adapted to different forms and interaction scenarios. In various embodiments, the blocks illustrated in FIG. 2 and described below can be implemented by any of the elements illustrated in FIG.

[0068] In block 202, the system accesses a graph representing a motion control policy for a simulated character in a virtual environment (e.g., a simulated environment, a virtual space, etc.). In some implementations, this graph may be considered a base control policy. In some implementations, the graph is a spatial graph, and the graph includes multiple nodes, each node corresponding to a component of the simulated character's anatomy or an element in the character's environment. In some implementations, links may be established between particular nodes to indicate anatomical components of the simulated character, such as body parts (e.g., limbs or portions thereof, such as upper arms, lower arms, upper legs, etc.). In some implementations, the nodes include data about the character or environment, such as the position, velocity, and orientation of each character or environment component.

[0069] A character has a particular morphology, and different characters may have different morphologies. A morphology may be characterized by a skeleton of nodes or links. In some examples, characters with similar body parts but different limb lengths may have different morphologies. In a further example, characters with different numbers of arms, heads, or other body parts have different morphologies.

[0070] Edges in a graph define relationships between nodes. For example, edges may indicate connectivity between particular nodes. These relationships may indicate, for example, articulations between anatomical components, such as limb joints and limbs, or interactions between a character and an environmental object. Edges enable identification and differentiation of interactions within a virtual environment. For example, a graph edge may connect two adjacent joints, such as in a representation of a particular limb of a character. A graph edge can also connect any two nodes in a graph, which may not be adjacent. A node in a graph may be "external" to a character, in which case it does not represent a joint on the character's body. Graph edges may therefore be used to represent any type of connection between nodes in a graph.

[0071] As used herein, "interaction" refers to dynamic engagement between simulated characters and elements (e.g., other characters or objects external to the simulated characters) in a virtual environment. This encompasses a spectrum of activities, such as the movement of characters relative to their own body parts or external characters or objects. For example, the movement of an avatar's hand relative to its shoulder or the manipulation of an object in virtual space may be considered a distinct interaction.

[0072] A graph structure representing the control policy serves as an overall framework for encapsulating these interactions. Each node in the graph corresponds to a key feature of the simulated character's body and virtual environment, forming a representation that mirrors the character's morphology and any external objects with which the character interacts. Edges in the graph represent connections between these nodes, encapsulating relational information and spatial dependencies.

[0073] As used herein, an "interaction scenario" refers to control policy information regarding different contexts and factors that influence how a simulated character conducts any given interaction. This may include, for example, specific conditions, objects, or entities associated with the interaction. For example, an interaction scenario for a character lifting a cup from a table may unfold as the character encounters different objects, such as a ball, a trophy, a pen, or a phone. In another example, the same character may pick up the same ball in different ways if they pick it up in a storm or if various other external objects are present in the path the character must navigate around. In this way, an interaction scenario represents a tailored instance of engagement that reflects the adaptability and flexibility of control policies to navigate different virtual environments and scenarios and respond appropriately to numerous situational nuances.

[0074] In some implementations, the system generates a graph data structure (or other representation) representing a graph by training a neural network on different examples of simulated character movements. Each such movement of these characters may be represented, for example, as a different interaction graph. The system then generates a graph representing the neural network trained for the particular movement interaction. This graph is a movement control policy that, when executed on the character, functions to reproduce the movement interaction. The system then accesses this graph representing the movement control policy.

[0075] In some implementations, the system accesses the graph by retrieving a graph data structure from a storage medium. In some implementations, this graph data structure models the state of both the character and associated environmental entities through a network of nodes and edges. Nodes are assigned to specific points that represent the character's body and environment, and edges represent connections that define their relationships.

[0076] In some implementations, the nodes may embody positional data and potentially other kinematic data for parts of the character's body, such as, for example, the character's joints and / or limbs, and environmental objects with which the character can interact. In some examples, each node may represent the head, neck, each hand, each wrist, each forearm, each upper arm, each upper leg, each lower leg, each ankle, each foot, etc. (See FIGS. 4A and 4B for some examples). Edges connect these nodes to formulate possible interactions within a graph framework. The graph may serve as input to a motion control policy capable of rendering the character's interactions with its surroundings. In some implementations, the system can retrieve the graph through a computational method, such as, for example, a query or read operation, which extracts the graph from its storage location and places it in working memory for processing.

[0077] In some implementations, the simulated character represents a human avatar using a skeletal structure. In some implementations, the skeletal structure within such an avatar may be configurable using the number and arrangement of bones. This configuration allows the system to accommodate diverse forms of human avatars across different implementations. Other forms of avatars may also be used, such as avatars representing animals or objects.

[0078] In some implementations, the avatar is operable within a virtual environment in which the avatar can interact with other entities (e.g., other avatars, virtual objects, etc.). In some implementations, the motion control policy generates signals that direct the movement of the avatar's joints or limbs. These signals are based on the avatar's skeletal structure and affect the avatar's position, rotation, and articulation within the virtual environment.

[0079] In some implementations, motion control policies are designed for generality across different avatars. A motion control policy is developed through training on a variety of skeletal structures, with the policy governing animation of a wide range of human avatars. This generalization allows the use of a single policy to animate different human avatar morphologies in a virtual environment.

[0080] In some implementations, the virtual environment includes at least one additional character with which the simulated character interacts as part of a motion control policy. In some implementations, a defined algorithm governs the interaction between the simulated character and the additional character, and the simulated character's policy adapts according to predetermined dynamics of the environment and the other characters' actions. In some implementations, the system may utilize the motion control policy to detect and respond to the presence and activities of other characters. The policy operates to manage the simulated character's navigation functions and object manipulation, as well as its interactions with other characters. In some embodiments, the simulated character is programmed to modify its responses according to predefined interaction protocols, which may include a wide range of collaborative and independent activities. For example, one predefined interaction protocol may involve self-interaction, where the simulated character is programmed to interact with itself, such as by moving its hand to scratch its nose or slapping its body. Another example of an interaction protocol may be interaction with an object, such as catching a ball. Yet another example of an interaction protocol may be interaction with another avatar, such as a simulated character dancing with another simulated character.

[0081] In some implementations, the motion control policy includes spatial and temporal information about the character's position and movement within the environment. Algorithms calculate the position data and predict potential interactions. In some implementations, this allows the simulated character to adjust its behavior in real time.

[0082] In some implementations, the nodes of the graph represent the number of interior points on the simulated character and the number of exterior points that are exterior to the simulated character. In some implementations, the interior nodes may represent specific points on the simulated character, such as, for example, joints and / or centers of gravity.

[0083] In some implementations, the graph integrates external nodes that represent specific points in the surrounding virtual environment outside of the character nodes and / or edges. In some implementations, at least a subset of the external nodes relate to locations where interactions between the simulated character and other characters or objects are expected. For example, one such location may be the space in the virtual environment near the character's head where the ball is heading, or the space where two characters are both attempting to grasp a single object in the virtual environment (e.g., near the two characters' hands). In some implementations, the external nodes may be randomly sampled, for example, nodes surrounding different body components of the character. In some implementations, the use of external nodes allows control policies to attend to specific locations of joints to the external nodes, thereby allowing the control policies to attend to different interactions.

[0084] In some implementations, nodes are selected and presented in a graph based on their location on a simulated character or within a virtual environment, and the graph is structured to include both character nodes and environment-related external nodes.

[0085] In some implementations, the edges of the graph represent the nearest neighbors of each node, which may be determined according to a K-nearest neighbor (hereinafter "KNN") algorithm or a similar algorithm. This algorithm determines edges by identifying a predefined number of nearest nodes to each node based on a spatial proximity metric. For example, in some implementations, the four nearest nodes may be used, or in various other implementations, other numbers may be used. In some implementations, the KNN algorithm processes multidimensional data by considering the location and orientation of the nodes. The KNN algorithm may calculate proximity using predetermined distance metrics, which may include standard criteria such as Euclidean distance or application-specific criteria. These metrics guide the KNN algorithm in forming a graph that reflects the spatial organization and relationships relevant to the simulation task.

[0086] In some implementations, the graph structure is dynamically updated by a KNN algorithm that recalculates node connections in response to the character's movements and interactions within the virtual environment, resulting in a graph that consistently reflects the character's current state with respect to the virtual environment, including the character's position and interaction status.

[0087] In some implementations, the graph is dynamically updated based on character interactions within the virtual environment. For example, the graph may be updated in response to interactions occurring within the virtual environment. These updates adjust the attributes and associations of nodes in the graph that correspond to body parts of the character and elements in the surrounding environment.

[0088] In some implementations, the update mechanism alters the graph by adding, removing, or modifying nodes and edges. For example, node attributes such as position, velocity, or orientation are changed as required by a character's actions or interactions with objects and other characters. In some implementations, the system updates the graph by detecting changes in the simulated environment using one or more algorithms. These algorithms track relevant data points, such as shifts in position or changes in velocity, to maintain an up-to-date representation of interactions within the graph structure.

[0089] In some implementations, the graph is dynamically updated based on the changing location of the target character or object within the virtual environment. For example, the system may utilize a graph update mechanism to track and reflect changes in the position and movement of the character or object within the virtual environment. The system adjusts graph elements, including nodes and edges, according to movement to ensure the graph remains current with respect to the state of the virtual environment. In some implementations, the dynamic updating of the graph is determined in part by the changing location of the target character or object within the simulated environment. As the graph is updated, the motion control policy continues to respond to changes in the spatial context and position of the target character or object. In some implementations, the system takes into account the real-time movement and position of the target character or object and triggers adjustments in the graph structure to address the changing interaction dynamics. In one example, if a simulated character is attempting to grasp a ball object (i.e., a target object) moving through the air in an environment according to the simulated physics of the ball in the air, the graph may be dynamically adjusted in real time as the position or trajectory of the ball changes to reflect this evolving spatial relationship.

[0090] In some implementations, the system allows simulated characters to modify their behavior in response to changes in the virtual environment. Dynamic updates of the graph underpin these modifications, allowing characters to adjust their movement strategies and align their actions with updated environmental conditions. In some implementations, the system dynamically updates the graph by triggering updates based on specified events. In one example, an avatar in a game environment may enable new movement control capabilities depending on whether zero gravity is enabled for a particular virtual room. In another example, an avatar may receive new movement control policies based on different simulated environmental conditions, such as rain, wind, or freezing temperatures. The system may operate periodically or continuously to assess and implement changes in the virtual environment and / or for the character. In some implementations, the system performs updates by recalibrating the nodes and edges of the graph as required by virtual environment movement and interaction.

[0091] In block 204, the system encodes information from the nodes and edges of the graph. Encoding information about the graph requires collecting and representing specific data attributes for each node. Nodes correspond to characteristics of simulated characters and environmental elements. In some implementations, the system captures position data for each node in the graph to define where the element is spatially located within the simulation space. In some implementations, the system encodes velocity data for each node, which captures the movement over time for all elements in the simulation.

[0092] In some implementations, orientation details, including the directional orientation of each node, are encoded to define how the element is oriented in the simulated environment. An angular velocity, specifying the rate of change of orientation associated with actions requiring rotation, is encoded for each node.

[0093] In some implementations, the system encodes relational data about connected pairs of nodes in the graph, including information about how the nodes interact with each other based on their connections, indicated by edges. These edges provide the structure of the simulated character's body and define its relationships with environmental elements.

[0094] In some implementations, the encoding includes calculating an angular velocity for each node. In some implementations, the system encodes the angular velocity for each node to quantify the rotational dynamics of the simulated character's movement and interactions with objects and other characters in the virtual environment.

[0095] In block 206, the system processes the encoded graph information through a neural network based on a graph neural network (hereinafter "GNN") architecture. In some implementations, the neural network may be a graph attention network (hereinafter "GAT") and / or may include a GAT layer to generate latent embeddings for the graph nodes. In some implementations, the operation requires an attention mechanism applied to node features including, for example, position information, velocity information, orientation information, and relational information from node pairs connected by edges. In this context, attention mechanism refers to a computational technique that selectively focuses on specific aspects of node features, enabling the neural network to assign varying importance to different elements in the graph representation.

[0096] "Relational information" refers to details about the connections or relationships between pairs of nodes in a graph. In some implementations, the system captures not only the intrinsic properties of each individual node, such as its position, velocity, or orientation, but also the inter-node relationships defined by the edges connecting them. These relationships may encompass various spatial and contextual aspects that can be used to provide elements of the system with information about how different parts of a simulated character's body or the virtual environment interact. For example, in the case of a human avatar represented by a graph, relational information may include, for example, the spatial proximity of one joint to another, the relative orientation between body parts, or the dynamic interactions between nodes representing different elements of the character's anatomy.

[0097] In some implementations, the neural network's attention mechanism is calibrated to assign weights to node features during processing, resulting in each feature being assigned significance based on the graph structure. These weights are dynamically determined based on the graph structure, allowing the neural network to assign different importance to each feature. The different weights allow the neural network to selectively address input features during the creation of latent embeddings. For example, in some implementations, the neural network's weights are automatically learned using well-established machine learning optimization algorithms, such as gradient descent. For reinforcement learning tasks, algorithms such as proximal policy optimization ("PPO") may be utilized.

[0098] After processing the encoded graph information through the GNN, a reduced-dimensional representation of the input data is obtained in the form of a latent embedding. For example, a graph that can be represented as an N-dimensional vector may be processed by the GNN into an M-dimensional vector that is a latent embedding, where M is less than N. In some implementations, the structure of this embedding corresponds to the subsequent operation of the policy network and enables further processing to output control signals for the movement of the simulated character.

[0099] In some implementations, the system uses one or more other types of graph neural networks (GNNs) instead of or in addition to GATs to encode the graph structure. The graph structure corresponds to the movement policy of a simulated character in a virtual environment. The GNN utilizes node connectivity that correlates elements of the character's body and environmental objects. This connectivity enables processing of spatial features, such as position, orientation, and velocity, through the network.

[0100] In some implementations, GNNs can be used with a variety of graph structures, varying in size and complexity. This configuration allows for the representation of characters with different morphologies and the simulation of a variety of different interaction scenarios. The system utilizes GNNs to process and encode relational data that reflects the dynamic interactions between characters and elements in the environment. In some implementations, the GNN design is structured to support generalization across a variety of interaction scenarios. GNN layers manage the necessary computational load associated with character-environment interactions. In some implementations, this includes performing more complex learning tasks necessitated by high-dimensional policy training operations.

[0101] In block 208, the system determines a fixed-length latent vector that represents a modified structure compared to the structure of one or more latent embeddings. For example, the system may modify the latent embeddings to obtain a fixed-length latent vector. For example, in the example of FIG. 2, the system discards a portion of the latent embeddings to obtain a fixed-length latent vector. The discarding may require the removal of data corresponding to some nodes in the graph representation.

[0102] In some implementations, during discarding operations, a predetermined algorithm or set of rules identifies and removes node embeddings that are deemed non-essential to preserve the functionality of the motion control policy. The criteria for discarding node embeddings may be predefined and may be based on various characteristics, such as the relative influence or spatial relationship of the nodes within the graph structure. "Relative influence" refers to the importance or influence of a node within the overall graph, and nodes that exhibit greater influence are more likely to be retained. "Spatial relationship" here refers to the location and connections between nodes, and the discarding algorithm may prioritize nodes that contribute important information based on their spatial proximity or specific relationship within the graph.

[0103] The output of this block is a latent vector with fixed dimension that contains condensed information from the original graph. This vector is formed by keeping only the embeddings necessary to generate the core motor control signals and removing excess data that is not necessary for the core motor control function. By discarding a certain amount of unnecessary or redundant information in the latent space, the result can be a fixed-size latent vector that can capture the desired motor properties. This discarding allows the neural network training to be decoupled from the structure of the graph, allowing the policy to generalize to different character morphologies and different interactions of the character with its own nodes and / or with nodes of other objects in the virtual environment.

[0104] In some implementations, discarding a portion of the potential embeddings requires selectively compressing information specific to one or more of the nodes of the graph. This may include identifying and removing nodes that represent redundant or non-essential information regarding the graph-based representation of the control policy. In some implementations, the system utilizes selective compression to reduce the dimensionality of the interaction graph by removing certain nodes and their associated attributes, which may include, for example, position data, orientation data, or velocity data.

[0105] In some implementations, data in the embedded space that represents external nodes (e.g., outside the character) is discarded, and the internal character node data is kept in the latent embedding. This allows the character's internal nodes to contain information based on the (prior) existence of external nodes. Thus, the graph attention layer can "pinch" additional information about the external nodes to their neighboring internal nodes, so that the policy knows which nodes it should attend and that interactions are likely to occur at those nodes.

[0106] In some implementations, the system utilizes heuristics to determine which nodes to discard during the selective compaction process. These heuristics are applied to evaluate the relevance of each node to the desired outcome of the interaction, and nodes with minimal impact on the outcome are identified as candidates for removal. In some examples, nodes that are furthest or beyond a certain distance from the interaction may be deemed less relevant and may be discarded. In some implementations, random nodes in the graph may be discarded, allowing the neural network to learn and adapt based on these random discards and new graph structures, as described below. In some implementations, the system discards nodes randomly, so that the neural network is trained based on a large diversity of graph structures. This random discarding of nodes in the graph may be implemented to make the learned policy robust to different graph structures.

[0107] In block 210, the system sends the fixed-length latent vector as input to a feed-forward neural network (hereinafter "FNN") to generate control signals for the simulated character. The FNN structure includes an input layer, a hidden layer, and an output layer consisting of nodes. Each node is associated with a set of weights and biases that are applied to the input data as part of the transformation process. One example of an FNN that may be used is a multi-layer perceptron ("MLP"). In various implementations, other neural network architectures may be utilized, such as a convolutional neural network applied to a graph (i.e., a graph convolutional network).

[0108] In some implementations, during operation, the latent vectors are processed successively through hidden layers. At each layer, the vectors undergo a weighted sum followed by an activation function to introduce nonlinearity. Typical activation functions used include sigmoid, hyperbolic tangent, or rectified linear unit (ReLU). The outputs of the hidden layers are fed forward to the output layer to produce control signals. These control signals correspond to specific actions or movements required by the joints of the simulated character or actuators in the virtual environment.

[0109] The design of the FNN, in terms of the number and configuration of layers and nodes, the choice of activation functions, and the learning rate, allows for mapping higher-dimensional data represented by latent vectors into actionable control signals appropriate for various simulated character morphologies and interaction scenarios. In some implementations, the network's parameters are determined during a training phase and fixed prior to operation. The output of this block includes the joint angles or torques required to direct the character's movement according to the intended interaction objective.

[0110] In some implementations, an FNN includes a sequence of fully connected layers that are used to map latent embeddings to control signals for a simulated character in a virtual environment. Each layer in the FNN transforms input features in a hierarchical manner to produce an output vector that represents the control signal.

[0111] In some implementations, an FNN receives encoded information in the form of fixed-length latent vectors from a previous layer, potentially a graph-based processing layer such as a GAT layer. The fully connected layers of the FNN process these latent vectors and adjust the network weights and biases to generate outputs.

[0112] In some implementations, the adaptability of the FNN is provided by the ability to modify the number and density of neurons in the fully connected layer. This adjustability caters to various complexities and differences in character morphology in interaction scenarios. In some implementations, control signals output by the FNN are applied to the simulated character to enable movement and interaction within the virtual environment. These signals generally correspond to, for example, the joint angle, velocity, and torque specifications required to animate the character.

[0113] In some implementations, the control signals determine one or both of the joint angles and joint torques in the simulated character. This information can be used to manipulate the joint movements of the simulated character. Specifically, in some implementations, joint angles and torques are required to articulate each joint. In some implementations, this enables movement within the virtual environment that conforms to specified motion parameters. These signals direct the character's physical actions, including object manipulation and character-to-character interactions, as determined by inputs from the virtual environment and objectives encoded in the control policy. In some implementations, the FNN architecture outputs a vector of control signals for each joint in the simulated character's body. This architecture allows for the creation of specific articulation commands that are processed and applied to the character model.

[0114] In block 212, the system applies control signals to the body of the simulated character. Applying control signals to the body of the simulated character in the virtual environment consists of translating the output from the FNN into physical commands that control the movement of the character. These control signals determine joint angles and joint positions that correspond to the movement of body segments within the constraints of the physical simulation model of the character.

[0115] In some implementations, each control signal corresponds to a movement parameter, such as a joint torque or joint position setpoint. The virtual environment processes these parameters and applies them to a character model, which may consist of a varying number of joined segments or joints. As the simulation progresses, these movements are executed according to the physics programmed into the simulation, including collision detection and response.

[0116] In some implementations, the virtual environment collects data regarding the execution of these control signals, including spatial and temporal precision relative to the desired outcome. In some implementations, the virtual environment provides real-time feedback based on the dynamics of the character's movements and interactions with other entities and objects in the simulation space.

[0117] In block 214, the system evaluates the simulated character's actions via a reward function. The reward function quantitatively evaluates the movement and interaction quality of actions performed by the simulated character. The reward function operates by comparing the simulated actions to one or more reference models. In this context, the reference models serve as benchmarks or standards against which the simulated character's actions are assessed for movement and interaction quality, which are determined based on predefined standards or models that encapsulate desired characteristics or behaviors. During the evaluation process, the reward function systematically compares the simulated character's performed actions to the attributes specified by the reference models. This comparative analysis forms the basis for quantifying the simulated character's movement and interaction quality and provides a measurable basis for reinforcement learning adaptation. For example, movement quality can be assessed by testing the simulated character's movement imitation against the reference movements, e.g., by examining certain metrics, such as joint angles, joint velocities, accuracy of end-effector placement, and the state of the character's base (e.g., base reference points in the simulated character's skeletal structure, such as the pelvis or base of the spine). Comparison against these metrics allows the character's movements to be consistent with the target reference motion specified in the reference model.

[0118] In some implementations, interaction quality is measured by checking the character's simulated interactions with one or more objects against reference interactions between objects specified in a reference model. This aspect of the reward function focuses on the differences between edges in the character's interaction graph that connect the character's node to one or more nodes of other objects, such as nodes belonging to entities in the environment or one or more nodes belonging to other characters.

[0119] In some implementations, the reward function includes one or more of joint angle alignment, joint velocity alignment, end-effector position alignment, and root state alignment. The joint angle alignment term may be used to measure the joint angles of the simulated character relative to a reference model. The joint velocity alignment term may be used to measure the velocities of the joints of the simulated character relative to a predetermined velocity profile. The end-effector position alignment term may be used to assess the accuracy of the position of the end-effector, which is the point on the character that is to interact with the environment. The root state alignment term may be used to evaluate the position and orientation of a core part of the simulated character's body, which in some character forms may be, for example, the pelvis or torso.

[0120] In some implementations, the reward function is the product of two terms: a joint matching term and an interaction matching term. In some implementations, the joint matching term can be a different type of matching term. In some implementations, the joint matching term can function as a "motion-matching term," or an imitation reward, meaning that it penalizes deviations from a reference motion. In some implementations, the reward term can be explicitly defined and implicitly learned, for example, by training a motion discriminator.

[0121] In some implementations, the joint consistency term quantifies the consistency of the simulated character's joints relative to a reference motion, which may include, for example, joint angles and velocities, end-effector positions, and the character's base state. In some implementations, the interaction consistency term in the reward function evaluates the character's interactions within the virtual environment. This term calculates the difference between node-pair information along the edges of an interaction graph, which connect nodes external to the character with other elements in the environment. In some implementations, the interaction consistency term rewards minimizing the distance between character internal nodes and environment external nodes, e.g., rewarding movement of internal nodes toward external nodes. In some implementations, the reward function utilizes both the joint consistency term and the interaction consistency term to assess motor imitation and environmental interaction. The control policy triggered by this combined reward function adapts to produce outputs that reveal both accurate motion replication and interaction dynamics.

[0122] An example of an overall reward function according to one or more implementations follows: For internal motion quality, the following reward function expressions are respectively related to joint state alignment (r joint ), end effector position alignment (r ee ), and basal joint alignment (r root ) regarding.

[0123]

number

[0124] r ee =exp(k3Σ||x sim -x ref ||)

[0125] r root =exp(k4[||p sim -p ref ||+||q sim -q ref ||])

[0126] For external interaction quality, the following reward function formula relates the external edge similarity between the simulation and the reference:

[0127] r edge =exp(k5[Σw||∈ sim -∈ ref ||])

[0128] In these formulas, q is the joint orientation,

[0129]

number

[0130] where x is the angular velocity, x is the center of mass position, p is the position, ∈ is the position attribute of the edge (e.g., based on the difference in position between connected nodes), w is the weight of the term, k1 through k5 are the sensitivities of the terms, "sim" refers to the simulation graph, and "ref" refers to the reference graph. In some implementations, the interaction reward function may be determined by using the weighted edges to measure the distance between the positions and velocities of the reference and simulated graphs and determining the interaction reward function based on the measured distance, for example, to reward minimizing the distance between internal and external nodes and / or to reward movement of internal nodes towards external nodes.

[0131] In block 216, the system adapts the control policy through reinforcement learning by iteratively adjusting the graph based on an evaluation process via a reward function. The adaptation generalizes to one or both of different character morphologies and different interaction scenarios within the virtual environment. The adaptation phase involves improving the control policy through reinforcement learning, which provides adaptability and generalization for the motor control system. Through the learning process via the reward function, the control policy undergoes iterative adjustment, refinement, and / or optimization based on an evaluation of simulated character actions.

[0132] To adapt the control policy, the system employs reinforcement learning, a framework in which a system learns through trial and error and continuously adjusts its behavior to minimize a defined reward. This iterative optimization aims to account for different character morphologies and diverse interaction scenarios within the virtual environment. The system actively learns from simulated character interactions and refines control signals based on a specified reward function, ensuring that the adapted control policy is not limited to a specific character type or interaction but instead extends its applicability to a broader spectrum.

[0133] In some implementations, the process of adapting a control policy to different interaction scenarios involves leveraging reinforcement learning to adapt the control policy across a spectrum of contextual variations within a broader interaction category. For example, a simulated character may be tasked with an interaction scenario involving lifting an object from a surface. This general interaction type may be manifested in a variety of ways, depending on the specific object involved. The reinforcement learning-driven control policy undergoes adaptation to account for the situational nuances associated with different objects. In this way, the control policy is trained to dynamically adjust the character's same lifting motion to address scenarios featuring distinct objects, such as a cup, ball, trophy, pen, or phone. Each object may introduce unique spatial requirements and characteristics that affect interaction execution. Through the adaptation process driven by reinforcement learning, the control policy gains the ability to generalize its approach across a myriad of interaction scenarios within a specified interaction type.

[0134] In some implementations, the system utilizes reinforcement learning to adapt a control policy to various different character morphologies, allowing the control policy to generalize across different character anatomies. For example, consider a control policy designed for a human avatar with a particular skeletal structure. Through reinforcement learning, this policy can be adapted to address diverse character morphologies by learning motion representations that apply broadly across specific skeletal configurations. The reinforcement learning mechanism enables the control policy to identify motion patterns, features, and characteristics that are applicable across a wide range of morphological variations and then apply them to specific, different morphologies. For example, a control policy can be trained to adapt its motion representations to different human avatars with various limb lengths, joint placements, or body proportions. The adaptation process ensures that a single control policy can be adapted across different avatars with distinct morphologies, while taking into account and applying information from the control policy regarding the broader interactions to be achieved.

[0135] In some implementations, the system utilizes random sampling techniques to insert additional nodes during the training phase. In some implementations, the training phase utilizes variable node placement for different regions of the character to develop control policies that are generalizable across a wide range of virtual environment interactions and character morphologies. In some implementations, the additional nodes are randomly placed on the character and / or in the virtual environment during training. This random placement alters the character-environment spatial relationships in the graph representation. Adding such nodes allows the control policies to dynamically respond to changes in the graph structure.

[0136] In some implementations, the introduction of randomly sampled nodes diversifies the set of data available for reinforcement learning. This randomization affects the graph representation at each training iteration, which influences how neural networks process graph structure, with a significant emphasis on motor control policies. The system can develop control policies applicable to different graph configurations without being limited to a specific training configuration.

[0137] In some implementations, the system adapts control policies for additional simulated characters by utilizing transfer learning. Transfer learning techniques allow the system to utilize an initial model trained for a particular character or task as a starting point for training other characters. In some implementations, the system keeps some network layers and parameters from the original model, while other parameters are adjusted to reflect the structure and interaction attributes of the new character. Thus, policies can then be specific to different characters.

[0138] In some implementations, during transfer learning, the latent embeddings in the simulated character model undergo modification to cater to the requirements of different character morphologies or interaction types. This adjustment involves the system re-encoding the latent embeddings based on new character specifications, such as joint configurations or body part proportions.

[0139] In some implementations, the system applies transfer learning to control policies to adapt to different forms and interactions without requiring complete retraining from scratch: the system maintains the motor and interaction characteristics encoded in the policies while making the necessary adjustments to accommodate new simulated characters.

[0140] In some implementations, the system synchronizes actions by simulated characters in a virtual environment. The characters perform collaborative tasks according to a control policy that regulates the actions based on timing and spatial data associated with each character's role in the task. The actions may include coordinated speeds, matching movements, or sequential activities. For example, for coordinated complex movements, such as a complex dance, the movements may be decomposed into smaller steps performed in a specific sequence between the interacting characters.

[0141] In some implementations, reward functions in a reinforcement learning framework are constructed to address synchronization. These functions measure timing and movement accuracy across characters. A policy refines its output by optimizing against these reward functions, which are designed to evaluate the alignment of character actions with predefined movement sequences. In some implementations, a prediction mechanism in the control policy aligns character interactions with the impending task state. The policy adjusts to temporal shifts in the task environment, allowing characters to maintain harmony across various scenarios without reference to underlying intentions or interests.

[0142] In some implementations, the FNN output is utilized to generate a database of movement patterns for one or both of a set of different character morphologies and a set of different interaction scenarios. In some implementations, utilizing the FNN output to create the database requires capturing and storing movement patterns associated with different character morphologies. This database represents a repository of predefined movement sequences tailored to a particular character type, e.g., a character morphology such as a particular skeletal type. In some implementations, the database includes several interaction scenarios. The FNN output contributes to the compilation of movement patterns corresponding to various different virtual interactions.

[0143] FIG. 3 illustrates how control signals for simulated character movement are generated and applied from control policies according to some implementations.

[0144] The interaction graph 302 shown on the left side of Figure 3 is a virtual representation of a control policy for the movement of a simulated character. This graph is trained for the specific motion of the simulated character. The graph 302 encapsulates the dynamics of the simulated character's movement within a virtual environment. The graph is made up of a set of nodes and edges. The graph models the characteristics of the simulated character's body and the elements present in the virtual environment. The nodes correspond to descriptive points, including internal aspects of the character and external entities in the environment, that form a comprehensive network encapsulating the character's interactions. The edges represent connections between these nodes, representing the relationships and dependencies required to define the simulated character's behavior.

[0145] GAT layer 304 (represented as an arrow labeled "GAT layer") represents a system that utilizes a graph attention network layer (and / or other GNN) to process interaction graph 302. GAT layer 304 receives interaction graph 302 and provides a latent embedding that has fewer dimensions than graph 302. GAT layer 304 is responsible for processing the information encoded in the interaction graph. These layers implement a specialized neural network architecture designed for graph-based data, allowing the system to identify and prioritize the importance of different nodes in the graph. GAT layer contributes to generating the latent embedding by weighting the contributions of neighboring nodes.

[0146] The graph structure 306 represents the latent embeddings generated through application of the GAT layer 304. The upper portion 307 of the graph is depicted as "Latent Embeddings," which represent encoded information encapsulating essential properties of the simulated character's body and the virtual environment. The lower portion 309 is labeled "Discard," which represents the removal of a subset of information from the graph nodes through a discarding process. This selective retention and discarding of information is performed by the system to generate a fixed-length latent vector.

[0147] Following the generation of fixed-length latent vectors through the latent embedding and discarding phases, the information is then fed to the FNN 308 for further processing. The FNN, representing a sequence of fully connected layers, is responsible for converting the encoded latent information into actionable control signals for the simulated character. This conversion is a result of the FNN's ability to learn complex patterns and relationships within the encoded data. The action phase 310 (arrow labeled "Action") depicts the output of the FNN in the form of control signals that direct the movement and interaction of the simulated character within the virtual environment. The action phase 310 involves the system applying these control signals to the simulated character's body within the virtual environment.

[0148] 4A is a diagram illustrating an example of a simulated character in a virtual environment, according to some implementations. The simulated character is shown as a polygonal model depicting a human avatar 402 with a basic form encompassing distinguishable features, such as a head, arms, legs, and feet, and the joints between these features. Internal nodes 404 within the character's body are visually represented and correspond to specific points on the avatar, describing the avatar's movement and interactions, and may represent features (such as the head or hands) and / or joints between features (e.g., elbows, shoulders, knees, ankles, etc.). Internal links 405 within the character's body are also visually represented and may connect two nodes 404 and correspond to features of the avatar, such as limbs (e.g., upper arm, lower arm, upper thigh, etc.).

[0149] Edges 406 in the figure indicate relational connections between nodes 404, illustrating the internal structure of the character. These edges 406 may be determined by a K-nearest neighbor (KNN) algorithm to emphasize close-proximity relationships between nodes within the character's body. The selection of the KNN algorithm helps capture the spatial proximity of nodes, which is a factor in representing realistic and contextually relevant interactions within the character's body. The techniques described herein may use connections between nearest-neighbor nodes, shown in this example by edges 406, to preserve spatial relationships for generating coherent and plausible movement interactions by the character.

[0150] The illustrated external node 408 is an example of one or more nodes external to the character's body that may be added to the graph to support external interaction with elements in the surrounding environment as described herein. In this example, the external node 408 may be a ball or other object in the environment external to the character's body, or may be a particular spatial area that has been determined to potentially provide interaction for the character, such as a space within which another object can move. Additional external nodes may be located in different areas around the character.

[0151] FIG. 4B illustrates an example of reinforcement learning for multi-character interaction, according to some implementations. Specifically, the diagram illustrates the results of reinforcement learning for multi-character interaction in a virtual environment. The virtual representation displays two human avatars, each designed with a polygonal model, facing each other in the simulated environment. In particular, a first one of the avatars (the avatar closer to the right edge of the diagram) extends or reaches its hand back to touch the hand of a second avatar (the avatar closer to the left edge of the diagram), while the second avatar extends its hand slightly forward to touch the hand of the first avatar. This represents a specific interaction scenario achieved through the adaptation of the avatars' respective control policies. In some examples, an external node is placed between the avatars' hands as a potential interaction node.

[0152] The control policies for both avatars are iteratively optimized based on the received reward signals, as described above. Through this process, the avatars learn to coordinate their actions in a synchronized manner, resulting in the desired interaction of holding hands. The adaptability of the control policies is emphasized, demonstrating their ability to generalize to different character morphologies and different interaction scenarios within the virtual environment.

[0153] Various example implementations of one or more features described herein may also or alternatively include one or more of the following.

[0154] Control policies implemented by a machine learning model (e.g., a machine learning controller) via a neural network govern interactions between the character and one or more other characters or other objects. In some implementations, the character may be represented by a graph, such as a graph data structure.

[0155] A control policy implemented by a machine learning model (e.g., a machine learning controller) via a neural network is adapted to manage interactions between the character and one or more other characters or other objects.

[0156] In some implementations, a higher-dimensional graph representing a character is compressed into a simpler representation using fewer dimensions in the embedding space that are specific to the behavior being created for the control policy. Because the forms can all use the same (lower-dimensional) latent space, multiple different character forms can be controlled by a control policy to perform its behavior.

[0157] In some implementations, the method may include obtaining a controller to control the movement of a reference graph representation of a character, the controller having been trained based on a base control policy to cause the character to follow a reference motion trajectory (e.g., a predetermined reference motion trajectory). The reference graph representation of the character is processed by a neural network into a latent space having different dimensions from the reference graph representation to reshape the reference graph representation into a modified graph representation having one or more structures different from the reference graph representation to adapt the base policy to the modified policy. In some implementations, this allows a network trained on a first graph with N nodes to support runtime estimation for a second graph with any number of nodes; for example, the second graph can represent a character having a different morphology with a different skeletal structure and / or fewer or more nodes than the original graph / character. In some implementations, the one or more different structures of the modified graph representation may include discarding and / or adding nodes.

[0158] In some implementations, a method for providing a generalized control policy for a character's interactions with other objects may include steps of obtaining data indicating a reference movement of a reference avatar and / or obtaining (or constructing) and storing a reference graph for the reference movement, where the graph captures the spatial relationships of the character and external objects and the reference graph is a set of spatial descriptors that encode relative information between vertices in the reference graph; initializing a simulation graph for the simulated movement; assigning a weight to each edge of the reference graph and the simulated graph; measuring the distance between the positions and velocities of the reference graph and the simulated graph using the weighted edges; determining an interaction reward function based on the measured distance, for example, to reward minimizing the distance between internal nodes and external nodes or to reward movement of internal nodes toward external nodes; and optimizing a combination of the interaction reward function and a movement reward function that rewards the simulated graph for mimicking the movement of the reference graph.

[0159] 5 is a block diagram of an exemplary computing device 500 that may be used to implement one or more features described herein. In one example, device 500 implements a computer device (e.g., 102 and / or 110 of FIG. 1 ) and may be used to perform suitable method implementations described herein. Computing device 500 may be any suitable computer system, server, or other electronic or hardware device. For example, computing device 500 may be a mainframe computer, desktop computer, workstation, portable computer, or electronic device (portable device, mobile device, cell phone, smartphone, tablet computer, television, TV set-top box, personal digital assistant (PDA), media player, gaming device, wearable device, etc.). In some implementations, device 500 includes a processor 502, memory 504, input / output (I / O) interface 506, and audio / video input / output device(s) 514.

[0160] Processor 502 may be one or more processors and / or processing circuits for executing program code and controlling the basic operations of device 500. A "processor" includes any suitable hardware and / or software system, mechanism, or component for processing data, signals, or other information. A processor may include a general-purpose central processing unit (CPU), multiple processing units, a system with dedicated circuitry for achieving functions, or other systems. Processing need not be limited to a particular geographic location or have time limitations. For example, a processor may perform its functions in "real time," "offline," "batch model," "batch mode," etc. Portions of processing may be performed by different (or the same) processing systems, at different times, and in different locations. A computer may be any processor in communication with memory.

[0161] Memory 504 is generally provided within device 500 for access by processor 502 and may be any suitable processor-readable storage medium, e.g., random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc., suitable for storing instructions for execution by the processor and located separately from and / or integrated with processor 502. Various engines, modules, instructions, machine learning models, software code, and other blocks used in the described features may be stored in memory 504 and / or other connected storage devices (e.g., database 512). Memory 504 may store software run on server device 500 by processor 502, including operating system 508, one or more applications 510, and database 512, which can store data used by components of device 500. In some implementations, application 510 may include a software application, engine, machine learning model, etc. that includes instructions that enable processor 502 to perform (or control the functioning of) the functions described herein, e.g., some or all of the methods described with respect to FIG. 2. For example, application 510 may include a control policy module that implements one or more machine learning models used in the techniques described herein. Application 510 may include a GNN, such as GAT layer 304 and FNN 308 (e.g., MLP) of FIG. 3. Database 512 (and / or other connected storage) may store various data used in the described techniques, including graph representations, latent embeddings, latent vectors, training data, etc.

[0162] The elements of software in memory 504 may alternatively be stored on any other suitable storage location or computer-readable medium. Additionally, memory 504 (and / or other connected storage devices) may store instructions and data used in the features described herein. Memory 504 and any other type of storage (such as magnetic disk, optical disk, magnetic tape, or other tangible medium) may be considered "storage" or "storage device."

[0163] The I / O interface 506 may provide functionality to enable interaction between the server device 500 and other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data store 120), and input / output devices may communicate through the interface 506. In some implementations, the I / O interface may connect to interface devices, including input devices (keyboards, pointing devices, touchscreens, microphones, cameras, scanners, etc.) and / or output devices (display devices, speaker devices, printers, monitors, etc.).

[0164] The audio / video input / output device(s) 514 may be a variety of devices, including user input devices (e.g., a mouse, etc.) that may be used to receive user input, audio output devices (e.g., speakers), and display devices (e.g., a screen, monitor, etc.), and / or combined input and display devices that may be used to provide graphical and / or visual output.

[0165] For ease of illustration, FIG. 5 shows one block for each of the processor 502, memory 504, I / O interface 506, and software blocks of the operating system 508 and virtual experience application 510. These blocks may represent one or more processors or processing circuits, operating systems, memories, I / O interfaces, applications, and / or software engines. In other implementations, the device 500 may not have all of the components shown and / or may have other elements, including other types of elements, instead of or in addition to the elements shown herein. Although the online virtual experience server 102 is described as performing the operations described in some implementations herein, any suitable component or combination of components of the online virtual experience server 102, client device 110, or similar system, or any suitable one or more processors associated with such systems, may perform the described operations.

[0166] Device 500 may be a server device or a client device. An exemplary client or user device may be a computer device that includes several components similar to device 500, e.g., a processor 502, a memory 504, and an I / O interface 506. An operating system, software, and applications suitable for the client device may be provided in the memory and used by the processor. The I / O interface for the client device may be connected to a network communication device, as well as input and output devices, such as a microphone for capturing sound, a camera for capturing images or video, a mouse for capturing user input, a gesture device for recognizing user gestures, a touchscreen for detecting user input, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices. A display device within audio / video input / output device(s) 514 may be connected to (or included within) device 500, for example, to display the image pre- and post-processing described herein, and such a display device may include any suitable display device, e.g., an LCD, LED, or plasma display screen, a CRT, a television, a monitor, a touchscreen, a 3D display screen, a projector, or other visual display device. Some implementations may provide an audio output device, e.g., speech output of spoken text or speech synthesis.

[0167] One or more methods described herein (e.g., method 600) may be implemented by computer program instructions or code that can be executed on a computer. For example, the code may be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and stored on a computer program product that includes a non-transitory computer-readable medium (e.g., storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), flash memory, rigid magnetic disks, optical disks, solid-state memory drives, etc. The program instructions may be contained within or provided as an electronic signal, for example, in the form of a software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods may be implemented in hardware (e.g., logic gates) or a combination of hardware and software. Exemplary hardware may be a programmable processor (e.g., a field programmable gate array (FPGA), complex programmable logic device), general-purpose processor, graphics processor, application-specific integrated circuit (ASIC), etc. One or more methods may be implemented as part of or a component of an application running on the system, or as an application or software running in conjunction with other applications and the operating system.

[0168] One or more methods described herein may be implemented within a standalone program that may run on any type of computing device, a program running on a web browser, or a mobile application (“app”) running on a mobile computing device (e.g., a cell phone, a smartphone, a tablet computer, a wearable device (such as a watch, an armband, jewelry, headwear, goggles, glasses, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives final output data for output (e.g., for display) from the server. In another example, all computations may be performed within a mobile app (and / or other apps) on the mobile computing device. In another example, computations may be divided between the mobile computing device and one or more server devices.

[0169] Although described with respect to specific implementations in this description, these specific implementations are merely examples and are not limiting, and concepts illustrated in the examples may be applied to other examples and implementations.

[0170] The functional blocks, operations, features, methods, devices, and systems described in this disclosure may be integrated or divided into different combinations of systems, devices, and functional blocks as will be known to those skilled in the art. Any suitable programming language and programming techniques may be used to implement the routines of a particular implementation. Different programming techniques, e.g., procedural or object-oriented, may be used. The routines may run on a single processing device or multiple processors. Although steps, blocks, operations, or computations may be presented in a particular order, the order may be changed in different particular implementations. In some implementations, multiple steps or operations shown as sequential herein may be performed simultaneously. [Explanation of symbols]

[0171] 100 System Architecture 102 Online virtual experience server, virtual experience server, online game server 104 Virtual Experience Engine 106 Virtual Experience 108 Graphics Engine 110 client devices 110a Client Device 110b client device 110n client devices 112 Virtual Experience Application 114 Input / Output (I / O) Interface, Output Device 120 data stores 122 Network 130 Developer Devices 130a Developer Device 130b Developer Devices 130n Developer Devices 132 Virtual Experience Applications 134 Input / Output (I / O) Interface 200 ways 302 Interaction graph, graph 304 GAT layer 306 Graph Structure 307 Upper 308 FNN 309 Lower 310 Action Phase 402 Avatar 404 Internal Node, Node 405 Internal Links 406 Edge 408 External Node 500 computing devices, server devices, devices 502 processor 504 memory 506 Input / Output (I / O) Interface, Interface 508 Operating Systems 510 Application, Virtual Experience Application 512 databases 514 Audio / Video Input / Output Device

Claims

1. 1. A computer-implemented method comprising: accessing a graph representing a motion control policy for a simulated character in a virtual environment, the graph including a plurality of nodes, the nodes including features of the body of the simulated character and the virtual environment; encoding information about the graph, including one or more of a position, a velocity, or an orientation of each node, the information including relational information of pairs of nodes connected by edges; processing the encoded information through a graph neural network to generate one or more potential embeddings for the nodes of the graph; determining a fixed-length latent vector from the one or more latent embeddings; sending the fixed-length latent vector as an input to a feed-forward neural network to output a control signal for the simulated character; applying the control signals to the body of the simulated character within the virtual environment to provide an action of the simulated character; evaluating the actions of the simulated character via a reward function to assess a motor quality of the actions and an interaction quality of the actions; adapting the control policy through reinforcement learning by iteratively adjusting the graph based on the evaluation via the reward function, wherein the adaptation generalizes to one or both of different character morphologies and different interactions within the virtual environment; and A method comprising:

2. 2. The method of claim 1, wherein determining the fixed-length latent vector comprises discarding a portion of the one or more potential embeddings to remove information about a subset of the nodes.

3. 10. The method of claim 1, wherein the graph neural network includes one or more graph attention network (GAT) layers.

4. The method of claim 1 , wherein the feedforward neural network comprises a multi-layer perceptron (MLP) including a sequence of fully connected layers.

5. The method of claim 1 , wherein the virtual environment includes at least one additional character with which the simulated character interacts as part of the motion control policy.

6. The method of claim 1 , wherein the nodes of the graph represent a plurality of interior points on the simulated character and a plurality of exterior points exterior to the simulated character.

7. The method of claim 1 , wherein the edges of the graph represent nearest neighbors of each node as determined by a K-nearest neighbor (KNN) algorithm.

8. The method of claim 1 , wherein the encoding step further comprises calculating an angular velocity for each node.

9. The method of claim 1 , wherein the graph is dynamically updated based on interactions of the characters within the simulated environment.

10. The method of claim 9 , wherein the graph is dynamically updated based on the changing location of a target character or object within the simulated environment.

11. The method of claim 1 , wherein the step of discarding the portion of the potential embedding comprises selectively compressing information specific to one or more of the nodes in the graph.

12. 1. A system comprising: one or more processors; A memory for storing instructions wherein the instructions, when executed by the one or more processors, cause the system to: accessing a graph representing a motion control policy for a simulated character in a virtual environment, the graph including a plurality of nodes, the nodes including features of the body of the simulated character and the virtual environment; encoding information about the graph, including one or more of a position, a velocity, or an orientation of each node, the information including relational information of pairs of nodes connected by edges; processing the encoded information through a graph neural network to generate one or more latent embeddings for the nodes of the graph; discarding a portion of the one or more potential embeddings to remove information about a subset of the nodes to obtain a fixed-length latent vector; sending the fixed-length latent vector as an input to a feed-forward neural network to output a control signal for the simulated character; applying the control signals to the body of the simulated character within the virtual environment to provide an action of the simulated character; evaluating the actions of the simulated character via a reward function to assess a motor quality of the actions and an interaction quality of the actions; adapting the control policy through reinforcement learning by iteratively adjusting the graph based on the evaluation via the reward function, wherein the adaptation generalizes to one or both of different character morphologies and different interactions within the virtual environment; and A system that performs an operation including:

13. The system of claim 12 , wherein the reward function comprises one or more of joint angle consistency, joint velocity consistency, end-effector position consistency, or root state consistency.

14. The system of claim 12 , wherein the reward function comprises a product of a joint matching term and an interaction matching term.

15. The system of claim 12 , wherein the control signals determine one or both of angles or torques of joints in the simulated character.

16. The instructions may include: random sampling to introduce additional nodes during the training phase; The system of claim 12 , further comprising:

17. the graph neural network includes one or more graph attention network (GAT) layers; 13. The system of claim 12, wherein the feedforward neural network comprises a multi-layer perceptron (MLP) comprising a sequence of fully connected layers.

18. 13. The system of claim 12, wherein the act of adapting the control policy includes transfer learning for additional simulated characters, the simulated characters performing tasks requiring synchronized actions.

19. 13. The system of claim 12, wherein the output of the feedforward neural network is utilized to generate a database of movement patterns for one or both of a plurality of different character morphologies and a plurality of different interaction scenarios.

20. 1. A non-transitory computer-readable medium containing instructions, the instructions comprising: accessing a graph representing a motion control policy for a simulated character in a virtual environment, the graph including a plurality of nodes, the nodes including features of the body of the simulated character and the virtual environment; encoding information about the graph, including one or more of a position, a velocity, or an orientation of each node, the information including relational information of pairs of nodes connected by edges; processing the encoded information through a graph neural network to generate one or more latent embeddings for the nodes of the graph; modifying a portion of the latent embedding to obtain a fixed-length latent vector; sending the fixed-length latent vector as an input to a feed-forward neural network to output a control signal for the simulated character; applying the control signals to the body of the simulated character within the virtual environment to provide an action of the simulated character; evaluating the actions of the simulated character via a reward function to assess a motor quality of the actions and an interaction quality of the actions; adapting the control policy through reinforcement learning by iteratively adjusting the graph based on the evaluation via the reward function, wherein the adaptation generalizes to one or both of different character morphologies and different interactions within the virtual environment; and 1. A non-transitory computer-readable medium comprising:

Images