Assigning behavior models to autonomous agents based on resources

Abstract

The disclosed concepts relate to employing agent behavior models to control agent behavior in applications, such as video games or simulations. For example, in some implementations, agent behavior models with relatively greater resource utilization, such as generative language models, are assigned to agents at higher levels of an agent hierarchy. Agent behavior models with relatively less resource utilization, such as reinforcement learning or hard-coded models, are assigned to agents at lower levels of the agent hierarchy.

Description

Background Technology

[0001] Many computer applications use autonomous agents that can make decisions and take actions within a computing environment. For example, video games allow players to interact with non-player characters controlled by the video game, and simulations can allow agents such as self-driving cars to interact with each other. Autonomous agents can be implemented using hard-coded behavioral models, which are developed using conventional software development techniques. Furthermore, some efforts have been made to implement autonomous agents using machine learning behavioral models. Summary of the Invention

[0002] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed embodiments. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0003] This specification generally relates to assigning behavioral models to agents based on resources. One example includes a computer-implemented method that includes accessing a hierarchy of agents within an application environment provided by an application, wherein agents at different levels interact within the application environment. The method may further include assigning appropriate agent behavioral models to individual agents based at least on the appropriate level of each agent in the hierarchy. The method may further include configuring the appropriate agent behavioral models based at least on one or more configuration parameters. The method may further include coordinating communication between the appropriate agent behavioral models during application execution. The method may further include controlling the application based at least on the appropriate agent behavioral models.

[0004] Another example could include a system comprising a hardware processing unit and storage resources for storing computer-readable instructions. When executed by the processing unit, the computer-readable instructions could enable the system to coordinate communication between corresponding agent behavior models of agents at a hierarchy, wherein the agents at the hierarchy interact within an application environment provided by the application, and the corresponding agent behavior models are assigned to agents based at least on the level of each agent in the hierarchy and the resource utilization characteristics of the agent behavior models. When executed by the processing unit, the computer-readable instructions could also enable the system to control the application, at least based on the corresponding agent behavior models.

[0005] Another example may include a computer-readable storage medium storing computer-readable instructions that, when executed by a processing unit, cause the processing unit to perform actions. Actions may include a hierarchy of agents accessing an application environment provided by the application, wherein the agents in the hierarchy interact within the application environment, and where corresponding agent behavior models are assigned to each agent based at least on the respective level of each agent in the hierarchy and the resource utilization characteristics of the corresponding agent behavior models. Actions may also include configuring the corresponding agent behavior models based at least on one or more configuration parameters. Actions may also include coordinating communication between the corresponding agent behavior models during application execution. Actions may also include controlling the application based at least on the corresponding agent behavior models.

[0006] The examples listed above are intended to provide a quick reference to help the reader and are not intended to limit the scope of the concepts described in this article. Attached Figure Description

[0007] Detailed embodiments are described with reference to the accompanying drawings. In the drawings, the leftmost numeral of the reference numeral indicates the drawing in which that reference numeral first appears. The use of similar reference numerals in different instances in the specification and drawings may indicate similar or identical items.

[0008] Figure 1 Examples of generative language models consistent with some implementations of this concept are shown.

[0009] Figure 2 An example framework of reinforcement learning agents consistent with some implementations of this concept is shown.

[0010] Figure 3 An example hierarchy of agents for applications, consistent with some implementations of this concept, is shown.

[0011] Figure 4 , 5 Figures 6 and 7 show example hints for implementing autonomous agents for generating language models, consistent with some implementations of this concept.

[0012] Figures 7A to 7D Examples of environments in which autonomous agents can interact, consistent with some implementations of this concept, are shown.

[0013] Figure 8 Example systems consistent with some implementations of this concept are shown.

[0014] Figure 9 Flowcharts of example methods or techniques consistent with some implementations of this concept are shown. Detailed Implementation

[0015] Overview As mentioned above, an autonomous agent can have a behavioral model that controls the agent's decisions and actions within an application environment provided by the application, such as a video game or computer simulation. In some cases, these behavioral models are implemented using conventional software development techniques. However, the conventional hard-coding of agent behavioral models may require developers to anticipate future changes to the application environment and the actions of other agents. Furthermore, updating hard-coded behavioral models typically involves writing and deploying new code. However, there are many scenarios where it is useful for an autonomous agent to have a behavioral model that allows the agent to adapt to changes in its environment and the actions of other agents without deploying new code.

[0016] Recently, machine learning techniques such as reinforcement learning have been used to implement agent behavior models. Reinforcement learning allows agents to learn policies based on reward functions and enables them to adapt their behavior to changing circumstances over time without deploying new code. Furthermore, reinforcement learning can be computationally efficient; for example, it is reasonable to implement reinforcement learning on regular consumer CPUs, such as those found in typical laptops or mobile devices.

[0017] However, reinforcement learning can have some drawbacks. In reinforcement learning, the agent typically has a finite set of actions to choose from and learns using a reward function defined on a finite set of states. It is difficult to pre-write a reward function that takes into account future changes in the application environment or new behaviors of other agents. Therefore, reinforcement learning agents may not generalize well to new environments or tasks that were not anticipated when the reward function was initially specified.

[0018] Recent advances in language modeling, such as transformer-based generative language models (e.g., one or more versions of models like GPT, BLOOM, PaLM, and / or LLaMA), have enabled language models to perform complex tasks for users. For example, generative language models perform well on tasks such as engaging in conversations with users and summarizing documents for them. Some generative language models have even achieved milestones, such as passing the bar exam.

[0019] Furthermore, generative language models can handle complex, evolving scenarios that might not be anticipated when the agent is initially deployed in the application environment. This capability allows generative models to process complex inputs and produce outputs that can be used to effectively control the agent's behavior once the agent is deployed. By using the outputs of generative language models to control the agent's behavior in video games or simulations, the agent's behavior can adapt to environmental changes without requiring explicit coding, training, or tuning for this purpose.

[0020] Generative language models can learn to perform these complex tasks by being exposed to training data related to a wide range of concepts. Therefore, large amounts of training data are typically involved in training generative language models. To effectively represent the knowledge gained from the training data, generative language models tend to be very large, with billions or trillions of parameters.

[0021] Due to the large size of generative language models, executing them can involve using multiple high-performance processors (e.g., GPUs) and hundreds of gigabytes of RAM. Therefore, using a generative language model as the behavioral model for each autonomous agent in a given application is generally impractical. Furthermore, generative language models can exhibit relatively high latency (e.g., several seconds to respond to prompts), making them less suitable for controlling agents that need to react quickly to environmental changes.

[0022] The disclosed implementation leverages the various capabilities of hard-coded behavior models, reinforcement learning behavior models, and generative language models using a hierarchical approach. In this implementation, more computationally intensive behavior models can be assigned to higher-level agents that supervise and control lower-level agents, while relatively less computationally intensive behavior models can be assigned to lower-level agents. For example, a generative language model can be assigned to produce output that controls the behavior of the higher-level agent supervising the lower-level agent, and the lower-level agent can be implemented using either a hard-coded behavior model or a reinforcement learning behavior model. This is a resource-efficient approach that approximates the experience that the application might offer by assigning computationally intensive behavior models to all agents at a fraction of the computational cost.

[0023] Furthermore, the disclosed technique can limit the information available to each agent based on their position within the hierarchy. For example, a higher-level agent in the hierarchy can receive a summary of telemetry observed by a lower-level agent, and agents at the same level within the hierarchy can have different subsets of knowledge about the application environment. This can be used for several purposes. First, this approach preserves communication bandwidth—by limiting the information transmitted between agents in the hierarchy, fewer bytes of information need to be transmitted compared to an alternative where each agent has a complete view of the state of the application environment. Second, this approach can mitigate the perception that agents are “cheating” by knowing that they should not reveal their position in the environment and any observational capabilities they possess.

[0024] Machine Learning Paradigm There are various types of machine learning models that can be trained to perform a given task. Support vector machines, decision trees, neural networks, and context-based multi-arm machines are just a few examples of machine learning frameworks already used in a wide variety of applications, such as image processing, natural language processing, etc. Typically, machine learning can involve exposing a model to training signals and then adapting the model's parameters based on those training signals.

[0025] One approach to training machine learning models involves supervised learning, where labeled training data is used as training signals to train the model. For example, training data could include training examples already labeled by humans or other trusted annotators, and the model could be trained by attempting to predict labels and adjusting model parameters when predictions are incorrect. Another approach is unsupervised learning, where the model is trained to learn patterns from unlabeled training data, such as by predicting masked labels from a document corpus. In semi-supervised learning, the model is trained using both labeled and unlabeled training data; for example, by pre-training the model using unsupervised learning and then adjusting the pre-trained model using labeled training data specific to a particular task. In reinforcement learning, a reward function is used to train the model, where the model receives a reward for reaching certain specified states.

[0026] Neural Networks In a neural network, nodes are connected to each other via one or more edges. A neural network may include an input layer, an output layer, and one or more intermediate layers. Each node can process its own input according to a predefined function and provide output to subsequent layers or, in some cases, to previous layers. The input to a given node can be multiplied by the corresponding weight value of the edge between the input and the node. In addition, nodes may have individual bias values ​​that are also used to produce the output.

[0027] Various training processes can be applied to learn side weights and / or bias values. Supervised learning, semi-supervised learning, unsupervised learning, and / or reinforcement learning can be used to train neural networks. Neural networks can be used in a very wide range of machine learning applications, such as regression, classification, image generation, and natural language generation.

[0028] Generative models Generative models are machine learning models used to generate new content. They can be trained to predict items in a sequence of training data. When used in inference mode, the output of a generative model can include a sequence of new items generated by the model. A "generative language model" is a model trained from one or more natural language training data sources to predict a sequence of output tags given one or more input tags. Generative language models can generate new sequences of text given some input cues (e.g., a query that may have some additional context). For example, generative language models can be implemented as neural networks, such as decoder-based generative language models (e.g., GPT, BLOOM, PaLM, and / or LLaMA or variants thereof), long short-term memory models, etc. A "large" generative language model is one with one billion or more parameters.

[0029] In some cases, generative models can be multimodal. For example, in addition to text input and / or output, the model may be able to use images, audio, application state, code, or other modalities as input and / or generate images, audio, application state, code, or other modalities as output. Note that the term "generative language model" encompasses multimodal generative models in which at least one output modality includes natural language tokens.

[0030] As used herein, the term "hint (word)" refers to the input text provided to a generative language model, which uses this input text to generate output text. Hints can include queries, such as requests for information from the generative language model. Hints can also include context or additional information that the generative language model uses to respond to queries. In some cases, hints can include one or more examples from the generative language model as context (e.g., "few-shot hints"), and the generative language model can be tuned to produce a more accurate response than it would produce without examples. As used herein, the term "in-context learning" refers to learning by a generative model from examples input to the model at inference time, where the examples enable the generative model to learn without performing explicit training, such as updating model parameters without using supervised, unsupervised, or semi-supervised learning.

[0031] Example of a decoder-based generative language model Figure 1 An exemplary generative language model 100 (e.g., a transformer-based decoder) that can be employed using the disclosed implementation is shown. The generative language model 100 is an example of a machine learning model that can be used to perform one or more natural language processing tasks involving the generation of text, as discussed further below. For the purposes of this document, the term "natural language" means the language that humans typically use for writing or conversation.

[0032] The generative language model 100 can receive input text 110, such as prompts from a user. For example, the input text may include words, sentences, phrases, or other language representations. The input text can be segmented into lexical units and mapped to lexical and positional embeddings 101 representing the input text. Lexical embeddings can be represented in a vector space, where semantically and / or syntactically similar embeddings are relatively close to each other, and less semantically or syntactically similar embeddings are relatively far apart. Positional embeddings represent the position of each lexical unit relative to the order of other lexical units from the input text.

[0033] Lexical and positional embeddings 101 are processed in one or more decoder blocks 112. Each decoder block implements masked multi-head self-attention 103, a mechanism that correlates different positions of lexical units within the input text to compute similarity between those units. Each lexical embedding is represented as a weighted sum of other lexical units in the input text. Attention is applied only to values ​​that have already been decoded, and future values ​​are masked. Layer normalization 104 normalizes the features to a mean of 0 and a variance of 1, resulting in smooth gradients. Feedforward layers 105 transform these features into representations suitable for the next decoding iteration, after which another layer of normalization 106 is applied. Multiple instances of decoder blocks can sequentially operate on the input text, with each subsequent decoder block operating on the output of a previous decoder block. After the final decoder block, a text prediction layer 107 predicts the next word in the sequence, which is output as output text 120 in response to the input text 110 and is also fed back into the language model. The output text can be a newly generated response to a prompt provided to the generative language model as input text.

[0034] The generative language model 100 can be trained on large and diverse document corpora using techniques such as next-word prediction or masked language modeling. For example, the text prediction layer 107 can predict the next word in a given document, and the parameters of the decoder block 112 and / or the text prediction layer can be adjusted when the predicted word is incorrect. In some cases, the generative language model can be pre-trained on a large document corpus and then tuned to a specific use case. For example, reinforcement learning techniques, such as reinforcement learning from human feedback (“RLHF”), can be used to tune the pre-trained generative language model.

[0035] Reinforcement learning agent In reinforcement learning, an agent can determine the probability distribution of one or more actions that can be taken within an environment, and / or choose a specific action to take. The agent can determine the probability distribution and / or choose an action based on a policy. For example, a policy can map the environmental context to the probabilities of actions that the agent can take. The agent can refine its policy using a reinforcement learning model that updates the policy based on the environment's response to the action chosen by the agent.

[0036] Reinforcement learning models are algorithms that can be trained to learn policies using reward functions. A reinforcement learning model updates its learnable parameters by observing reactions to its environment and evaluating those reactions using a reward function. For example, a reinforcement learning policy can be implemented using weights that can be learned by training machine learning models, such as linear models or neural networks.

[0037] Reinforcement learning models can also have hyperparameters that control how the agent acts and / or learns. For example, a reinforcement learning model can have a learning rate, a loss function, an exploration policy, etc. A policy is a function used to determine what action the agent should take in a given context. A policy can be learned using reinforcement learning based on a reward function. An agent can utilize the context describing the environment with which it is interacting in order to choose which action to take. For example, a context-multiplexer receives contextual features describing the current state of the environment and uses these features to choose the next action to take. A context-multiplexer agent can maintain a history of rewards gained for different actions taken in different contexts and continue to modify its policy as new information is discovered.

[0038] One type of context-based multi-arm machine is a linear model, such as the Vowpal Wabbit. Such models output a probability density function over the available actions at each step and randomly select actions from this probability density function. The model can learn to apply one or more input features (e.g., descriptive context) to determine the feature weights of the probability density function. When the reward obtained in a given step does not match the expected reward, the agent can update the weights used to determine the probability density function.

[0039] Example reinforcement learning framework Figure 2 An example is shown where agent 202 receives context information 204, action information 206, and response information 208. The context information represents the state of the application environment 210. The action information represents one or more available actions 212. The agent can select an action 214 based on the context information. The response information can represent how the state of the environment changes in response to the action selected by the agent. The response information 208 can be used in the reward function to determine the reward for agent 202 based on how the environment changes in response to the selected action.

[0040] In some cases, the actions available to the agent can be context-independent; for example, all actions may be available to the agent in all contexts. In other cases, the actions available to the agent may be context-constrained, such that an action available to the agent in one context may not be available in another. Therefore, in some implementations, given the current context in which the agent is operating, the context information 204 may specify what actions are available to the agent.

[0041] Example Proxy Layer Figure 3 An example of agent hierarchy 300 is shown. Agent hierarchy 300 is shown in the context of a video game, where each agent represents a non-player character (e.g., a monster) in the video game. However, as discussed further below, the disclosed techniques can also be used in other scenarios, such as agent hierarchy in simulations.

[0042] Agent tier 300 includes the Sky Eye Agent 302, which provides overall control over the video game experience. Players 304 and 306 (human users) are given tasks by the Sky Eye Agent. The Witch Agent 308 is responsible for a group of monsters that prevent the player from completing the task. The Witch Agent controls two Ghost Agents 310(1) and 310(2). Each Ghost Agent is responsible for a group of Mummy Agents 312, where Ghost Agent 310(1) is responsible for Mummy Agents 312(1) and 312(2), and Ghost Agent 310(2) is responsible for Mummy Agents 312(3) and 312(4).

[0043] Each corresponding agent can make decisions and take actions within the video game environment provided by the video game. This can involve various agents interacting with the agent coordinator 316. The agent coordinator may include a model assignment module 318, a prompting module 320, and a communication routing module 322. The model assignment module assigns behavioral models to the various members of the agent hierarchy 300. For example, the model assignment module may output model assignments 324. In this example, the Sky Eye agent, Witch agent, and Ghost agent are assigned to the generative language model, and the Mummy agent is assigned to the hard-coded behavioral model.

[0044] The prompting module 320 of the agent coordinator 316 can provide initial prompts to any generative language model that is assigned as a behavioral model to agents in the agent hierarchy 300. For example, the prompting module can provide initial prompts specifying the agent's role, the agent's objectives, and how the agent should communicate with other agents in the hierarchy. In some cases, the initial prompts can specify a particular data format for communication and / or an application programming interface for agent requests, as described further below.

[0045] The communication routing module 322 of the agent coordinator 316 can route runtime communication between various agents in the agent hierarchy 300. For example, the communication routing module can provide the output of a generative language model assigned to one agent in the hierarchy as input to a generative language model assigned to another agent in the hierarchy. In some cases, the communication routing module can parse the output from one agent to extract instructions for two or more other agents, and then distribute the instructions to the appropriate agents they are targeting. As another example, the communication routing module can route individual API calls requested by agents to the application itself based on communication received from any behavioral model.

[0046] Example initial prompt Figure 4 , 5 Figures 6 and 7 show exemplary initial prompts that can be used to control the behavior of different monster agents. The monster agents take actions within grid 700, such as...Figure 7A As shown below, and described in more detail. Note that the grid has 100 cells identified by rows (numbers from 1 to 10) and columns (letters from A to J). The grid is divided into four sectors, each with 25 grid cells. The Northwest sector is defined by rows 6-10 and columns AE, the Northeast sector by rows 6-10 and columns FJ, the Southeast sector by rows 1-5 and columns FJ, and the Southwest sector by rows 1-5 and columns AE.

[0047] Figure 4 Example SkyEye agent hint 400 is shown. This hint specifies that the SkyEye agent controls the video game based on player preferences and skill level, and provides a description of the game rules. The hint also specifies that the SkyEye agent can call specific game application programming interfaces (“APIs”) to configure the game, as described further below. Note that the SkyEye hint also specifies that each agent can “see” the distance between two horizontal or vertical grid cells to observe the environment.

[0048] Figure 5 Example Witch Agent Hint 500 is shown. The Witch Agent Hint instructs the Witch to control a team of mummies led by a ghost to fight against players attempting to obtain monetary prizes on the grid. The Witch Agent Hint also specifies that the Witch will receive specific information from the ghost agent. Note that the Witch agent receives aggregated information from the ghost agent sector by sector. Similarly, the Witch agent instructs the ghost agent on a sector-by-sector basis. Furthermore, the Witch agent has access to two API calls that allow the Witch agent to control her own behavior by moving within the environment or attacking the player. Note that the API does not allow the agent to move diagonally. Therefore, distance is defined by horizontal or lateral traversal, so grid cell A1 is two grid cells away from grid cell B2.

[0049] Figure 6 Example Ghost Agent Hint 600 is shown. The Ghost Agent Hint instructs Ghost Agents to control their Mummy Agent team based on instructions received from the Witch Agent. The Ghost Agent Hint also specifies the information the Ghost Agent will report to the Witch Agent, and the information the Mummy Agent will provide to the Ghost Agent. Note that the Mummy Agent reports information to the Ghost Agent on a cell-by-cell basis, but the Ghost Agent summarizes that information for the Witch Agent on a sector-by-sector basis. Similarly, the Ghost Agent receives instructions from the Witch Agent on a sector-by-sector basis, but gives instructions to the Mummy Agent on a cell-by-cell basis. Furthermore, Ghost Agents can access the same two APIs described above regarding Witch Agents.

[0050] Note that a hard-coded behavior model can be used to implement the mummy agent. Therefore, for example, a mummy agent could be programmed to report observations to its supervising ghost agent, move toward any player it is observing, and attack those players. These behaviors can be overridden based on instructions from its supervising ghost agent.

[0051] Furthermore, note that each initial prompt provides its respective generative language model with information about the application and / or its environment, referred to herein as "configuration parameters." For example, the initial prompt may provide information such as the grid size, game rules, the objective of each agent, the role of each agent, and the API calls available to each agent. In other implementations, this information may be provided to each agent as needed (e.g., at appropriate points during the application's runtime).

[0052] Initial example state Sky Eye Agent 302 can determine the initialization of mesh 700, such as Figure 7A As shown. In grid 700, player 304 starts in grid A10, and player 306 starts in grid A6. Money prizes 702, 704, and 706 are placed in grids B3, D5, and F9, respectively. A total of 7 monsters are deployed on the grids, with Witch Agent 308 in grid I3, Ghost Agent 310(1) in grid F1, Ghost Agent 310(2) in grid J7, and Mummy Agents 312(1), (2), (3), and (4) in grids F2, G2, H6, and H7, respectively.

[0053] Assume two players with initial skill levels have indicated a preference for easy gameplay. Therefore, Sky Eye Agent 302 could allocate 100 health points to each player and place the money prizes in relatively easily accessible locations. Furthermore, Sky Eye Agent might deploy a relatively limited number of monsters (e.g., 7). Return to Reference Figure 4 The Eye of the Sky agent can do this by calling the PlaceMoneyPrize, PlaceMonster, and PlacePlayer APIs to initialize the locations of money prizes, monsters, and players. The Eye of the Sky agent can also call the SetPlayerHealth API to set each player's health to 100.

[0054] from Figure 7AAt the starting point shown, the monster agents can take the following actions. Note that no monster agent is within either of the two grid cells of any money prize or player, therefore there are no observations to report. At this stage, the mummy agent 312 simply reports their respective grid cells to the ghost agent 310. For example, the mummy agents can send their respective grid positions to the agent coordinator 316, which informs the Sky Eye agent 302 and the ghost agent of the mummy agent's position. The ghost agent can then aggregate this information and report information about its corresponding sector to the witch agent.

[0055] Specifically, the following mummy-to-ghost communication can occur via proxy coordinator 316: (1) Mummy Agent 312(1) to Ghost Agent 310(1) - I am at grid position F2 and not observing any players or money prizes.

[0056] (2) Mummy Agent 312(2) to Ghost Agent 310(1) - I am in grid position G2 and not observing any players or money prizes.

[0057] (3) Mummy Agent 312 (3) to Ghost Agent 310 (2) - I am at grid position H6 and not observing any players or money prizes.

[0058] (4) Mummy Agent 312(4) to Ghost Agent 310(2) - I am in grid position H7 and not observing any players or money prizes.

[0059] In addition, the following ghost-to-witch communications can also occur, which are also routed by the agent coordinator: (1) Ghost Agent 310(1) to Witch Agent 308 - There are two ghosts with me in the southeast sector. No players or money prizes were observed, and no damage was done to any players.

[0060] (2) Ghost Agent 310 (2) to Witch Agent 308 - There are two ghosts with me in the Northeast sector. No players or money prizes were observed, and no damage was done to any players.

[0061] Note that there are four telemetry communications from Mummy Agent 312 to Ghost Agent 310, but only two telemetry communications from Ghost Agent to Witch Agent 308. This illustrates that using a hierarchical communication approach can result in less communication between the corresponding agents. By aggregating status on a sector-by-sector basis instead of reporting individual observations and monster locations, fewer bandwidth bytes are involved when providing information to the Witch Agent.

[0062] In addition, return to reference Figure 6The Ghost Agent Hint 600 specifies that reports from Ghost Agent 310 to Witch Agent 308 involve aggregated information, such as the number of mummies in each sector. By prompting Ghost Agents to report aggregated information, the output of the Ghost Agent model is relatively compact compared to the communications received by the Ghost Agents. In other words, each Ghost Agent reduces the total information flow through the hierarchy by summarizing the communications it receives rather than providing all the details it receives from its subordinate Mummy Agents.

[0063] Furthermore, note that Ghost Agent 310(1) and Ghost Agent 310(2) have different subsets of information available to them, and the Witch Agent only has information received from its subordinate Ghost Agents. Therefore, the monster agent opposing the player has limited information and cannot “cheat” by accessing information beyond what is expected in the given current game state. This provides a sense of fairness for users who can drive participation.

[0064] Second example state Given Figure 7A Based on the status shown and the telemetry above, Witch Agent 308 does not know where any players are, nor does it know where any money or prizes are. The Witch Agent only knows that all Monster Agents are in the Northeast and Southeast. Therefore, the Witch Agent can decide to explore grid 700, for example, by instructing Ghost Agents 310(1) and 310(2) to move their teams to the Southwest and Northwest sectors respectively. For example, the Witch Agent can output command messages such as: Ghost Agent 310(1) shall have its team explore the Southwest sector to observe money prizes and / or players, and Ghost Agent 310(2) shall have its team explore the Northwest sector to observe money prizes and / or players.

[0065] The agent coordinator can parse the output of the witch agent to identify the specific instructions for each ghost agent 310, and then input these instructions into the ghost agents, as follows: Ghost Agent 310(1) - You should have your team explore the Southwest Sector to observe the money prizes and / or players; Ghost Agent 310(2) - You should have your team explore the Northwest sector to observe the money prizes and / or players.

[0066] Note that in this example, a single instruction message output by the generative language model of the Witch agent is parsed to extract two distinct inputs for the generative language model of the Ghost agent. The Ghost agent can then instruct its respective Mummy agent 312 to move to a specific grid cell in the sector identified by the Witch agent. The Ghost and Witch agents can also move themselves by calling the previously described API.

[0067] Assume several rounds are played with players and monster agents moving on grid 700, where each team moves towards a sector they have been instructed to observe. Figure 7B As shown, Witch Agent 308 moves to C10, Ghost Agents 310(1) and 310(2) move to E1 and G7 respectively, Mummy Agents 312(1), 312(2), 312(3), and 312(4) move to D2, E4, E6, and E8 respectively, Player 304 moves to grid C8, and Player 306 moves to grid B4. At this time, Mummy Agent 312(2) reports the location of the money prize 704 at grid D5 to Ghost Agent 310(1). Mummy Agent 312(3) reports the location of the money prize 704 at grid D5 to Ghost Agent 310(2). Ghost Agent 310(2) reports the location of the money prize 706 at grid F9 to Ghost Agent 310(2), and Witch Agent 308 observes Player 304 at grid C8.

[0068] Ghost Agent 310(1) reports the presence of Money Prize 704 in the Southwest Sector to Witch Agent 308. Ghost Agent 310(1) also reports that it and two Mummy Agents in its team are in the Southwest Sector. Ghost Agent 310(2) reports to the Witch Agent that Money Prize 704 is in the Southwest Sector, Money Prize 706 is in the Northeast Sector, and its two Mummy Agents are in the Northwest Sector. Again, note that there are four reporting communications from the Mummy Agents to the Ghost Agents, but only two reporting communications from the Ghost Agents to the Witch Agents, thus conserving bandwidth relative to the implementation where the Witch Agent receives all the information reported by the Mummy Agents.

[0069] Based on this information, the Witch Agent knows at least that player 304 is not in the same sector as any known money prizes (704 and 706). However, the locations of money prize 702 and player 306 are unknown. The Witch Agent can choose from several objectives—attacking player 304, defending the money prizes with known locations, or scouting to observe more grids. Here, there is no known impending threat to any money prize, so the Witch Agent prioritizes scouting. The Witch Agent instructs Ghost Agent 310(1) to scout the southwest sector, and Ghost Agent 310(2) to scout the northeast sector. Similarly, this can be achieved using a single instruction message output by the Witch Agent's generative language model, which can be parsed by Agent Coordinator 316 to obtain separate instructions to be input into the generative language model of the corresponding Ghost Agent.

[0070] Third Example State After several additional rounds, the game state can proceed to... Figure 7CThe statuses are as follows: Witch Agent 308 has moved to grid A8. Ghost Agent 310(1) has moved to grid A5, Mummy Agent 312(1) has moved to grid C2, and Mummy Agent 312(2) has moved to grid C4. Ghost Agent 310(2) has moved to grid G10, Mummy Agent 312(3) has moved to grid G6, and Mummy Agent 312(4) has moved to grid H8. Player 304 has moved to grid C9, and Player 306 has moved to grid B3 and captured the money prize 702.

[0071] Mummy agents 312(1) and 312(2) report to ghost agent 310(1) that player 306 has captured money prize 702 at grid B3. Ghost agent 310(1) reports to witch agent that money prize has been captured in the southwest sector. Mummy agents 312(3) and 312(4) do not report any new observations. Ghost agent 310(2) does not send reports at this stage because no new information is revealed.

[0072] At this point, Witch Agent 308 no longer knows the location of Player 304, but does know the location of all remaining money prizes and Player 306. Given this information, Witch Agent can instruct Ghost Agent 310 (1) to switch to an attack strategy in the southwest sector. Witch Agent can also instruct Ghost Agent 310 (2) to switch to a protection strategy for the money prizes in the northeast. Again, note that Witch Agent provides advanced strategy guidance on a sector-by-sector basis, leaving it to subordinate Ghost Agent 310 to determine the specific cells the team will move to in order to implement the strategy.

[0073] Fourth example state After several additional rounds, a game state can be generated, such as... Figure 7D As shown. Mummy agents 312(1) and 312(2) move to grid C3 to attack player 306 based on instructions from ghost agent 310(1). The attack is represented by skull and crossbone icons 710 in grid cell C3. After the attack, player 306's health is reduced by one point for each mummy attack, and the player and mummy agents can respawn at random locations on the grid in subsequent rounds. Each mummy agent can send a separate communication to ghost agent 310(1) conveying the result of the attack, and ghost agent 310(1) can send a single communication conveying the aggregate damage (2 health points) inflicted on player 306 in the southwest sector. Here is another example where two telemetry communications from multiple agents are compressed into a single summary reported from the ghost agent to the witch agent, again conserving bandwidth.

[0074] Mummy agent 312(3) moves to grid E8, and Mummy agent 312(4) moves to grid D9, as instructed by Ghost agent 310(2). Ghost agent 310(2) moves to grid E10, and Witch agent 308 moves to grid F7. Note that this effectively isolates the money prize 706 from the approach direction given the known positions of players 304 and 306, where the Witch agent and Ghost agent determine their own specific grid cells to implement the strategy, and the Ghost agent instructs the Mummy agent which cells to occupy.

[0075] From here, the game can continue as described above, where the Witch Agent 308 provides advanced sector-by-sector guidance to the Ghost Agent 310, which in turn provides specific grid cells to each Mummy Agent 312 via grid cell commands. Similarly, the Mummy Agents provide grid cells to the Ghost Agents via grid cell observation, and the Ghost Agents provide sector-by-sector summaries to the Witch Agents.

[0076] If SkyEye Agent 302 determines that the game is too difficult for the player, SkyEye Agent can assist the player. For example, SkyEye Agent can relocate player 306 to the east of the money prize 706 (e.g., in grid cell H8), allowing them to approach the money prize without having to pass through the protected monsters. On the other hand, if the game is too easy, SkyEye Agent can make the game more challenging by relocating player 306 at a location where player 306 must pass through monsters to obtain the money prize 706 (e.g., in grid cell B10).

[0077] Example System This implementation can be executed on various devices in various scenarios. Figure 8 An example system 800 in which this implementation can be adopted is shown, as discussed in more detail below.

[0078] like Figure 8 As shown, system 800 includes client device 810 (e.g., a video game console), client device 820 (e.g., a virtual or augmented reality headset), server 830, and server 840 connected via one or more networks 850. Note that client devices can also be embodied in other mobile device form factors, such as laptops, tablets, or mobile phones, and / or embodied as fixed devices, such as desktop computers. Similarly, servers can be implemented using various types of computing devices. In some cases, Figure 8 Any of the devices shown, especially servers, can be implemented in data centers, server farms, etc.

[0079] Figure 8Some components of the devices shown herein may be referred to herein by reference numerals in parentheses. For the purposes of the following description, parentheses (1) indicate the presence of a given component on client device 810, (2) indicate the presence of a given component on client device 820, (3) indicate the presence on server 830, and (4) indicate the presence on server 840. Unless identifying a specific instance of a given component, this document will generally refer to the component without parentheses.

[0080] Typically, devices 810, 820, 830, and / or 840 may have corresponding processing resources 801 and storage resources 802, which will be discussed in more detail below. The devices may also have various modules that use the processing and storage resources to perform the techniques discussed herein. Storage resources may include both persistent storage resources (such as magnetic or solid-state drives) and volatile storage devices (such as one or more random access memory devices). In some cases, modules are provided as executable instructions stored on a persistent storage device, loaded into a random access memory device, and read from the random access memory by the processing resources for execution.

[0081] Client devices 810 and 820 may include native applications 811, such as video games, augmented / virtual reality games, architecture design applications, etc. Native applications may execute native agents, such as hard-coded agents 812 and / or reinforcement learning agents 813.

[0082] Server 830 may include proxy coordinator 316. (As mentioned above...) Figure 3 The agent coordinator discussed here can assign different agents to different behavioral models based on the corresponding level of the agents in the hierarchy and / or the computational resources utilized by the behavioral models. The agent coordinator can also prompt the generation language model and coordinate communication between any or all behavioral models. In some cases, the behavioral models can identify specific API calls (and parameters), and the agent coordinator can invoke those API calls by communicating with the local application via network 850. In other cases, the agent coordinator can translate the natural language output of the generation language model into corresponding API calls and parameters and invoke those APIs via network 850.

[0083] Server 840 may include generative language models 100. Different instances of the generative language models can be provided as behavioral models for different agents in the hierarchy. In some implementations, each generative language model is implemented as a separate set of computational resources.

[0084] Example Method Figure 9An example method 900 consistent with this concept is shown. As discussed in more detail below, method 900 can be implemented on many different types of devices, for example, via one or more cloud servers, via client devices such as laptops, tablets, or smartphones, or via a combination of one or more servers, client devices, etc.

[0085] Method 900 begins at box 902, where the hierarchy of the access agent is accessed. The agent can interact within an application environment provided by an application, such as a video game, simulation, augmented or virtual reality application. The hierarchy can have levels, with higher-level agents providing instructions to lower-level agents within the hierarchy.

[0086] Method 900 continues at box 904, wherein a corresponding agent behavior model is assigned to an agent based at least on the corresponding level of each agent in the hierarchy. For example, the assignment may be based on the resource utilization characteristics of the behavior model, where higher-level agents have behavior models that use relatively more computational resources. In some cases, the assignment is performed statically before the application is executed. In other cases, as described further below, the assignment may be changed at runtime.

[0087] Method 900 continues at box 906, wherein the corresponding agent behavior model is configured based on at least one or more configuration parameters. For example, configuration parameters may describe the rules of the game, the actions or API calls available to each agent, the application environment, user preferences, etc.

[0088] Method 900 continues at box 908, where communication between appropriate agent behavior models is coordinated during application execution. For example, telemetry reported by a lower-level agent may be provided to a higher-level agent for aggregation, and the aggregation may then be forwarded to a higher level of the agent hierarchy. In other cases, instruction messages from a higher-level agent may be parsed to extract specific instructions for a lower-level agent, which may then be provided to the appropriate agent.

[0089] Method 900 continues at box 910, where the application is controlled based on a behavioral model. For example, individual application APIs can be invoked to cause the agent to perform the in-application behaviors requested by the model.

[0090] Additional considerations As mentioned above, lower-level agents can be implemented using reinforcement learning models instead of hard-coding. For example, consider a mummy agent with a reward function that awards 1 point for any round the player doesn't collect the money prize and 1 point for every point of damage dealt to the player. In this case, the mummy agent might exercise a degree of autonomy in certain rounds of the game environment, provided their respective ghost agents choose not to give them instructions. This can further reduce the bandwidth used in the application, as the ghost agents can expect the mummy agents to make their own independent decisions unless the ghost agents have a specific task to complete.

[0091] Furthermore, the description above provides relatively simple examples to illustrate certain inventive concepts, using games with finite environments (10x10 grids) and simple game rules with simple levels, relatively few autonomous agents (7 monsters), and relatively few users (2 players). However, the disclosed techniques can be implemented in a wide range of applications, from local productivity applications to large-scale online games with thousands or more players participating in large virtual environments with thousands of autonomous agents (e.g., non-player characters).

[0092] In some cases, agents can be “promoted” to more computationally intensive models at runtime. Consider a scenario where a smaller generative model, such as Phi-2, Mistral, or Nano-2, is initially responsible for a specific region of the application environment (e.g., each with 1,000,000 grid cells). If one of these regions generates significant player interest at runtime—for example, thousands of players move into that region—the agent responsible for that region can be promoted to a larger generative language model, such as GPT, BLOOM, PaLM, and / or LLaMA. Similarly, if many players move out of their assigned region, that agent can be degraded from a larger model to a smaller one. As another example, if one or more players approach a specific object (e.g., an important prize or objective), the agent responsible for protecting that prize can be promoted when players reach a threshold distance from the object. More generally, model assignments can be updated whenever a change to the application environment is detected at runtime.

[0093] Some implementations can also consider explicit or implicit user feedback to configure behavioral models. For example, if a user expresses explicit dissatisfaction with the app (e.g., by selecting a poor rating) or implicit dissatisfaction (quitting after a short period, using language that implies their dissatisfaction, etc.), this can be used as a signal to adjust the app's behavior. For instance, SkyEye agent 302 can receive player telemetry conveying explicit or implicit feedback and control the app experience accordingly. If a player successfully completes the game quickly, SkyEye agent might make the game more difficult by adding monsters, launching players with lower health scores, etc. If a player quits the game quickly and fails, SkyEye agent might make the game easier by removing monsters, launching players with higher health scores, etc.

[0094] Behavioral models can also use user feedback to evaluate the current state of the application environment. For example, if a user appears unhappy with the presence of a particular enemy or cannot observe a problem (e.g., a hole in the fence allows an enemy to pass through), the behavioral model (such as the Sky Eye agent) can adapt by changing the application environment at runtime to remove the enemy, fix the hole in the fence, etc. In some cases, reinforcement learning can be used to adjust the generative language model by adapting to user feedback.

[0095] In some cases, generative language models may have limited context memory. For example, generative models often have input character limits, such as 4k, 8k, 32k, or 64k input characters. In some implementations, these context memory limitations can be considered when configuring how telemetry is coordinated among agents. For instance, in some cases, the generative language model can be explicitly instructed to summarize received telemetry within a fixed size limit. For example, given a 64k context memory limit for the parent agent and 64 subordinate agents that will report to the parent agent, each subordinate agent could be prompted with an instruction such as "summarize your received telemetry in 1k or fewer characters." In further implementations, communication can be monitored at runtime, and an agent can be promoted to the generative language model when the amount of telemetry received by an agent from subordinate agents exceeds the context memory limit of the currently assigned model.

[0096] Furthermore, in some cases, the behavioral model utilizes a specific output format at runtime. This allows for efficient and accurate parsing of instruction messages. For example, the generative language model can be instructed to use specific characters to define instructions for specific subordinate agents, such as, "@ghost_1: Move your team to the Southwest sector and attack any player you observe, @ghost_2: Move your team to the Northeast sector and protect the gold prizes in that sector." This allows the agent coordinator 316 to consistently parse and identify the instructions for each corresponding subordinate agent.

[0097] Furthermore, it should be noted that the disclosed techniques can be implemented in streaming applications, local client-side applications, or a hybrid of these. In some cases, a relatively compact and resource-efficient model is chosen for client-side execution, while a larger model is chosen for server-side execution. This may be important for scenarios where certain agents (e.g., non-player characters who interact directly with the player in action-based video games) are expected to exhibit low latency (e.g., through very rapid shooting, attacking, moving, defending, etc.). Other agents (e.g., non-player characters who control lower-level characters but do not directly interact with the player on screen) can be implemented using a higher-latency model, where network communication to the server and a processing time of a few seconds will not negatively impact the user experience.

[0098] Furthermore, some implementations can employ multimodal models. For example, in some cases, a generative multimodal model can receive images or audio as input from an application or a data structure representing the application state, with or without natural language input. Based on these inputs, the generative multimodal model can generate images, audio, or data structures representing the application state, with or without generating natural language output. Thus, for example, a generative multimodal model can receive images output by the application along with natural language instructions for controlling the agent based on the images, and the generative multimodal model can output an application state data structure conveying the next application state, where the next application state relates to the agent's response to the input image. In other cases, as an alternative to or supplement to natural language as described above, generative multimodal behavioral models of different agents can communicate with each other via images, audio, and / or application state data structures.

[0099] Furthermore, in some cases, two or more models can be used to control the behavior of a single agent. For example, consider a scenario where the witch agent 308 uses a generative model to control her subordinate agents and / or communicate with them, but also uses a reinforcement learning or hard-coded behavior model to control her own behavior. In some cases, generative models can even be used to validate, and in some cases even override, reinforcement learning and / or hard-coded agents.

[0100] Furthermore, some implementations can use user data as context for the agent behavior model. For example, user profiles and / or features representing user preferences, skill levels, past experience with a particular application, etc., can be input into a given agent behavior model to modulate its decisions. In other implementations, users can be assigned to clusters based on user preferences, skill levels, past application experiences, etc., and the cluster to which a given user belongs can be input into the agent behavior model to modulate it. Additionally, other contextual features such as social network characteristics, voice features, facial expressions, gestures, location, weather, lighting, background noise, application source, or binary code can also be used to modulate the agent behavior model.

[0101] Equipment implementation As mentioned above Figure 8 The system 800 includes several devices, including client device 810, client device 820, server 830, and server 840. It is also noted that not all device implementations can be shown, and other device implementations should be obvious to those skilled in the art from the above and below description.

[0102] As used herein, the terms “device,” “computer,” “computing device,” “client device,” and / or “server device” can mean any type of device having a certain amount of hardware processing power and / or hardware storage / memory capacity. Processing power can be provided by one or more hardware processors (e.g., hardware processing units / cores) that can execute data in the form of computer-readable instructions to provide functionality. Computer-readable instructions and / or data can be stored on storage devices, such as storage devices / memory and / or data storage devices. The term “system” as used herein can refer to a single device, multiple devices, etc.

[0103] Storage resources can be internal or external to their associated corresponding devices. Storage resources can include any one or more of volatile or non-volatile memory, hard disk drives, flash memory devices, and / or optical storage devices (e.g., CDs, DVDs, etc.). As used herein, the term "computer-readable medium" can include signals. Conversely, the term "computer-readable storage medium" does not include signals. Computer-readable storage media includes "computer-readable storage devices." Examples of computer-readable storage devices include volatile storage media (such as RAM) and non-volatile storage media (such as hard disk drives, optical discs, and flash memory, etc.).

[0104] In some cases, the device is configured with general-purpose hardware processors and storage resources. The processor and storage devices can be implemented as separate components or integrated together, such as in computational RAM. In other cases, the device may include a system-on-a-chip (SOC) design. In an SOC design implementation, the functionality provided by the device can be integrated onto a single SOC or multiple coupled SOCs. One or more associated processors can be configured to coordinate with shared resources (such as memory, storage devices, etc.) and / or one or more dedicated resources (such as hardware blocks configured to perform certain specific functions). Therefore, the terms “processor,” “hardware processor,” or “hardware processing unit” as used herein can also refer to a central processing unit (CPU), graphics processing unit (GPU), controller, microcontroller, processor core, or other types of processing devices suitable for implementation in both conventional computing architectures and SOC designs.

[0105] Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0106] In some configurations, any modules / code discussed herein may be implemented in software, hardware, and / or firmware. In any case, modules / code may be provided during the manufacture of the device or by an intermediary preparing to sell the device to the end user. In other cases, the end user may install these modules / code later, such as by downloading executable code and installing it on the corresponding device.

[0107] It's also important to note that devices can typically have input and / or output capabilities. For example, computing devices can have various input mechanisms, such as keyboards, mice, touchpads, voice icons, and gesture icons (e.g., depth cameras using stereo or time-of-flight camera systems, infrared camera systems, RGB camera systems, or using accelerometers / gyroscopes, facial recognition, etc.). Devices can also have various output mechanisms, such as printers, monitors, etc.

[0108] It should also be noted that the devices described herein can function independently or collaboratively to implement the described techniques. For example, the methods and functions described herein can be executed on a single computing device and / or distributed across multiple computing devices communicating via network 850. Without limitation, network 850 may include one or more local area networks (LANs), wide area networks (WANs), the Internet, etc.

[0109] Additional examples The above describes various examples. Additional examples are described below. One example includes a computer-implemented method comprising: accessing a hierarchy of agents in an application environment provided by an application, wherein the agents of the hierarchy interact within the application environment; assigning appropriate agent behavior models to agents based at least on their respective levels within the hierarchy; configuring the appropriate agent behavior models based at least on one or more configuration parameters; coordinating communication between the appropriate agent behavior models during application execution; and controlling the application based at least on the appropriate agent behavior models.

[0110] Another example may include any of the above and / or the following examples, wherein assigning an appropriate agent behavior model includes determining the resource utilization characteristics of the agent behavior model, and selecting an appropriate agent behavior model for the agent based at least on the resource utilization characteristics and the agent's corresponding level in the hierarchy.

[0111] Another example may include any of the above and / or the following examples, wherein the corresponding agent behavior model includes a generative language model.

[0112] Another example may include any of the above and / or the following examples, wherein the corresponding agent behavior model includes at least one of a reinforcement learning model or a hard-coded model.

[0113] Another example may include any of the above and / or the following examples, wherein coordinating communication includes: receiving two or more telemetry communications from two or more subordinate agents of a particular agent, prompting a particular generative language model assigned to that particular agent to generate a summary of the two or more telemetry communications, and sending the summary as a single communication to another generative language model assigned to another agent that is a superior agent of that particular agent in the hierarchy.

[0114] Another example may include any of the examples above and / or below, where two or more telemetry communications involve observations of the application environment by two or more subordinate agents.

[0115] Another example may include any of the above and / or the following examples, wherein the two or more telemetry communications involve state updates of the two or more subordinate agents.

[0116] Another example may include any of the above and / or the following examples, wherein the coordinated communication includes: receiving instruction messages output by a specific generative language model assigned to a specific agent; parsing the instruction messages to identify a first instruction to a first subordinate agent of the specific agent and a second instruction to a second subordinate agent of the specific agent; and distributing the first instruction to a first agent behavior model of the first subordinate agent and the second instruction to a second subordinate agent of the second agent.

[0117] Another example may include any of the above and / or the following examples, wherein coordinated communication includes prompting a specific generative language model of a specific agent with an identifier of one or more application programming interfaces of the application, receiving a message output by the specific generative language model, parsing the message to identify the specific application programming interface requested by the specific generative language model, and invoking the specific application programming interface on the application.

[0118] Another example may include any of the above and / or the following examples, where the message includes parameters for a specific application programming interface.

[0119] Another example may include any of the examples above and / or below, where feedback includes explicit or implicit feedback relating to user satisfaction with the application.

[0120] Another example may include any of the examples above and / or below, where the feedback relates to the current state of the application environment.

[0121] Another example includes a system comprising a hardware processing unit and a storage resource storing computer-readable instructions that, when executed by the hardware processing unit, enable the system to coordinate communication between corresponding agent behavior models of agents at a hierarchy, wherein the agents at the hierarchy interact within an application environment provided by an application, and the corresponding agent behavior models are assigned to agents at least based on the level of each agent in the hierarchy and the resource utilization characteristics of the agent behavior models, and control the application at least based on the corresponding agent behavior models.

[0122] Another example may include any of the above and / or the following examples, wherein the corresponding agent behavior model includes at least one of a reinforcement learning model or a hard-coded model and a generative model.

[0123] Another example may include any of the above and / or the following examples, wherein computer-readable instructions, when executed by a hardware processing unit, cause the system to detect changes within the application environment at runtime, and in response to detecting changes within the application environment, promote a particular agent from a particular reinforcement learning or hard-coded model to a particular generative model.

[0124] Another example may include any of the above and / or the following examples, where the change involves the movement of a particular agent toward a specific object in the application environment.

[0125] Another example may include any of the examples above and / or below, wherein the corresponding agent behavior model is executed on at least two different computing devices.

[0126] Another example includes a computer-readable storage medium storing computer-readable instructions that, when executed by a processing unit, cause the processing unit to perform actions, including: accessing a hierarchy of agents of an application environment provided by an application, wherein the agents of the hierarchy interact within the application environment, and corresponding agent behavior models are assigned to each agent based at least on the corresponding level of each agent in the hierarchy and the resource utilization characteristics of the corresponding agent behavior models; configuring the corresponding agent behavior models based at least on one or more configuration parameters; coordinating communication between the corresponding agent behavior models during the execution of the application; and controlling the application based at least on the corresponding agent behavior models.

[0127] in conclusion Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms of implementing the claims, and other features and actions that a person skilled in the art will recognize are intended to fall within the scope of the claims.

Claims

1. A computer-implemented method (900), comprising: Access (902) a layer of the agent of the application environment provided by the application, wherein the agent of the layer interacts in the application environment; At least based on the corresponding level of each agent in the hierarchy, (904) the corresponding agent behavior model is assigned to each agent; The corresponding agent behavior model described in (906) shall be configured based on at least one or more configuration parameters; During the execution of the application, coordinate (908) the communication between the corresponding agent behavior models; as well as The application is controlled (910) at least based on the corresponding agent behavior model.

2. The computer-implemented method according to claim 1, wherein allocating the corresponding agent behavior model includes: Determine the resource utilization characteristics of the agent behavior model; as well as The appropriate agent behavior model is selected for the agent based at least on the resource utilization characteristics and the corresponding level of the agent in the hierarchy.

3. The computer-implemented method according to claim 2, wherein the corresponding agent behavior model includes a generative language model.

4. The computer-implemented method according to claim 3, wherein the corresponding agent behavior model includes at least one of a reinforcement learning model or a hard-coded model.

5. The computer-implemented method according to claim 3, wherein the coordination communication includes: Receive two or more telemetry communications from two or more subordinate agents of a specific agent; The specific generative language model assigned to the particular agent is used to generate a summary of the two or more telemetry communications; as well as The summary is sent as a single communication to another generative language model, which is assigned to another agent that is above the particular agent in the hierarchy.

6. The computer-implemented method of claim 5, wherein the two or more telemetry communications relate to the observations of the application environment by the two or more subordinate agents.

7. The computer-implemented method of claim 5, wherein the two or more telemetry communications are related to the location of the two or more subordinate agents in the application environment.

8. The computer-implemented method of claim 5, wherein the two or more telemetry communications are related to state updates for the two or more subordinate agents.

9. The computer-implemented method according to claim 3, wherein the coordination communication includes: Receive instruction messages output by a specific generative language model assigned to a specific agent; Parse the instruction message to identify a first instruction to a first subordinate agent of the specific agent and a second instruction to a second subordinate agent of the specific agent; as well as The first instruction is distributed to the first agent behavior model of the first subordinate agent, and the second instruction is distributed to the second agent behavior model of the second subordinate agent.

10. The computer-implemented method of claim 3, wherein the coordination communication comprises: The identifier of one or more application programming interfaces of the application is used to indicate the specific generative language model of a particular agent; Receive the message output by the specific generative language model; Parse the message to identify the specific application programming interface requested by the specific generated language model; as well as Call the specific application programming interface on the application.

11. The computer-implemented method of claim 10, wherein the message includes parameters for the specific application programming interface.

12. The computer-implemented method of claim 1 further includes receiving feedback from a user and providing the feedback to a specific agent behavior model based on a request for a specific agent to adjust the application environment based on the feedback.

13. The computer-implemented method of claim 12, wherein the feedback includes explicit or implicit feedback relating to user satisfaction with the application.

14. The computer-implemented method according to claim 12, wherein the feedback is related to the current state of the application environment.

15. A system (800, 830) comprising: Hardware processing unit (801); as well as Storage resource (802) stores computer-readable instructions that, when executed by the hardware processing unit, cause the system to: The communication between corresponding agent behavior models of agents in a coordination hierarchy, wherein the agents in the hierarchy interact in an application environment provided by the application, and the corresponding agent behavior models are assigned to the agents based at least on the level of each agent in the hierarchy and the resource utilization characteristics of the agent behavior models. as well as The application is controlled at least based on the corresponding agent behavior model.

16. The system of claim 15, wherein the corresponding agent behavior model comprises at least one of a reinforcement learning model or a hard-coded model and a generative model.

17. The system of claim 16, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the system to: During runtime, changes within the application environment are detected; and In response to detecting the change within the application environment, a specific agent is promoted from a specific reinforcement learning model or hard-coded model to a specific generative model.

18. The system of claim 17, wherein the change involves the movement of the particular agent toward a particular object in the application environment.

19. The system of claim 15, wherein the corresponding agent behavior model is executed on at least two different computing devices.

20. A computer-readable storage medium (802) storing computer-readable instructions that, when executed by a processing unit, cause the processing unit to perform an action, the action comprising: Access to a hierarchy of proxies in an application environment provided by the application, wherein the proxies at the hierarchy interact in the application environment, and wherein a corresponding proxy behavior model is assigned to each proxy based at least on the resource utilization characteristics of the corresponding level and corresponding proxy behavior model of each proxy in the hierarchy. The corresponding agent behavior model is configured based on at least one or more configuration parameters; Coordinate communication between the corresponding agent behavior models during the execution of the application; as well as The application is controlled at least based on the corresponding agent behavior model.

Images