Multi-agent collaborative virtual object control method based on world model
The multi-agent collaborative virtual object control method addresses low sample efficiency in multi-agent systems by using Transformer-based world models to generate accurate joint sequences and reduce computational costs, enhancing expressibility and performance.
Patent Information
- Application Number
- CN202410983527.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-12-20
AI Technical Summary
Existing multi-agent systems face challenges in efficiently learning optimal cooperation strategies due to the large action space and partial observability, leading to low sample calculation efficiency in multi-agent environments.
A multi-agent collaborative virtual object control method utilizing a world model that leverages Transformer's sequence modeling potential to generate accurate and consistent joint sequences, aggregating perceptual information flexibly, and performing parallel trajectory fitting to improve sample calculation efficiency.
Enhances expressibility of the world model by generating accurate joint sequences and reduces computational cost through parameter sharing, resulting in improved sample efficiency and overall performance in multi-agent strategy learning.
Smart Images
Figure CN119158264A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent agent control, and in particular to a multi-agent collaborative virtual object control method based on a world model. Background Art
[0002] At present, in the technical field of network virtual object control, the use of intelligent agents for automatic decision-making is a common technology, which can realize functions such as automatic combat in online games. However, when controlling virtual objects, multiple agents are often required to control each object one by one. However, multi-agent systems have their own inherent challenges, such as the huge action space formed by the cooperation of multiple agents, and the partial observable characteristics of each agent to the environment. These algorithms require a very large number of trial and error samples to learn the optimal cooperation strategy. In the prior art, a world model can be used for multi-agent control, but the existing multi-agent world model scheme uses the excellent performance of the cyclic state space model in the single-agent world model as the base to fit the dynamics of the multi-agent environment. However, this method only considers the associated challenges of migrating the single-agent world model to the multi-agent scenario, and does not further consider the accuracy requirements of the imagined trajectory. Since the strategy learning of the agent depends on the quality of the imagined trajectory of the world model, the sample calculation efficiency in the optimal strategy learning of the multi-agent is low. Summary of the invention
[0003] The purpose of the present invention is to provide a multi-agent collaborative virtual object control method based on a world model in order to improve the sample calculation efficiency in multi-agent optimal strategy learning. It makes full use of the powerful sequence modeling potential of Transformer to make it possible to imagine a more accurate and consistent world model and generate an accurate complete joint sequence. In the process of generating a complete joint sequence, the global Transformer network flexibly aggregates perceptual information according to the number of different agents. At this time, the discretized sequence of each agent obtained, that is, the complete joint sequence already contains all the necessary information within a single time t, which can enhance the expressibility of the world model, and then predict the rewards and discount factors required for agent training based on the historical sequence. At the same time, the process of fitting the local trajectory sequence of the present invention is carried out in parallel on all n agents. Transformer also alleviates the high computational cost by sharing parameters, thereby improving the sample calculation efficiency in multi-agent optimal strategy learning.
[0004] The purpose of the present invention can be achieved by the following technical solutions:
[0005] A multi-agent collaborative virtual object control method based on a world model, the method comprising the following steps:
[0006] The trained agents corresponding to each virtual object are placed in the environment where the virtual object is located, and the current actual local observation of each trained agent is obtained. The trained agent outputs the current actual action based on the current actual local observation, and the virtual object executes the current actual action to realize the control of the virtual object. The training process of the agent is specifically as follows:
[0007] S1. Obtain training data, where the training data includes the current local observations of each agent at each moment, and select the current local observation at the first moment as the current local observation;
[0008] S2. Obtain an initial world model network, wherein the initial world model network includes an encoding network, a global Transformer network, and a multi-agent shared Transformer network, and train the encoding network based on a decoder to obtain an encoding model;
[0009] S3, the current local observation Input the coding model, and transform the current local observation Converted into discrete digital representation, the discrete digital representation and the action output by each agent and the discrete digital representation are integrated to obtain a discretized joint sequence. The encoding model includes an encoder E and a codebook
[0010] S4, the discretized joint sequence is input into the global Transformer network, and the global Transformer network compresses and aggregates the discretized joint sequence using the Query matrix to obtain the global perception characteristics of each agent for the entire environment;
[0011] S5. Integrate the global perception features and the discretized joint sequence to obtain a complete joint sequence of n agents. Based on the complete joint sequence, train the multi-agent shared Transformer network and the global Transformer network to obtain a multi-agent shared Transformer model and a global Transformer model. Then input the complete joint sequence of n agents into the multi-agent shared Transformer model. The multi-agent shared Transformer model outputs the individual rewards and individual discount factors of the n agents at the current moment. Based on the individual rewards and individual discount factors, train the strategy network and value network of the agents. Update the current moment to the next moment. Repeat S3 to S5 until convergence to obtain a trained agent.
[0012] Furthermore, the specific steps of training the encoding network based on the decoder to obtain the encoding model are:
[0013] The current local observation of each agent The input encoding network converts the local observations of each agent into The discrete digital representation is converted into a discrete digital representation, the discrete digital representation is input into a decoder, the decoder outputs an inverse discrete digital representation, and a coding network is trained based on the error between the discrete digital representation and the inverse discrete digital representation to obtain a coding model.
[0014] Furthermore, the global Transformer network uses the Query matrix to compress and aggregate the discretized joint sequence. The specific steps to obtain the global perception characteristics of each agent for the entire environment are as follows:
[0015] n discretized joint sequences are input into the global Transformer network, where n represents the total number of agents. The global Transformer network calculates the Key matrix and Value matrix corresponding to the n discretized joint sequences, and calculates the Query matrix corresponding to the latent matrix. The Key matrix, Value matrix and Query matrix are input into the cross-attention mechanism module. The output of the cross-attention mechanism module is used as the input of the perception Transformer module. The perception Transformer module outputs the global perception features of each agent for the entire environment.
[0016] Furthermore, the complete joint sequence is input into the multi-agent shared Transformer model, and the multi-agent shared Transformer model outputs the individual rewards and individual discount factors of the n agents at the current moment in the specific steps as follows:
[0017] The complete joint sequence is input into the multi-agent shared Transformer model. The multi-agent shared Transformer network predicts the individual reward and individual discount factor of each agent at the current moment based on the historical sequence.
[0018] Furthermore, the historical sequence is: Where x represents local observation, a represents action, and e represents global perception feature.
[0019] Furthermore, the specific steps of training the multi-agent shared Transformer network and the global Transformer network based on the complete joint sequence to obtain the multi-agent shared Transformer model and the global Transformer model are:
[0020] The complete joint sequence of n agents is input into the global Transformer network, and the output of the global Transformer network is input into the multi-agent shared Transformer network. The loss function is calculated based on the output of the multi-agent shared Transformer network, and the parameters of the global Transformer network and the multi-agent shared Transformer network are optimized based on the loss function. The multi-agent shared Transformer model and the global Transformer model are obtained by iterative training.
[0021] Furthermore, the multi-agent shared Transformer network includes a shared Transformer module.
[0022] Furthermore, the discretized joint sequence is input into the global Transformer network, and the global Transformer network uses the Query matrix to compress and aggregate the discretized joint sequence to obtain the global perception characteristics of each agent for the entire environment, which is performed in parallel on all n agents.
[0023] Furthermore, the multi-agent shared Transformer model shares parameters during the process of inputting the complete joint sequence of n agents into the multi-agent shared Transformer model.
[0024] Furthermore, the discrete digital representation is written as t represents the current time, i represents the i-th agent, x represents the local observation, and the discretized joint sequence is a represents an action, and the complete joint sequence is e represents the global perception feature.
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] The present invention proposes to use distributed Transformer and centralized Perceiver to jointly build a multi-agent world model, innovatively converting complex multi-agent dynamic prediction into a discrete sequence learning problem, making full use of the powerful sequence modeling potential of Transformer, making it possible to imagine a more accurate and consistent world model, and generating an accurate complete joint sequence. In the process of generating a complete joint sequence, the global Transformer network flexibly aggregates perceptual information according to the number of different agents. At this time, the discretized sequence of each agent obtained, that is, the complete joint sequence already contains all the necessary information within a single moment t, which can enhance the expressibility of the world model, and then predict the rewards and discount factors required for agent training based on the historical sequence. At the same time, the process of fitting the local trajectory sequence of the present invention is carried out in parallel on all n agents. Transformer also alleviates the high computational cost by sharing parameters, thereby improving the sample computation efficiency in multi-agent optimal strategy learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a structural schematic diagram of the present invention;
[0028] Figure 2 the imaginative process of a network of world models;
[0029] Figure 3 The figure is the result of the experiment. DETAILED DESCRIPTION
[0030] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0031] The present invention proposes a multi-agent collaborative virtual object control method based on a world model. The world model network constructed by the method is as follows: Figure 1 The method comprises the following steps:
[0032] The trained agents corresponding to each virtual object are placed in the environment where the virtual object is located, and the current actual local observation of each trained agent is obtained. The trained agent outputs the current actual action based on the current actual local observation, and the virtual object executes the current actual action to realize the control of the virtual object. The training process of the agent is specifically as follows:
[0033] S1. Obtain training data, where the training data includes the current local observations of each agent at each moment, and select the current local observation at the first moment as the current local observation;
[0034] S2. Obtain an initial world model network, wherein the initial world model network includes an encoding network, a global Transformer network, and a multi-agent shared Transformer network, and train the encoding network based on a decoder to obtain an encoding model;
[0035] S3, the current local observation Input the coding model, and transform the current local observation Converted into discrete digital representation, the discrete digital representation and the action output by each agent and the discrete digital representation are integrated to obtain a discretized joint sequence. The encoding model includes an encoder E and a codebook
[0036] S4, the discretized joint sequence is input into the global Transformer network, and the global Transformer network compresses and aggregates the discretized joint sequence using the Query matrix to obtain the global perception characteristics of each agent for the entire environment;
[0037] S5. Integrate the global perception features and the discretized joint sequence to obtain a complete joint sequence of n agents. Based on the complete joint sequence, train the multi-agent shared Transformer network and the global Transformer network to obtain a multi-agent shared Transformer model and a global Transformer model. Then input the complete joint sequence of n agents into the multi-agent shared Transformer model. The multi-agent shared Transformer model outputs the individual rewards and individual discount factors of the n agents at the current moment. Based on the individual rewards and individual discount factors, train the strategy network and value network of the agents. Update the current moment to the next moment. Repeat S3 to S5 until convergence to obtain a trained agent.
[0038] In the present invention, the multi-agent shared Transformer network includes a shared Transformer module.
[0039] Figure 1 The model in is divided into three parts: discretization of local observations of a single agent; global Transformer aggregating the local perception of all agents at each moment; and multi-agent shared Transformer fitting the discretized trajectory of a single agent.
[0040] For discretization of local observations of a single agent, in order to be able to use a unique discrete representation for each local observation of each agent, the present invention uses an encoder E and a codebook The local observations of each agent are transformed into Convert to discrete digital representation At the same time, in order to ensure that this representation can be inversely transformed and restored to the original local observation, a decoder D is introduced to map these digital representations back to the original local observation space. By reconstructing the errors before and after, a concise but meaningful discrete digital representation is learned.
[0041] With the above discretization tools, the local trajectory of a single agent Can be converted into a discretized sequence At each time t, the global Perceiver Transformer receives the discretized joint sequence of all n agents. Finally, a variable Query matrix with the same length as the number of agents is used to compress and aggregate the sequence, returning the global perception features of each agent for the entire environment. These global perception features will then be provided to the Transformer to enhance local dynamic predictions and accurately capture the interrelated relationships behind each agent.
[0042] Based on the complete discretized local trajectory sequence of each agent obtained above The shared Transformer module performs unsupervised learning in an autoregressive manner to predict the discrete digital representation of each future local observation. At the same time, in order to ensure that the strategy can be learned in this world model, the world model is based on the historical sequence Predict individual rewards for each agent and individual discount factors These two are necessary elements for learning the strategy and value function of the agent. The above local trajectory sequence fitting process is carried out in parallel on all n agents, and Transformer also alleviates the high computational cost by sharing parameters.
[0043] In the present invention, the imagination process of the world model network is as follows Figure 2 As shown. Taking any local observation sampled in the real environment as the starting point of imagination, at each moment t, each agent makes a decision based on the reconstructed local observation, and then the centralized aggregation module Perceiver aggregates the perception information of all agents and feeds it back to the discretized local sequence of each agent. At this time, the discretized sequence of each agent already contains all the necessary information within a single moment t. Transformer performs autoregressive prediction based on this sequence to generate imagination of future local observations. The above process is repeated until the limit of the imagination length of the world model is reached.
[0044] The multi-agent world model based on the Transformer base in the present invention is the main innovation of the present invention. The present invention proposes to use distributed Transformers and centralized Perceivers to jointly build a multi-agent world model, innovatively converting complex multi-agent dynamic predictions into discrete sequence learning problems, making full use of the powerful sequence modeling potential of Transformers, and making it possible to imagine a more accurate and consistent world model. At the same time, since it is the first multi-agent world model based on the Transformer base, the present invention provides a feasible solution for designing an effective and low-computational-cost centralized aggregation module under the base, that is, using the Perceiver model with a variable query matrix to flexibly aggregate perceptual information according to the number of different agents, thereby enhancing the expressibility of the world model.
[0045] Compared with the results of other currently optimal model-free policy learning algorithms trained in external real environments, the method proposed in this invention has improved the sample efficiency by ten or even dozens of times. At the same time, compared with the predictive dynamic multi-agent reinforcement learning algorithm based on the same learning paradigm, it has also achieved better sample efficiency and overall performance.
[0046] The present invention was tested on virtual objects and provided 23 different multi-agent task scenarios. In each scenario, the algorithm can manipulate one of the parties to defeat the preset enemy team. The experimental results show that the sample efficiency and overall winning rate achieved by the present invention are better than other related technical methods. The test in more than half of the representative task scenarios shows that the world model in the patent method has the ability to more accurately model the multi-agent environment, thereby enabling a large number of parallel imaginations of the world model to provide sufficient strategy learning. The results of the experiment are as follows Figure 3 shown.
[0047] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A multi-agent collaborative virtual object control method based on a world model, characterized in that: The method comprises the following steps: The trained agents corresponding to each virtual object are placed in the environment where the virtual object is located, and the current actual local observation of each trained agent is obtained. The trained agent outputs the current actual action based on the current actual local observation, and the virtual object executes the current actual action to realize the control of the virtual object. The training process of the agent is specifically as follows: S1. Obtain training data, where the training data includes the current local observations of each agent at each moment, and select the current local observation at the first moment as the current local observation; S2. Obtain an initial world model network, wherein the initial world model network includes an encoding network, a global Transformer network, and a multi-agent shared Transformer network, and train the encoding network based on a decoder to obtain an encoding model; S3, the current local observation Input the coding model, and transform the current local observation Converted into discrete digital representation, the discrete digital representation and the action output by each agent and the discrete digital representation are integrated to obtain a discretized joint sequence. The encoding model includes an encoder E and a codebook S4, the discretized joint sequence is input into the global Transformer network, and the global Transformer network compresses and aggregates the discretized joint sequence using the Query matrix to obtain the global perception characteristics of each agent for the entire environment; S5. Integrate the global perception features and the discretized joint sequence to obtain a complete joint sequence of n agents. Based on the complete joint sequence, train the multi-agent shared Transformer network and the global Transformer network to obtain a multi-agent shared Transformer model and a global Transformer model. Then input the complete joint sequence of n agents into the multi-agent shared Transformer model. The multi-agent shared Transformer model outputs the individual rewards and individual discount factors of the n agents at the current moment. Based on the individual rewards and individual discount factors, train the strategy network and value network of the agents. Update the current moment to the next moment. Repeat S3 to S5 until convergence to obtain a trained agent.
2. According to the method of multi-agent collaborative virtual object control based on world model in claim 1, it is characterized in that: The specific steps of training the encoding network based on the decoder to obtain the encoding model are: The current local observation of each agent The input encoding network converts the local observations of each agent into The discrete digital representation is converted into a discrete digital representation, the discrete digital representation is input into a decoder, the decoder outputs an inverse discrete digital representation, and a coding network is trained based on the error between the discrete digital representation and the inverse discrete digital representation to obtain a coding model.
3. A multi-agent collaborative virtual object control method based on a world model according to claim 2, characterized in that: The global Transformer network uses the Query matrix to compress and aggregate the discretized joint sequence. The specific steps to obtain the global perception characteristics of each agent for the entire environment are as follows: n discretized joint sequences are input into the global Transformer network, where n represents the total number of agents. The global Transformer network calculates the Key matrix and Value matrix corresponding to the n discretized joint sequences, and calculates the Query matrix corresponding to the latent matrix. The Key matrix, Value matrix and Query matrix are input into the cross-attention mechanism module. The output of the cross-attention mechanism module is used as the input of the perception Transformer module. The perception Transformer module outputs the global perception features of each agent for the entire environment.
4. The multi-agent collaborative virtual object control method based on a world model according to claim 3 is characterized in that: The complete joint sequence is input into the multi-agent shared Transformer model, and the multi-agent shared Transformer model outputs the individual rewards and individual discount factors of n agents at the current moment. The specific steps are: The complete joint sequence is input into the multi-agent shared Transformer model. The multi-agent shared Transformer network predicts the individual reward and individual discount factor of each agent at the current moment based on the historical sequence.
5. The multi-agent collaborative virtual object control method based on a world model according to claim 4 is characterized in that: The historical sequence is: Where x represents local observation, a represents action, and e represents global perception feature.
6. The multi-agent collaborative virtual object control method based on a world model according to claim 4 is characterized in that: The specific steps of training the multi-agent shared Transformer network and the global Transformer network based on the complete joint sequence to obtain the multi-agent shared Transformer model and the global Transformer model are as follows: The complete joint sequence of n agents is input into the global Transformer network, and the output of the global Transformer network is input into the multi-agent shared Transformer network. The loss function is calculated based on the output of the multi-agent shared Transformer network, and the parameters of the global Transformer network and the multi-agent shared Transformer network are optimized based on the loss function. The multi-agent shared Transformer model and the global Transformer model are obtained by iterative training.
7. The multi-agent collaborative virtual object control method based on a world model according to claim 6 is characterized in that: The multi-agent shared Transformer network includes a shared Transformer module.
8. The multi-agent collaborative virtual object control method based on a world model according to claim 7 is characterized in that: The discretized joint sequence is input into the global Transformer network, and the global Transformer network uses the Query matrix to compress and aggregate the discretized joint sequence to obtain the global perception characteristics of each agent for the entire environment, which is performed in parallel on all n agents.
9. The multi-agent collaborative virtual object control method based on a world model according to claim 7 is characterized in that: The multi-agent shared Transformer model shares parameters during the process of inputting the complete joint sequence of n agents into the multi-agent shared Transformer model.
10. The multi-agent collaborative virtual object control method based on a world model according to claim 1, characterized in that: The discrete digital representation is written as t represents the current time, i represents the i-th agent, x represents the local observation, and the discretized joint sequence is a represents an action, and the complete joint sequence is e represents the global perception feature.