Multi-agent cooperative modeling method and system for an adversary based on incomplete information
Patent Information
- Application Number
- CN202611109510.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-24
- Publication Date
- 2026-10-30
AI Technical Summary
[0004]为解决现有多智能体协同对抗场景中，对手建模方法因依赖对手完整观测数据而在部分可观测且通信受限场景下无法有效应用的问题，本发明主要提供了一种基于不完备信息的多智能体协同对抗对手建模方法及系统，以实现仅依靠己方智能体局部信息进行对手建模，进而提升多智能体深度强化学习算法的对抗性能
[0015]本发明产生的有益效果是：本发明通过建立的对手联合动作预测模型，在集中式训练阶段融合全局状态信息和己方智能体的交互轨迹信息进行己方智能体局部轨迹与近似对手联合观测信息之间的映射关系建立，从而实现在分布式执行阶段根据己方智能体的局部轨迹和映射关系进行对手动作预测，能够在部分可观测且通信受限的场景下，近似对手的联合观测信息，突破了现有方法对对手完整观测数据的依赖，显著提升了在不完备信息下的对手建模能力。
Smart Images
Figure CN123014067A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multi-agent reinforcement learning technology, and in particular to a method and system for modeling multi-agent cooperative adversaries based on incomplete information. Background Technology
[0002] In multi-agent cooperative adversarial scenarios, multiple agents from the same faction need to cooperate against an opposing faction, and these agents typically make decisions under partially observable conditions. Currently, multi-agent reinforcement learning has become an important tool for solving such problems, and related research has made some progress, but many challenges remain.
[0003] Existing adversary modeling methods have limitations in multi-agent cooperative adversarial scenarios. Most of these methods assume the availability of local trajectory data of the adversary during the training or execution phase. For example, in centralized training, they rely on acquiring complete observation-action data of the adversary and using supervised learning or imitation learning to build the adversary model; in distributed execution, they depend on the adversary's real-time local observation information to predict its actions. However, in real-world scenarios where observation is partially observable and communication is limited, agents often cannot communicate and lack a global perspective. Each agent can only make decisions based on its current observation information and historical trajectory data. Furthermore, due to the lack of a global perspective and the unknown adversary observation model, adversary observation information is difficult to obtain, and the adversary's observation range is also affected by environmental noise and dynamic occlusion, introducing uncertainty. This makes the assumptions of existing adversary modeling methods difficult to uphold, severely limiting their application in real-world scenarios. Summary of the Invention
[0004] To address the problem that existing adversary modeling methods in multi-agent cooperative adversarial scenarios cannot be effectively applied in partially observable and communication-limited scenarios due to their reliance on complete adversary observation data, this invention mainly provides a multi-agent cooperative adversary modeling method and system based on incomplete information. This allows for adversary modeling based solely on local information from one's own agents, thereby improving the adversarial performance of multi-agent deep reinforcement learning algorithms.
[0005] The technical solution adopted in this invention is: A method for modeling multi-agent cooperative adversaries based on incomplete information is provided, the method comprising: Acquire local trajectory information of your own intelligent agent and input it into the opponent's joint action prediction model; Solve the opponent's joint action prediction model to obtain the predicted opponent's actions, and conduct friendly agent confrontation based on the prediction results; Specifically, the adversary joint action prediction model establishes a mapping relationship between the local trajectory of the agent and the approximate adversary joint observation information during the centralized training phase, and solves the predicted adversary actions based on the local trajectory information of the agent and the mapping relationship during the distributed execution phase. Specifically, the centralized training phase acquires global state information and the interaction trajectory information of the agent, decomposes them into discrete features and continuous features, maps the continuous features to discrete features through a codebook, and concatenates the continuous features and the mapped discrete features to obtain approximate adversary joint observation information. Based on the interaction trajectory information of the agent and the obtained approximate adversary joint observation information, a mapping relationship is established between the local trajectory of the agent and the approximate adversary joint observation information.
[0006] According to the above scheme, the interaction trajectory information of the agent includes the agent's observation data, coordinate position, action probability distribution and executed actions at each time step; the global state information includes global discrete variables and global continuous variables.
[0007] According to the above scheme, continuous features are obtained by splicing together the coordinate positions and observation data of the agent at each time step, as well as the continuous variables of the global state, and mapping them.
[0008] According to the above scheme, the discrete features are obtained by the low-dimensional embedding of the global discrete variables through the gated recurrent unit, and the low-dimensional embedding of the global discrete variables and the continuous features are concatenated through the codebook. The vector closest to the concatenated data in the codebook is found and assigned to the nearest cluster center to obtain the discrete features.
[0009] According to the above scheme, the adversary joint action prediction model also reconstructs the continuous features obtained from the mapping to obtain the reconstructed action probability distribution and global continuous variables of the friendly agent. Based on the reconstructed action probability distribution and global continuous variables of the friendly agent and the action probability distribution and global continuous variables of the friendly agent before reconstruction, the reconstruction loss of the continuous features is calculated.
[0010] According to the above scheme, the adversary joint action prediction model also reconstructs the discrete features obtained by mapping, and obtains the reconstructed action probability distribution and global discrete variables of the friendly agent. Based on the reconstructed action probability distribution and global continuous variables of the friendly agent, the action probability distribution and global discrete variables of the friendly agent before reconstruction, and the vector quantization loss function, the reconstruction loss of the discrete features is calculated.
[0011] According to the above scheme, the vector quantization loss function is specifically calculated based on the hyperparameters of the codebook update loss, alignment loss, and the proportion of alignment loss to codebook update loss in the representation loss function.
[0012] A multi-agent cooperative adversary modeling system based on incomplete information is also provided, the system comprising: The information acquisition module is used to acquire local trajectory information of the friendly agent and input it into the opponent's joint action prediction model; The prediction module is used to solve the prediction model of the opponent's joint actions, obtain the predicted opponent actions, and conduct the friendly agent's confrontation based on the prediction results. Specifically, the adversary joint action prediction model establishes a mapping relationship between the local trajectory of the agent and the approximate adversary joint observation information during the centralized training phase, and solves the predicted adversary actions based on the local trajectory information of the agent and the mapping relationship during the distributed execution phase. Specifically, the centralized training phase acquires global state information and the interaction trajectory information of the agent, decomposes them into discrete features and continuous features, maps the continuous features to discrete features through a codebook, and concatenates the continuous features and the mapped discrete features to obtain approximate adversary joint observation information. Based on the interaction trajectory information of the agent and the obtained approximate adversary joint observation information, a mapping relationship is established between the local trajectory of the agent and the approximate adversary joint observation information.
[0013] A computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-agent cooperative adversary modeling method based on incomplete information described above.
[0014] A computer storage medium is also provided, which stores a computer program that can be executed by a processor, the computer program executing the multi-agent cooperative adversary modeling method based on incomplete information described above.
[0015] The beneficial effects of this invention are as follows: By establishing an adversary joint action prediction model, this invention integrates global state information and the interaction trajectory information of the friendly agent during the centralized training phase to establish a mapping relationship between the local trajectory of the friendly agent and the approximate adversary joint observation information. This enables adversary action prediction based on the local trajectory and mapping relationship of the friendly agent during the distributed execution phase. It can approximate the adversary's joint observation information in scenarios where some observations are available and communication is limited. This breaks through the dependence of existing methods on complete adversary observation data and significantly improves the adversary modeling capability under incomplete information. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the method flow of the multi-agent cooperative adversary modeling method based on incomplete information according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the hybrid encoder structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the opponent model construction according to an embodiment of the present invention; Figure 4 This is an overall flowchart of the opponent joint action prediction model algorithm according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the system structure of a multi-agent cooperative adversary modeling system based on incomplete information according to an embodiment of the present invention. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0018] Existing adversary modeling methods have limitations in multi-agent cooperative adversarial scenarios and are difficult to model adversaries with incomplete information. This invention addresses these limitations by using adversary faction agents... The actual observation information can be broken down into the following four parts:
[0019] in, It is an intelligent agent of the opposing camp. At time step The observed local environmental conditions; It is the state information of the teammate intelligent agents within its observation range; It is the state information of the adversary intelligent agent within its observation range; This refers to its own state information, including location and other information. Based on this decomposition, the joint observation information of the opposing camp can be represented as:
[0020] It is the number of intelligent agents in the opposing camp; among them That is, environmental information within the joint observation range of the adversaries. Teammate information and its own information All are global states A subset of, and This reflects the results of its observation of the state of its own agents. Therefore, the joint observation information of the opposing camp is essentially only related to the global state and the state of its own agents. The joint observation of the opposing camp can be inferred in reverse based on the global state and the state of its own agents.
[0021] Therefore, to address the problems of existing technologies, embodiments of the present invention provide a method for modeling multi-agent cooperative adversaries based on incomplete information, such as... Figure 1 As shown, the method includes: S1. Obtain the local trajectory information of your own agent and input it into the opponent's joint action prediction model.
[0022] S2. Solve the opponent's joint action prediction model to obtain the predicted opponent's actions, and conduct the friendly agent's confrontation based on the prediction results.
[0023] Specifically, the adversary joint action prediction model establishes a mapping relationship between the local trajectory of the agent and the approximate adversary joint observation information during the centralized training phase, and solves the predicted adversary actions based on the local trajectory information of the agent and the mapping relationship during the distributed execution phase. Specifically, the centralized training phase acquires global state information and the interaction trajectory information of the agent, decomposes them into discrete features and continuous features, maps the continuous features to discrete features through a codebook, and concatenates the continuous features and the mapped discrete features to obtain approximate adversary joint observation information. Based on the interaction trajectory information of the agent and the obtained approximate adversary joint observation information, a mapping relationship is established between the local trajectory of the agent and the approximate adversary joint observation information.
[0024] Specifically, the centralized training phase constructs a system such as Figure 2 The hybrid encoder shown addresses the issue of missing adversary joint observations during distributed execution. It comprises a continuous feature encoder and a discrete feature encoder, used to extract features of continuous variables and discrete variables from the global state and the friendly agent's state, respectively. Furthermore, it fuses feature information from both the global state and the friendly agent's state to obtain an approximate estimate of the adversary joint observations. .
[0025] Specifically, the interaction trajectory information of the agent includes the agent's observation data, coordinate position, action probability distribution, and executed actions at each time step; the global state information includes global discrete variables and global continuous variables. The agent's interaction trajectory information and global state information can be formally represented as:
[0026] in, Indicates the first Each intelligent agent at time step Observational data; Indicates the first Each intelligent agent at time step The coordinates below; For the first Each intelligent agent at time step Action probability distribution This indicates the action it performs. These represent continuous and discrete variables in the global state, respectively; the trajectory length terminates at the total time step length T. .
[0027] Specifically, this embodiment uses a parameterized VAE as a continuous feature encoder / decoder to extract continuous features from the global state and the state of the friendly agent, which is formally expressed as follows:
[0028] Among them, continuous feature encoder Observe your own intelligent agent The position of one's own intelligent agent and continuous variables in the state Mapping to continuous features The formal expression is as follows:
[0029] in It's a splicing operation. , Together they constitute continuous features Gaussian distribution, continuous features obtained by using reparameterization techniques. .
[0030] Furthermore, the adversary joint action prediction model reconstructs the continuous features obtained from the mapping to obtain the action probability distribution of the friendly agent and global continuous variables, and calculates the reconstruction loss. Specifically, the reconstruction process can be represented as follows:
[0031] The reconstruction loss can be expressed as:
[0032] Furthermore, the discrete features are specifically obtained by acquiring the low-dimensional embeddings of the global discrete variables through a gated recurrent unit. The low-dimensional embeddings of the global discrete variables are then concatenated with the continuous features using a codebook. The nearest vector in the codebook to the concatenated data is then found and assigned to the nearest cluster center, thus mapping to the discrete features. Specifically, this embodiment uses a parameterized VQ-VAE as the discrete feature encoder to mine discrete features, formally expressed as follows:
[0033] Among them, discrete variables in the environment are learned through gated cyclic units. Low-dimensional embedding And by introducing a codebook, continuous features are... Low-dimensional embedding of discrete environmental variables Mapping to discrete features , can be represented as:
[0034] in, During the mapping process, The vector is replaced with the nearest discrete codeword from the codebook. The codebook is a learnable matrix, represented as follows: ,in ; The number of vectors in the codebook determines the granularity of the discrete representation in the feature space. VQ-VAE achieves this by finding vectors in the codebook that correspond to... The nearest vector is used to achieve discretization, that is:
[0035] in, It is Euclidean distance, and this process will be continuous. The feature space is discretized, forming a finite set of discrete states. Each codeword can be considered a cluster center. Its function is to assign the input vector to the nearest cluster center.
[0036] To avoid the discrete feature encoder arbitrarily outputting vectors unrelated to the codebook, and to enable the codebook to dynamically adapt to the feature distribution of the discrete feature encoder, the vector quantization loss function of the discrete feature encoder can be expressed as:
[0037] in, This is a gradient clipping operation; It is a hyperparameter used to balance the proportion of alignment loss and codebook update loss in the loss function; This represents the codebook update loss, where only the codebook is trained and the encoder is not updated. This represents the alignment loss, where only the encoder is trained without updating the codebook; the alignment loss is mainly used to prevent... The frequent switching of coordinates in the codebook leads to model instability.
[0038] Preferably, the adversary joint action prediction model further reconstructs the mapped discrete features to obtain the action probability distribution of the friendly agent and global discrete variables:
[0039] The total loss function of the discrete feature codec can then be expressed as:
[0040] in, Is it the agent's own intelligence at the time step? The actual joint action.
[0041] Therefore, the formula for calculating the total loss of a hybrid encoder can be expressed as:
[0042] By concatenating the outputs of the continuous feature encoder and the discrete feature encoder, the discrete and continuous features of the global state and the friendly agent's state are effectively fused, resulting in a more comprehensive approximate estimate of the joint observation prediction of the adversary, which can be expressed as:
[0043] Specifically, intelligent agents The local trajectory can be represented as ,in Represents intelligent agents At time step The observation, Representing intelligent agents respectively At time step The action, reward, and termination identifiers. This embodiment assumes that at any time step... Below, space There are latent variables in it. This makes it include intelligent agents Local trajectory The information also includes joint observations similar to those of the adversary. The information is obtained through latent variables. This represents the established mapping relationship between the local trajectory of the friendly agent and the joint observation information of the approximate adversary.
[0044] Specifically, this embodiment utilizes a parameterized encoder to control the intelligent agent. Local trajectory Mapped to low-dimensional latent variables ,Right now ,in These are the encoder parameters. Subsequently, the hybrid encoder utilizes the global state and the state of the friendly agent to generate approximate joint observations from the adversary. Finally, the latent variables are decoded. Reconstructed into approximate adversary joint observations ,Right now ,in These are the parameters of the decoder. The formula for calculating the reconstruction loss of the codec is:
[0045] Specifically, this embodiment constructs a system as follows: Figure 3The Opponent Transformer model shown uses the mapping relationship between the local trajectory of the friendly agent and the joint observation information of the approximate opponent obtained in the centralized training phase. In the distributed execution phase, the encoder only needs the local trajectory of the agent. This allows the generation of joint observations that include approximate adversaries. Latent variables of information The decoder will then... Reconstructed This enables the agent to effectively estimate the opponent's joint observations without acquiring information about the opponent, relying only on its own local trajectory.
[0046] Specifically, the overall algorithm flow of the opponent joint action prediction model in the above embodiment is as follows: Figure 4 As shown.
[0047] Furthermore, embodiments of the present invention also provide a multi-agent cooperative adversary modeling system based on incomplete information, used to implement the multi-agent cooperative adversary modeling method based on incomplete information described in embodiments of the present invention, such as... Figure 5 As shown, the system includes: The information acquisition module is used to acquire local trajectory information of the friendly agent and input it into the opponent's joint action prediction model; The prediction module is used to solve the prediction model of the opponent's joint actions, obtain the predicted opponent actions, and conduct the friendly agent's confrontation based on the prediction results. Specifically, the adversary joint action prediction model establishes a mapping relationship between the local trajectory of the agent and the approximate adversary joint observation information during the centralized training phase, and solves the predicted adversary actions based on the local trajectory information of the agent and the mapping relationship during the distributed execution phase. Specifically, the centralized training phase acquires global state information and the interaction trajectory information of the agent, decomposes them into discrete features and continuous features, maps the continuous features to discrete features through a codebook, and concatenates the continuous features and the mapped discrete features to obtain approximate adversary joint observation information. Based on the interaction trajectory information of the agent and the obtained approximate adversary joint observation information, a mapping relationship is established between the local trajectory of the agent and the approximate adversary joint observation information.
[0048] The various modules or mechanisms of the system are mainly used to implement the various steps of the above method embodiments, and will not be described in detail here.
[0049] Furthermore, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-agent cooperative adversary modeling method based on incomplete information described above.
[0050] Finally, this embodiment also provides a computer storage medium storing a computer program that can be executed by a processor, the computer program executing the multi-agent cooperative adversary modeling method based on incomplete information described above.
[0051] This invention establishes an adversary joint action prediction model. During the centralized training phase, it integrates global state information and the interaction trajectory information of the friendly agent to establish a mapping relationship between the local trajectory of the friendly agent and the approximate adversary joint observation information. This enables adversary action prediction based on the local trajectory of the friendly agent and the mapping relationship during the distributed execution phase. It can approximate the adversary's joint observation information in scenarios where some observations are available and communication is limited. This breaks through the dependence of existing methods on complete adversary observation data and significantly improves the adversary modeling capability under incomplete information.
[0052] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.
[0053] The order of the steps in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0054] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A method for modeling multi-agent cooperative adversaries based on incomplete information, characterized in that, The method includes: Acquire local trajectory information of your own intelligent agent and input it into the opponent's joint action prediction model; Solve the opponent's joint action prediction model to obtain the predicted opponent's actions, and conduct friendly agent confrontation based on the prediction results; Specifically, the adversary joint action prediction model establishes a mapping relationship between the local trajectory of the agent and the approximate adversary joint observation information during the centralized training phase, and solves the predicted adversary actions based on the local trajectory information of the agent and the mapping relationship during the distributed execution phase. Specifically, the centralized training phase acquires global state information and the interaction trajectory information of the agent, decomposes them into discrete features and continuous features, maps the continuous features to discrete features through a codebook, and concatenates the continuous features and the mapped discrete features to obtain approximate adversary joint observation information. Based on the interaction trajectory information of the agent and the obtained approximate adversary joint observation information, a mapping relationship is established between the local trajectory of the agent and the approximate adversary joint observation information.
2. The multi-agent cooperative adversary modeling method based on incomplete information according to claim 1, characterized in that, The interaction trajectory information of the agent includes the agent's observation data, coordinate position, action probability distribution and executed actions at each time step; the global state information includes global discrete variables and global continuous variables.
3. The multi-agent cooperative adversary modeling method based on incomplete information according to claim 2, characterized in that, Continuous features are obtained by concatenating and mapping the coordinate positions of the agent at each time step, the observation data, and the continuous variables of the global state.
4. The multi-agent cooperative adversary modeling method based on incomplete information according to claim 2, characterized in that, Discrete features are obtained by using a gated recurrent unit to acquire low-dimensional embeddings of global discrete variables. The low-dimensional embeddings of global discrete variables are then concatenated with continuous features using a codebook. The codebook then finds the vector closest to the concatenated data and assigns the vector to the nearest cluster center, thus mapping the discrete features.
5. The multi-agent cooperative adversary modeling method based on incomplete information according to claim 2, characterized in that, The adversary joint action prediction model also reconstructs the continuous features obtained from the mapping to obtain the reconstructed action probability distribution and global continuous variables of the friendly agent. Based on the reconstructed action probability distribution and global continuous variables of the friendly agent and the action probability distribution and global continuous variables of the friendly agent before reconstruction, the reconstruction loss of the continuous features is calculated.
6. The multi-agent cooperative adversary modeling method based on incomplete information according to claim 2, characterized in that, The adversary joint action prediction model also reconstructs the discrete features obtained from the mapping to obtain the reconstructed action probability distribution and global discrete variables of the friendly agent. Based on the reconstructed action probability distribution and global continuous variables of the friendly agent, the action probability distribution and global discrete variables of the friendly agent before reconstruction, and the vector quantization loss function, the reconstruction loss of the discrete features is calculated.
7. The multi-agent cooperative adversary modeling method based on incomplete information according to claim 6, characterized in that, The vector quantization loss function is specifically calculated based on the hyperparameters of the codebook update loss, alignment loss, and the proportion of alignment loss to codebook update loss in the representation loss function.
8. A multi-agent cooperative adversary modeling system based on incomplete information, characterized in that, The system includes: The information acquisition module is used to acquire local trajectory information of the friendly agent and input it into the opponent's joint action prediction model; The prediction module is used to solve the prediction model of the opponent's joint actions, obtain the predicted opponent actions, and conduct the friendly agent's confrontation based on the prediction results. Specifically, the adversary joint action prediction model establishes a mapping relationship between the local trajectory of the agent and the approximate adversary joint observation information during the centralized training phase, and solves the predicted adversary actions based on the local trajectory information of the agent and the mapping relationship during the distributed execution phase. Specifically, the centralized training phase acquires global state information and the interaction trajectory information of the agent, decomposes them into discrete features and continuous features, maps the continuous features to discrete features through a codebook, and concatenates the continuous features and the mapped discrete features to obtain approximate adversary joint observation information. Based on the interaction trajectory information of the agent and the obtained approximate adversary joint observation information, a mapping relationship is established between the local trajectory of the agent and the approximate adversary joint observation information.
9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-agent cooperative adversary modeling method based on incomplete information as described in any one of claims 1-7.
10. A computer storage medium, characterized in that, It contains a computer program that can be executed by a processor, which performs the multi-agent cooperative adversary modeling method based on incomplete information as described in any one of claims 1-7.