Communication cooperation method and system based on reinforcement learning

Abstract

The invention relates to the technical field of multiple agents, in particular to a communication cooperation method and system based on reinforcement learning. The method comprises the steps that each agent obtains local observation information of the agent, and the hidden state and action are determined based on the local observation information; determining a hard interaction value between the intelligent agent and the neighbor intelligent agent based on the hidden state, and determining interaction strength between the intelligent agent and the neighbor intelligent agent based on the local observation information; determining a received information contribution based on the hard interaction value, the interaction strength and the hidden state; constructing Q values of the intelligent agents based on local observation information, actions and receiving information contribution of each intelligent agent, and training reinforcement learning based on the Q values of all the intelligent agents to output a communication cooperation strategy; and determining a communication cooperation strategy in the combat scene based on the reinforcement learning model. According to the invention, the Q value is constructed based on the local observation information, the action and the received information contribution, the efficient and fair credit distribution between energy bodies can be realized, and the communication cooperation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and multi-agent technology, and in particular to a communication collaboration method and system based on reinforcement learning. Background Art

[0002] In recent years, multi-agent systems (MAS) have played a vital role in the field of modern artificial intelligence and have been widely used in fields such as unmanned swarms, intelligent manufacturing, and traffic control. A multi-agent system consists of multiple autonomous agents that can solve complex tasks beyond the capabilities of a single agent through interaction, communication, and collaboration. Collaboration and competition between agents are crucial to achieving common goals. However, the design of effective communication and collaboration strategies between agents faces many challenges, especially in dynamic and uncertain environments. Traditional centralized control methods have obvious limitations when dealing with large-scale multi-agent systems because they are difficult to scale and have high communication bandwidth requirements. On the other hand, distributed control methods, while having advantages in scalability, face challenges in coordination and credit allocation between agents.

[0003] Multi-agent reinforcement learning (MARL) learns optimal strategies through interaction with the environment and can handle a wide range of tasks. However, existing MARL approaches, when used in large-scale combat scenarios, suffer from the uncertainty introduced by environmental changes. This leads to irrational allocation of contributions to individual agents, resulting in low collaboration efficiency among multiple agents.

[0004] Therefore existing technology still needs to be improved and improved. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a communication collaboration method and system based on reinforcement learning to address the shortcomings of the existing technology.

[0006] In order to solve the above technical problems, the first aspect of the present application provides a communication collaboration method based on reinforcement learning, wherein the communication collaboration method based on reinforcement learning specifically includes: Each agent obtains its own local observation information and determines its current hidden state and action based on the local observation information; Each agent obtains the neighboring agents with which it has a communication relationship, determines the hard interaction value between the agent and the neighboring agents based on the hidden state, and determines the interaction strength between the agent and the neighboring agents based on the local observation information; Each agent determines its contribution to the received information based on its hard interaction value, interaction strength, and the hidden states of neighboring agents; Constructing a Q-value for each agent based on the local observation information, the action, and the received information contribution of each agent, and training reinforcement learning based on the Q-values ​​of all agents to output a communication cooperation strategy in a combat scenario; Each intelligent agent performs communication cooperation based on the communication cooperation strategy.

[0007] The communication collaboration method based on reinforcement learning, wherein determining the hidden state and action at the current moment based on the local observation information specifically includes: Determine hidden state and output data through the mamba model based on the local observation information; The posterior distribution of the output data is sampled to obtain an uncertainty vector, and an action at a current moment is determined based on the output data and the uncertainty vector.

[0008] In the communication and collaboration method based on reinforcement learning, the step of each intelligent agent acquiring neighboring intelligent agents with which it has a communication relationship specifically includes: Each agent determines a communication subgraph centered on each agent based on its own local observation information; Each agent selects neighboring agents with which it has a communication relationship based on its corresponding communication subgraph.

[0009] In the communication and cooperation method based on reinforcement learning, the step of determining the hard interaction value between the agent and the neighboring agents based on the hidden state specifically includes: Feed the hidden state of the agent and the hidden states of its neighboring agents into the hard attention model; The hard interaction value between the agent and the neighboring agent is input through the hard attention model, wherein the hard interaction value is used to reflect whether the agent and the neighboring agent need to interact.

[0010] In the communication and cooperation method based on reinforcement learning, determining the interaction intensity between an agent and its neighboring agents based on local observation information specifically includes: Obtaining a first embedding representation corresponding to the local observation information of the agent and a second embedding representation corresponding to the local observation information of the neighboring agent; Based on the first embedding representation and the second embedding representation, the interaction strength between the agent and the neighboring agents is determined through a bidirectional gated recurrent model.

[0011] In the communication and cooperation method based on reinforcement learning, each agent determines the received information contribution of each agent based on the hard interaction value, interaction strength, and hidden state of neighboring agents, specifically including: Each agent uses the product of the hard interaction value and the interaction strength as the weight, and weights the hidden states of each neighboring agent to obtain the agent's contribution to received information.

[0012] The communication and collaboration method based on reinforcement learning, wherein the step of constructing the Q value of each agent based on the local observation information, the action, and the received information contribution of each agent specifically includes: Extract features from the local observation information and actions of each agent to obtain local observation features and action features; The local observation features, action features and received information contribution of each agent are input into the multi-layer perceptron, and the Q value of each agent is output through the multi-layer perceptron.

[0013] The communication and cooperation method based on reinforcement learning, wherein the local observation information includes position information, health value, shield value, field of view, surrounding path information, terrain height information, enemy unit information and distance within the field of view, and friendly unit information and distance within the field of view.

[0014] The communication and cooperation method based on reinforcement learning, wherein the action space of the combat scene includes movement direction, attack unit identification, stop and no operation, the dead intelligent agent can only take no operation, and the surviving intelligent agent takes other actions except no operation.

[0015] A second aspect of the present application provides a communication collaboration system based on reinforcement learning, wherein the communication collaboration system based on reinforcement learning includes a plurality of intelligent agents, each of which is deployed with a communication collaboration strategy based on reinforcement learning, and performs communication collaboration in a combat scenario based on the communication collaboration strategy, wherein the process of determining the communication collaboration strategy specifically includes: Each agent obtains its own local observation information and determines its current hidden state and action based on the local observation information; Each agent obtains the neighboring agents with which it has a communication relationship, determines the hard interaction value between the agent and the neighboring agents based on the hidden state, and determines the interaction strength between the agent and the neighboring agents based on the local observation information; Each agent determines its contribution to the received information based on its hard interaction value, interaction strength, and the hidden states of neighboring agents; The Q value of each intelligent agent is constructed based on the local observation information, the action and the received information contribution of each intelligent agent, and reinforcement learning is trained based on the Q values ​​of all intelligent agents to output the communication cooperation strategy in the combat scenario.

[0016] Beneficial Effects: Compared with the prior art, the present application provides a communication cooperation method and system based on reinforcement learning, the method comprising: each intelligent agent obtaining its own local observation information and determining its current hidden state and action based on the local observation information; each intelligent agent obtaining neighboring intelligent agents with which it has a communication relationship, determining the hard interaction value between the intelligent agent and the neighboring intelligent agent based on the hidden state, and determining the interaction strength between the intelligent agent and the neighboring intelligent agent based on the local observation information; each intelligent agent determines the received information contribution of each intelligent agent based on the hard interaction value, the interaction strength, and the hidden state of the neighboring intelligent agent; constructing the Q value of each intelligent agent based on the local observation information, the action, and the received information contribution of each intelligent agent, and training reinforcement learning based on the Q values ​​of all intelligent agents to output a communication cooperation strategy in a combat scenario; and determining the communication cooperation strategy in a combat scenario based on a trained reinforcement learning model. In the present application, each intelligent agent constructs its Q value based on its own local observation information, the action, and the received information contribution, which can achieve efficient and fair credit distribution between agents and improve the efficiency of communication cooperation. At the same time, this application optimizes the decentralized strategy under the centralized training and decentralized execution paradigm, enabling the intelligent agent to quickly adapt and adjust the strategy in a dynamic environment, thereby enhancing the scalability and robustness of the multi-agent system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A flowchart of a communication collaboration method based on reinforcement learning provided in an embodiment of the present application.

[0019] Figure 2 This is a flow chart of the principles of the communication collaboration method based on reinforcement learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The embodiments of this application provide a communication collaboration method and system based on reinforcement learning. To make the objectives, technical solutions, and effects of this application more clear and explicit, the application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to explain this application and are not intended to limit this application.

[0021] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0022] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0023] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0024] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0025] This embodiment provides a communication and cooperation method based on reinforcement learning, which can be applied to combat scenarios. The combat scenarios include various maps, units, and tasks. In the combat scenarios, multiple agents compete with each other or with built-in game AI to form a game round. There will be two competing teams in the game round. A game round ends when all units of any team die or a pre-specified time limit is reached. The goal of each game round is to maximize the winning rate of the learned strategy. Each unit in each game round is controlled by an independent agent, which improves its skills through training to solve challenging combat scenarios. For example, the combat scenario is a multi-agent combat scenario implemented by the game engine of "StarCraft 2".

[0026] Each agent in the combat scenario participates in a Markov game , the game consists of tuples Indicates. Among them, Represents a global state set, which may include the coordinates of the map center, the health of all units, skill cooldowns, position information, field of view, available actions, and the last action performed; Represents the joint action space. The action space of the agent is a discrete set of actions, which can include movement direction (such as east, south, west, and north), attack unit identification, stop and no operation. The dead agent can only take no operation, and the surviving agent takes other actions except no operation. Each agent In time Based on local observation information Execute an action , the local observation of the agent is limited to a limited range centered on the agent, including position information, health value, shield value, field of view, surrounding path information, terrain height information, enemy unit information and distance within the field of view, and friendly unit information and distance within the field of view. represents the transition probability function, which means that Transfer to state probability; Represents a shared reward, which is shared by the entire multi-agent system under a fully cooperative setting. To increase the difficulty of the task, this application chooses a sparse reward setting, where the environment only returns a reward of +1 (win) and a reward of -1 (failure); Represents a discount factor that weighs the importance of future rewards against immediate rewards.

[0027] Specifically, if Figure 1 As shown, the communication collaboration method based on reinforcement learning provided in the embodiment of the present application specifically includes: S10. Each agent obtains its own local observation information and determines its hidden state and action at the current moment based on the local observation information.

[0028] Specifically, each agent is included in a multi-agent system. Each agent in the multi-agent system corresponds to a combat unit in the combat scenario. The goal of each agent is to learn a strategy network. , the policy network maximizes the expected discounted return obtained from its interaction with the environment and other agents, so that the agent optimizes its own policy network while considering the behavior of other agents.

[0029] Furthermore, the policy network carried by each agent can adopt the Mamba model. The Mamba model can be expressed as the equation describing the system state evolution and the output equation, where the state update equation and the output equation are: , , in, Indicates time The hidden state of represents the state transition matrix, represents the input matrix, Indicates time The hidden state of Indicates time The input data, Indicates time The output data, Represents the transformation matrix.

[0030] Based on this, the agent can determine the hidden state and action (i.e., the output at the current moment) at the current time through the Mamba model by obtaining its own local observation information and the previous hidden state. In addition, to address the randomness and dynamics of the multi-agent system, the embodiments of the present application, based on the above-mentioned Mamba model, continuously update the latent state based on the historical input sequence, and use the posterior distribution of the hidden state to represent the uncertainty of environmental changes, so as to avoid the impact of the uncertainty caused by environmental changes on multi-agent learning.

[0031] Exemplarily, determining the hidden state and action at the current moment based on the local observation information specifically includes: Determine hidden state and output data through the mamba model based on the local observation information; The posterior distribution of the output data is sampled to obtain an uncertainty vector, and an action at a current moment is determined based on the output data and the uncertainty vector.

[0032] Specifically, if Figure 2 As shown, the previous hidden state is the hidden state at the previous moment. The mamba model stores the previous hidden state. After obtaining the local observation information, the local observation information will be input into the mamba model. The mamba model will first perform linear mapping on the local observation information to obtain a local observation vector, and then pass the local observation vector through a convolution layer to obtain an input vector. Finally, the input vector is input into the selective state space model (Selective State Space Model, SSM for short). The hidden state and initial output data at the current moment are determined by the selective state space model, and then the initial output data is fused with the local observation vector to obtain fused output data. Finally, the fused output data is linearly mapped to obtain output data.

[0033] After obtaining the output data, the posterior distribution of the output data is sampled to obtain an uncertainty vector, and the uncertainty vector is combined with the output data to obtain the action at the current moment. The uncertainty vector and the action at the current moment can be expressed as: , , in, Indicates time The uncertainty vector, represents the posterior distribution, Indicates time The output data, Indicates time No. The actions of an agent, Represents a join operation.

[0034] The policy network in the reinforcement learning model in the embodiment of the present application adopts the core idea of ​​the Mamba model, which can efficiently capture the complex dependencies in the sequence data. At the same time, the selective state space model and uncertainty vector are introduced into the Mamba model. The selective state space model allows the agent to dynamically adjust its state representation during the decision-making process to adapt to the changing environment and task requirements; the uncertainty vector is determined by the posterior sampling technique. The uncertainty vector can reflect the uncertainty of environmental changes and can then handle the randomness and dynamics of the multi-agent system. The uncertainty vector is used to enhance the output of the policy network. The adaptability of the two-tooth policy network to non-stationarity helps the agent make more robust decisions.

[0035] S20. Each intelligent agent obtains neighboring intelligent agents with which it has a communication relationship, determines the hard interaction value between the intelligent agent and the neighboring intelligent agent based on the hidden state, and determines the interaction strength between the intelligent agent and the neighboring intelligent agent based on the local observation information.

[0036] Specifically, neighboring agents are agents within the local interaction range of an agent and in communication with it. In other words, the multi-agent system utilizes game theory principles to construct a collaborative model between agents and enables effective communication between agents through a graph neural network (GNN). Each agent determines its corresponding communication subgraph based on its own local observation information and, based on this communication subgraph, determines its neighboring agents with which it has a communication relationship. Based on this, each agent acquires neighboring agents with which it has a communication relationship, specifically by: Each agent determines a communication subgraph centered on each agent based on its own local observation information; Each agent selects neighboring agents with which it has a communication relationship based on its corresponding communication subgraph.

[0037] Specifically, the agent obtains local observation information including position information, health value, shield value, field of view, surrounding path information, terrain height information, enemy unit information and distance within the field of view, and friendly unit information and distance within the field of view. Based on this local observation information, the agent can determine the information of friendly units within the field of view, and then establish a communication subgraph centered on the agent based on the obtained friendly unit information. The communication subgraph is then input into the graph neural network to determine friendly units with effective communication with the agent. Then, friendly units that can send information to the agent are selected from the friendly units with effective communication as neighboring agents. In other words, the neighboring agent can receive the local observation information and hidden state sent by the neighboring agent, and thus, based on the local observation information and hidden state sent by the neighboring agent, the hard interaction value and interaction strength between the agent and the neighboring agent can be determined based on the local observation information and hidden state of the agent itself and the local observation information and hidden state of the neighboring agent.

[0038] After acquiring its corresponding neighboring agent, the agent receives interaction information from the neighboring agent. This interaction information may include the neighboring agent's local observation information and its current hidden state. In other words, after acquiring a neighboring agent, the agent can send an interaction request to the neighboring agent, which in turn sends interaction information back to the agent based on the interaction request. The agent can then determine the hard interaction value and interaction strength between the agent and the neighboring agent based on the interaction information sent by the neighboring agent, its own local observation information, and its current hidden state.

[0039] The hard interaction value between an agent and its neighboring agents is used to reflect whether the agent and its neighboring agents need to interact. The hard interaction value is determined based on the hidden state of the agent and the hidden state of its neighboring agents. Specifically, the hard interaction value between an agent and its neighboring agents is determined based on the hidden state. The hidden state of the agent and the hidden state of the neighboring agent are input into the hard attention model, and the hard interaction value between the agent and the neighboring agent is input through the hard attention model, wherein the hard interaction value is used to reflect whether the agent and the neighboring agent need to interact.

[0040] Specifically, the hard attention model is used to interactively learn the hidden state of the agent and the hidden state of the neighboring agent to determine the attention value of the agent and the neighboring agent, and use the attention value as the hard interaction value, where the hard interaction value is a binary value, that is, the hard interaction value is 0 or 1, 0 means that there is no need for interaction between the agent and the neighboring agent, and 1 means that there is a need for interaction between the agent and the neighboring agent.

[0041] After the hidden state of the agent and the hidden state of the neighboring agents are input into the hard attention model, the hard attention model will combine the hidden state of the agent and the hidden state of the neighboring agents to form a state vector, and construct a query vector, value vector, and key vector based on the state vector. Then, based on the query vector, value vector, and key vector, the hard interaction value between the agent and the neighboring agents is determined through the attention mechanism. Based on this, the process of obtaining the hard interaction value can be expressed as: , in, Representing an agent and agents Neighbor Agent The hard interaction value between represents the hard attention model, Representing an agent The hidden state of Represents neighbor agents The hidden state of .

[0042] Of course, in practical applications, other methods can also be used to obtain the hard interaction value between the agent and its neighboring agents, for example, based on the distance between the agent and its neighboring agents.

[0043] Furthermore, the interaction strength is used to reflect the closeness of the interaction between the agent and its neighboring agents. The interaction strength can make the contribution value of the neighboring agents more accurate. The interaction strength is determined based on local observation information, and its determination process may include: Obtaining a first embedding representation corresponding to the local observation information of the agent and a second embedding representation corresponding to the local observation information of the neighboring agent; Based on the first embedding representation and the second embedding representation, the interaction strength between the agent and the neighboring agents is determined through a bidirectional gated recurrent model.

[0044] Specifically, the first embedding representation and the second embedding representation are obtained by mapping the local observation information of the agent and the local observation information of the neighboring agent to the same vector space. For example, the local observation information of the agent and the local observation information of the neighboring agent can be encoded by a preset encoder to determine the first embedding representation and the second embedding representation. After obtaining the first embedding representation and the second embedding representation, the first embedding representation and the second embedding representation are input into the bidirectional gated recurrent model, and the interaction strength between the agent and the neighboring agent is determined based on the bidirectional gated recurrent model. The calculation formula of the interaction strength can be expressed as: , in, Representing an agent and agents Neighbor Agent The interaction strength between Represents neighbor agents A bidirectional gated recurrent model centered on Representing an agent The corresponding first embedding representation is, Represents neighbor agents The corresponding second embedding representation is, and Both represent weight matrices.

[0045] The embodiments of the present application simplify the multi-agent learning process by abstracting the game, introduce graph neural networks and attention mechanisms to learn the interactive relationships between agents, and use the local observation information of the agents as communication content, thereby improving the fairness and efficiency of individual contribution calculations and achieving a better multi-agent collaboration strategy.

[0046] S30. Each agent determines the received information contribution of each agent based on the hard interaction value, interaction strength, and hidden state of neighboring agents.

[0047] Specifically, the received information contribution is used to reflect the information contribution received by an agent from other agents. This allows an agent to learn about the information contribution of other agents to itself in a complex multi-agent system, thereby enabling agents to achieve effective collaboration in a complex multi-agent system. The received information contribution can be obtained by weighting the hidden state of each neighboring agent using the hard interaction value and interaction strength as weighting coefficients. That is, each agent determines the received information contribution of each agent based on the hard interaction value, interaction strength, and the hidden state of the neighboring agents. Specifically, each agent uses the product of the hard interaction value and interaction strength as the weight, and weights the hidden state of each neighboring agent to obtain the received information contribution of the agent.

[0048] Based on this, the calculation formula for the contribution of received information can be: , in, Representing an agent The corresponding received information contribution, Representing an agent and agents Neighbor Agent The interaction strength between Representing an agent and agents Neighbor Agent The hard interaction value between Represents neighbor agents The hidden state of .

[0049] Of course, in practical applications, other methods can also be used to determine the contribution of received information, such as first calculating the individual contribution of the agent to each neighboring agent. , then configure a weighted coefficient for the monomer reception information contribution based on the local observation information of the neighboring agents, and finally weight the reception information contribution of each monomer based on the weighted coefficient to obtain the reception information contribution of the agent, etc.

[0050] S40, constructing the Q value of each intelligent agent based on the local observation information, the action and the received information contribution of each intelligent agent, and training reinforcement learning based on the Q values ​​of all intelligent agents to output the communication cooperation strategy in the combat scenario.

[0051] Specifically, the Q-value of an agent is an indicator of its contribution to its team. It can be used to assess the potential impact of different actions and select actions that maximize the team's overall performance. The Q-value of an agent is determined based on local observations, actions, and the contribution of received information. By using the contribution of received information to construct the Q-value, agents can effectively assist other agents.

[0052] Exemplarily, constructing the Q value of each agent based on the local observation information, the action, and the received information contribution of each agent specifically includes: Extract features from the local observation information and actions of each agent to obtain local observation features and action features; The local observation features, action features and received information contribution of each agent are input into the multi-layer perceptron, and the Q value of each agent is output through the multi-layer perceptron.

[0053] Specifically, local observation features are obtained by mapping local observation information to feature space, and action features are obtained by mapping actions to feature space. In other words, when determining the Q-value of an agent, the local observation features of the local observation information and the action features of the action are first extracted. Then, the Q-value of each agent is determined based on the local observation features, action features, and the contribution of the received information. The local observation features and action features can be extracted using a preset feature extraction model, such as the feature extraction module in VGG.

[0054] Based on this, the Q-value function corresponding to the Q-value of the agent can be expressed as: , in, Representing an agent The Q-value function, Represents the preset feature extraction model, represents a multilayer perceptron, Representing an agent The local observation information of Representing an agent action, Representing an agent The corresponding received information contribution.

[0055] Furthermore, after obtaining the Q-value of each agent, a Q-value loss term is constructed based on the Q-values ​​of all agents, and the policy network of each agent is updated based on this Q-value loss term to perform reinforcement learning. By adopting a centralized training and decentralized execution paradigm, the embodiment of the application optimizes decentralized strategies, enabling agents to quickly adapt and adjust strategies in a dynamic environment, thereby enhancing the scalability and robustness of the system.

[0056] S50. Each intelligent agent performs communication cooperation based on the communication cooperation strategy.

[0057] Specifically, after obtaining the communication cooperation strategy, each intelligent agent uses the communication cooperation strategy as its corresponding strategy network, and performs communication cooperation in the combat scenario based on the communication cooperation strategy.

[0058] In summary, this embodiment provides a communication cooperation method based on reinforcement learning, which includes each agent obtaining its own local observation information and determining the hidden state and action at the current moment based on the local observation information; each agent obtains the neighboring agents with which it has a communication relationship, determines the hard interaction value between the agent and the neighboring agent based on the hidden state, and determines the interaction intensity between the agent and the neighboring agent based on the local observation information; each agent determines the received information contribution of each agent based on the hard interaction value, the interaction intensity, and the hidden state of the neighboring agent; constructs the Q value of each agent based on the local observation information, the action, and the received information contribution of each agent, and trains reinforcement learning based on the Q values ​​of all agents to output a communication cooperation strategy in a combat scenario; and determines the communication cooperation strategy in a combat scenario based on a trained reinforcement learning model. In this application, each agent constructs the Q value of each agent based on its own local observation information, the action, and the received information contribution, which can achieve efficient and fair credit distribution between agents and improve the efficiency of communication cooperation. At the same time, this application optimizes the decentralized strategy under the centralized training and decentralized execution paradigm, enabling the intelligent agent to quickly adapt and adjust the strategy in a dynamic environment, thereby enhancing the scalability and robustness of the multi-agent system.

[0059] Based on the above-mentioned reinforcement learning-based communication collaboration method, this embodiment provides a reinforcement learning-based communication collaboration system, which includes multiple intelligent agents, each of which is deployed with a reinforcement learning-based communication collaboration strategy. The intelligent agents perform communication collaboration in a combat scenario based on the communication collaboration strategy. The communication collaboration strategy determination process specifically includes: Each intelligent agent obtains its own local observation information and determines the hidden state and action at the current moment based on the local observation information; each intelligent agent obtains the neighboring intelligent agents with which it has a communication relationship, determines the hard interaction value between the intelligent agent and the neighboring intelligent agent based on the hidden state, and determines the interaction strength between the intelligent agent and the neighboring intelligent agent based on the local observation information; each intelligent agent determines the received information contribution of each intelligent agent based on the hard interaction value, interaction strength and the hidden state of the neighboring intelligent agent; the Q value of each intelligent agent is constructed based on the local observation information, the action and the received information contribution of each intelligent agent, and reinforcement learning is trained based on the Q values ​​of all intelligent agents to output the communication cooperation strategy in the combat scenario.

[0060] Furthermore, the process of determining the communication collaboration strategy can be stored as one or more programs in a computer-readable storage medium, and the one or more programs can be executed by one or more processors to implement the steps in the process of determining the communication collaboration strategy as described in the above embodiment to determine the communication collaboration strategy.

[0061] Of course, the process of determining the communication cooperation strategy described above can also be implemented by a terminal device, which may include at least one processor 20, a display screen 21, and a memory 22. It may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logic instructions in the memory 22 to execute the steps of the process of determining the communication cooperation strategy described above.

[0062] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0063] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0064] The memory 22 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal device. In addition, the memory 22 may include high-speed random access memory and non-volatile memory. For example, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

[0065] In addition, the specific process of loading and executing the multiple instructions in the storage medium and the processor in the terminal device has been described in detail in the above method and will not be described here one by one.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A communication collaboration method based on reinforcement learning, characterized in that: The communication collaboration method based on reinforcement learning specifically includes: Each agent obtains its own local observation information and determines its current hidden state and action based on the local observation information; Each agent obtains the neighboring agents with which it has a communication relationship, determines the hard interaction value between the agent and the neighboring agents based on the hidden state, and determines the interaction strength between the agent and the neighboring agents based on the local observation information; Each agent determines its contribution to the received information based on its hard interaction value, interaction strength, and the hidden states of neighboring agents; Constructing a Q-value for each agent based on the local observation information, the action, and the received information contribution of each agent, and training reinforcement learning based on the Q-values ​​of all agents to output a communication cooperation strategy in a combat scenario; Each intelligent agent performs communication cooperation based on the communication cooperation strategy.

2. The communication collaboration method based on reinforcement learning according to claim 1, characterized in that: The determining of the hidden state and action at the current moment based on the local observation information specifically includes: Determine hidden state and output data through the mamba model based on the local observation information; The posterior distribution of the output data is sampled to obtain an uncertainty vector, and an action at a current moment is determined based on the output data and the uncertainty vector.

3. The communication collaboration method based on reinforcement learning according to claim 1, characterized in that: The step of each agent acquiring a neighboring agent with which it has a communication relationship specifically includes: Each agent determines a communication subgraph centered on each agent based on its own local observation information; Each agent selects neighboring agents with which it has a communication relationship based on its corresponding communication subgraph.

4. The communication collaboration method based on reinforcement learning according to claim 1, characterized in that: The method of determining the hard interaction value between the agent and the neighboring agent based on the hidden state specifically includes: Feed the hidden state of the agent and the hidden states of its neighboring agents into the hard attention model; The hard interaction value between the agent and the neighboring agent is input through the hard attention model, wherein the hard interaction value is used to reflect whether the agent and the neighboring agent need to interact.

5. The communication collaboration method based on reinforcement learning according to claim 1, characterized in that: Determining the interaction intensity between the agent and the neighboring agents based on local observation information specifically includes: Obtaining a first embedding representation corresponding to the local observation information of the agent and a second embedding representation corresponding to the local observation information of the neighboring agent; Based on the first embedding representation and the second embedding representation, the interaction strength between the agent and the neighboring agents is determined through a bidirectional gated recurrent model.

6. The communication collaboration method based on reinforcement learning according to claim 1, characterized in that: The determination of each agent's contribution to received information based on the hard interaction value, interaction strength, and hidden state of neighboring agents specifically includes: Each agent uses the product of the hard interaction value and the interaction strength as the weight, and weights the hidden states of each neighboring agent to obtain the agent's contribution to received information.

7. The communication collaboration method based on reinforcement learning according to claim 1, characterized in that: Constructing the Q value of each agent based on the local observation information, the action, and the received information contribution of each agent specifically includes: Extract features from the local observation information and actions of each agent to obtain local observation features and action features; The local observation features, action features and received information contribution of each agent are input into the multi-layer perceptron, and the Q value of each agent is output through the multi-layer perceptron.

8. The communication collaboration method based on reinforcement learning according to claim 1, characterized in that: The local observation information includes position information, health value, shield value, field of view, surrounding path information, terrain height information, enemy unit information and distance within the field of view, and friendly unit information and distance within the field of view.

9. The communication collaboration method based on reinforcement learning according to claim 1, characterized in that: The action space of the combat scenario includes moving direction, attack unit identification, stop and no operation. The dead agent can only take no operation, and the surviving agent takes other actions except no operation.

10. A communication collaboration system based on reinforcement learning, characterized in that: The reinforcement learning-based communication cooperation system includes a plurality of intelligent agents, each of which is deployed with a reinforcement learning-based communication cooperation strategy and performs communication cooperation in a combat scenario based on the communication cooperation strategy. The communication cooperation strategy determination process specifically includes: Each agent obtains its own local observation information and determines its current hidden state and action based on the local observation information; Each agent obtains the neighboring agents with which it has a communication relationship, determines the hard interaction value between the agent and the neighboring agents based on the hidden state, and determines the interaction strength between the agent and the neighboring agents based on the local observation information; Each agent determines its contribution to the received information based on its hard interaction value, interaction strength, and the hidden states of neighboring agents; The Q value of each intelligent agent is constructed based on the local observation information, the action and the received information contribution of each intelligent agent, and reinforcement learning is trained based on the Q values ​​of all intelligent agents to output the communication cooperation strategy in the combat scenario.

Images