A communication cooperation method and system based on reinforcement learning

Abstract

The application relates to the technical field of multi-agent, in particular to a communication cooperation method and system based on reinforcement learning. The method comprises the following steps: each agent acquires local observation information of itself, determines a hidden state and an action based on the local observation information; a hard interaction value between the agent and a neighbor agent is determined based on the hidden state, and an interaction intensity between the agent and the neighbor agent is determined based on the local observation information; a received information contribution is determined based on the hard interaction value, the interaction intensity and the hidden state; a Q value of the agent is constructed based on the local observation information, the action and the received information contribution of each agent, and a reinforcement learning is trained based on the Q values of all the agents to output a communication cooperation strategy; and the communication cooperation strategy in a combat scene is determined based on a reinforcement learning model. The Q value is constructed based on the local observation information, the action and the received information contribution, efficient and fair credit distribution between the agents can be realized, and the efficiency of the communication cooperation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and multi-agent technology, in particular to a communication cooperation method and system based on reinforcement learning. BACKGROUND

[0002] In recent years, multi-agent systems (MAS) play a crucial role in modern artificial intelligence and have wide applications in fields such as unmanned swarm, intelligent manufacturing, and traffic control. A multi-agent system is composed of multiple autonomous agents, and multiple agents can solve complex tasks beyond the capabilities of a single agent through interaction, communication, and cooperation. The cooperation and competition between agents are crucial for achieving common goals. However, the design of effective communication and cooperation strategies between agents faces many challenges, especially in dynamic and uncertain environments. Traditional centralized control methods have obvious limitations in handling large-scale multi-agent systems, as they are difficult to scale and have high communication bandwidth requirements. On the other hand, distributed control methods have advantages in scalability but face challenges in coordination and credit allocation between agents.

[0003] Multi-agent reinforcement learning (MARL) learns optimal strategies through interaction with the environment and can handle diverse tasks. However, existing MARLs face challenges in large-scale combat scenarios due to the uncertainty caused by environmental changes, leading to unreasonable allocation of contributions to individual agents and low cooperation efficiency between multi-agents.

[0004] Therefore, the existing technology needs to be improved and improved. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a communication cooperation method and system based on reinforcement learning to address the shortcomings of the prior art.

[0006] To solve the above technical problems, the first aspect of the present application provides a communication cooperation method based on reinforcement learning, which specifically includes:

[0007] Each agent obtains its own local observation information and determines the hidden state and action at the current time based on the local observation information;

[0008] Each agent obtains neighbor agents with a communication relationship with it, determines the hard interaction value between the agent and the neighbor agent based on the hidden state, and determines the interaction strength between the agent and the neighbor agent based on the local observation information;

[0009] Each agent determines a received information contribution of each agent based on a hard interaction value, an interaction strength, and a hidden state of a neighbor agent;

[0010] A Q value of each agent is constructed based on the local observation information, the action, and the received information contribution of each agent, and reinforcement learning is trained based on the Q values of all agents to output a communication cooperation strategy in a combat scenario;

[0011] Each agent performs communication cooperation based on the communication cooperation strategy.

[0012] The communication cooperation method based on reinforcement learning, wherein the determination of the hidden state and the action at the current time based on the local observation information specifically comprises:

[0013] The hidden state and output data are determined based on the local observation information through a mamba model;

[0014] The posterior distribution of the output data is sampled to obtain an uncertainty vector, and the action at the current time is determined based on the output data and the uncertainty vector.

[0015] The communication cooperation method based on reinforcement learning, wherein the acquisition of each agent of a neighbor agent having a communication relationship with the agent specifically comprises:

[0016] Each agent determines a communication subgraph centered on each agent based on its own local observation information;

[0017] Each agent selects a neighbor agent having a communication relationship with itself based on the respective communication subgraph.

[0018] The communication cooperation method based on reinforcement learning, wherein the determination of the hard interaction value between the agent and the neighbor agent based on the hidden state specifically comprises:

[0019] The hidden state of the agent and the hidden state of the neighbor agent are input into a hard attention model;

[0020] The hard interaction value between the agent and the neighbor agent is input through the hard attention model, wherein the hard interaction value is used to reflect whether the agent and the neighbor agent need to interact.

[0021] The communication cooperation method based on reinforcement learning, wherein the determination of the interaction strength between the agent and the neighbor agent based on the local observation information specifically comprises:

[0022] A first embedding representation corresponding to the local observation information of the agent and a second embedding representation corresponding to the local observation information of the neighbor agent are acquired;

[0023] determine the interaction strength between the agent and the neighbor agent based on the first embedding representation and the second embedding representation through a bidirectional gated recurrent model.

[0024] The communication cooperation method based on reinforcement learning, wherein the determination of the received information contribution of each agent based on the hard interaction value, the interaction strength and the hidden state of the neighbor agent specifically comprises:

[0025] Each agent weights the hidden state of each neighbor agent by the product of the hard interaction value and the interaction strength to obtain the received information contribution of the agent.

[0026] The communication cooperation method based on reinforcement learning, wherein the construction of the Q value of each agent based on the local observation information, the action and the received information contribution of each agent specifically comprises:

[0027] The local observation information and the action of each agent are subjected to feature extraction to obtain local observation features and action features;

[0028] The local observation features, the action features and the received information contribution of each agent are input into a multi-layer perception machine, and the Q value of each agent is output through the multi-layer perception machine.

[0029] The communication cooperation method based on reinforcement learning, wherein the local observation information comprises position information, life value, shield value, field of view range, surrounding path information, terrain height information, enemy unit information and distance within the field of view range and friendly unit information and distance within the field of view range.

[0030] The communication cooperation method based on reinforcement learning, wherein the action space of the combat scenario comprises a moving direction, an attack unit identifier, a stop and a null operation, a dead agent can only take the null operation, and a surviving agent can take other actions except the null operation.

[0031] The second aspect of the application provides a communication cooperation system based on reinforcement learning, wherein the communication cooperation system based on reinforcement learning comprises a plurality of agents, each agent is deployed with a communication cooperation strategy based on reinforcement learning, and performs communication cooperation in a combat scenario based on the communication cooperation strategy, wherein the determination process of the communication cooperation strategy specifically comprises:

[0032] Each agent obtains its own local observation information, and determines a hidden state and an action at a current time based on the local observation information;

[0033] Each agent acquires neighbor agents having a communication relationship with the agent, determines a hard interaction value between the agent and the neighbor agents based on the hidden state, and determines an interaction strength between the agent and the neighbor agents based on the local observation information;

[0034] Each agent determines a received information contribution of each agent based on the hard interaction value, the interaction strength, and the hidden state of the neighbor agents;

[0035] A Q value of each agent is constructed based on the local observation information, the action, and the received information contribution of each agent, and reinforcement learning is trained based on the Q values of all agents to output a communication cooperation strategy in a combat scenario.

[0036] Advantages: Compared with the prior art, the application provides a communication cooperation method and system based on reinforcement learning, which includes that each agent acquires its own local observation information, and determines a hidden state and an action at a current time based on the local observation information; each agent acquires neighbor agents having a communication relationship with the agent, determines a hard interaction value between the agent and the neighbor agents based on the hidden state, and determines an interaction strength between the agent and the neighbor agents based on the local observation information; each agent determines a received information contribution of each agent based on the hard interaction value, the interaction strength, and the hidden state of the neighbor agents; a Q value of each agent is constructed based on the local observation information, the action, and the received information contribution of each agent, and reinforcement learning is trained based on the Q values of all agents to output a communication cooperation strategy in a combat scenario; and the communication cooperation strategy in the combat scenario is determined based on the trained reinforcement learning model. In the application, each agent constructs a Q value of each agent based on its own local observation information, the action, and the received information contribution, which can realize efficient and fair credit distribution between agents, and improve the efficiency of communication cooperation. At the same time, the application optimizes the decentralized strategy under the centralized training and decentralized execution paradigm, so that the agent can quickly adapt and adjust the strategy in a dynamic environment, and the scalability and robustness of the multi-agent system are enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 A flowchart of the communication cooperation method based on reinforcement learning provided by the embodiments of the application.

[0039] Figure 2A principle flowchart of a communication cooperation method based on reinforcement learning provided by an embodiment of the present application. DETAILED DESCRIPTION

[0040] The embodiment of the present application provides a communication cooperation method and system based on reinforcement learning. In order to make the purpose, technical scheme and effects of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0041] It can be understood by those skilled in the art that, unless specifically stated, the singular forms "a", "an" and "the" used herein include plural forms. It should be further understood that the use of the term "include" in the specification of the present application means that the stated features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0042] It can be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood as having meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0043] It should be understood that the sequence numbers and sizes of the steps in the embodiments do not mean the order of execution, and the execution order of the processes is determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0044] The content of the application is further described below by describing the embodiments in combination with the drawings.

[0045] The embodiment provides a communication cooperation method based on reinforcement learning, which can be applied to a combat scene, the combat scene containing various maps, units and tasks, in which multiple agents compete with each other or with built-in game AI to form a game round in which two competing teams exist. A game round ends when all units of any team die or reach a pre-designated time limit, and the goal of each game round is to maximize the winning rate of the learning strategy. Each unit in each game round is controlled by an independent agent, and the skill of each agent is improved through training to solve challenging combat scenes. For example, the combat scene is a multi-agent combat scene implemented by a game engine of StarCraft II.

[0046] Each agent in the combat scene participates in a Markov game , which is represented by a tuple . Wherein, represents a global state set, which can include map center coordinates, life values, skill cooling times, position information, field of view ranges, available actions and last executed actions of all units; represents a joint action space, and the action space of an agent is a discrete action set, which can include moving directions (such as east, south, west and north moving directions), attack unit identifiers, stop and null operations, and a dead agent can only take a null operation, and a surviving agent can take other actions except the null operation. Each agent at time performs an action based on local observation information , and the local observation of the agent is limited to a limited range centered on the agent, including position information, life value, shield value, field of view range, surrounding path information, terrain height information, enemy unit information and distance within the field of view range, and friendly unit information and distance within the field of view range. represents a transition probability function, representing the probability of transitioning from state to state ; represents a shared reward, which is shared by the entire multi-agent system in a complete cooperation setting, and a sparse reward setting is selected to increase the difficulty of the task, and the environment only returns a reward of +1 (win) and a reward of -1 (lose); represents a discount factor, which balances the importance of future rewards and immediate rewards.

[0047] Specifically, as shown in Figure 1 , the communication cooperation method based on reinforcement learning provided by the embodiment specifically comprises:

[0048] S10, each agent obtains its own local observation information, and determines the hidden state at the current time and the action based on the local observation information.

[0049] Specifically, each agent is included in a multi-agent system, each agent in the multi-agent system corresponds to a combat unit in a combat scene, and the goal of each agent is to learn a policy network , the policy network maximizes the expected discounted reward obtained in the interaction with the environment and other agents, so that the agent optimizes its own policy network while considering the behavior of other agents.

[0050] Further, the policy network carried by each agent can adopt a mamba model, which can be represented as a description of the system state evolution equation and the output equation, wherein the state update equation and the output equation are respectively:

[0051] ,

[0052] ,

[0053] wherein, represents the hidden state at time , represents the state transition matrix, represents the input matrix, represents the hidden state at time , represents the input data at time , represents the output data at time , represents the conversion matrix.

[0054] Based on this, the agent can determine the hidden state at the current time and the action (i.e. the output at the current time) through the mamba model by obtaining its own local observation information and the previous hidden state. In addition, in order to deal with the randomness and dynamics of the multi-agent system, the latent hidden state is constantly updated according to the historical input sequence in the mamba model in the embodiment of the application, and the posterior distribution of the hidden state is used to represent the uncertainty of the environment change, so as to avoid the influence of the uncertainty caused by the environment change on the multi-agent learning.

[0055] Exemplarily, the determination of the hidden state at the current time and the action based on the local observation information specifically includes:

[0056] determining the hidden state and the output data through the mamba model based on the local observation information;

[0057] sample a posterior distribution of the output data to obtain an uncertainty vector, and determine an action at the current time based on the output data and the uncertainty vector.

[0058] Specifically, as shown in the figure, the previous hidden state is a hidden state at a previous time, and the mamba model stores the previous hidden state. Thus, after obtaining the local observation information, the mamba model inputs the local observation information, performs linear mapping on the local observation information to obtain a local observation vector, passes the local observation vector through a convolution layer to obtain an input vector, inputs the input vector into a selective state space model (SSM), determines a hidden state at the current time and initial output data through the selective state space model, fuses the initial output data and the local observation vector to obtain fused output data, and performs linear mapping on the fused output data to obtain output data. Figure 2

[0059] After obtaining the output data, a posterior distribution of the output data is sampled to obtain an uncertainty vector, and the uncertainty vector is combined with the output data to obtain an action at the current time. The uncertainty vector and the action at the current time can be represented as follows:

[0060] ,

[0061] ,

[0062] wherein, represents an uncertainty vector at time t, represents a posterior distribution, represents output data at time t, represents an action of an i-th agent at time t, represents a joint operation.

[0063] ​​​​​In this embodiment, the policy network in the reinforcement learning model adopts the core idea of ​​the Mamba model, which can efficiently capture complex dependencies in sequential data. Simultaneously, a selective state-space model and an uncertainty vector are introduced into the Mamba model. The selective state-space model allows the agent to dynamically adjust its state representation during the decision-making process to adapt to constantly changing environments and task requirements. The uncertainty vector is determined using posterior sampling techniques, reflecting the uncertainty of environmental changes and thus handling the randomness and dynamism of multi-agent systems. This uncertainty vector is used to enhance the output of the policy network. The two-tooth policy network's adaptability to non-stationarity helps the agent make more robust decisions.

[0064] S20. Each agent acquires neighboring agents with whom it has a communication relationship, determines the hard interaction value between the agent and its neighboring agents based on the hidden state, and determines the interaction strength between the agent and its neighboring agents based on local observation information.

[0065] Specifically, a neighboring agent is an agent located within the local interaction range of an agent and having a communication relationship with that agent. In other words, the multi-agent system utilizes game theory principles to construct a cooperative model among agents and achieves effective communication between agents through a graph neural network (GNN). Each agent determines its corresponding communication subgraph based on its own local observation information and identifies its neighboring agents with whom it has a communication relationship based on this subgraph. Therefore, the process of each agent acquiring its neighboring agents with whom it has a communication relationship specifically includes:

[0066] Each agent determines a communication subgraph centered on itself based on its own local observation information;

[0067] Each agent selects neighboring agents with whom it has a communication relationship based on its corresponding communication subgraph.

[0068] Specifically, the agent obtains local observation information including position information, health value, shield value, field of view range, surrounding path information, terrain height information, enemy unit information and distance within the field of view range, and friendly unit information and distance within the field of view range. Based on the local observation information, the agent can determine the friendly unit information within the field of view range, and then establish a communication subgraph centered on the agent based on the obtained friendly unit information, and then input the communication subgraph into the graph neural network to determine the friendly units that have effective communication with the agent, and then select the friendly units that can send information to the agent as neighbor agents from the friendly units that have effective communication, that is, the neighbor agent can receive the local observation information and hidden state sent by the neighbor agent, so that based on the local observation information and hidden state sent by the neighbor agent, the agent can determine the hard interaction value and interaction strength between the agent and the neighbor agent based on the local observation information and hidden state of the agent itself and the local observation information and hidden state of the neighbor agent.

[0069] After the agent obtains its corresponding neighbor agent, the agent receives interaction information interacted by the neighbor agent, wherein the interaction information can include the local observation information of the neighbor agent and the hidden state at the current time. That is, after the agent obtains the neighbor agent, the agent can send an interaction request to the neighbor agent, so that the neighbor agent sends the interaction information to the agent based on the interaction request, so that the agent can determine the hard interaction value and interaction strength between the agent and the neighbor agent according to the interaction information sent by the neighbor agent and its own local observation information and hidden state at the current time.

[0070] The hard interaction value between the agent and the neighbor agent is used to reflect whether the agent and the neighbor agent need to interact. The hard interaction value is determined based on the hidden state of the agent and the hidden state of the neighbor agent. Specifically, the determination of the hard interaction value between the agent and the neighbor agent based on the hidden state specifically includes:

[0071] The hidden state of the agent and the hidden state of the neighbor agent are input into a hard attention model, and the hard interaction value between the agent and the neighbor agent is input through the hard attention model, wherein the hard interaction value is used to reflect whether the agent and the neighbor agent need to interact.

[0072] Specifically, the hard attention model is used for interactive learning of the hidden state of the agent and the hidden state of the neighbor agent to determine the attention value of the agent and the neighbor agent, and the attention value is taken as the hard interaction value, wherein the hard interaction value is a binary value, that is, the hard interaction value takes 0 or 1, 0 represents that the agent and the neighbor agent do not need to interact, and 1 represents that the agent and the neighbor agent need to interact.

[0073] After the hidden state of the agent and the hidden state of the neighbor agent are input into the hard attention model, the hard attention model will combine the hidden state of the agent and the hidden state of the neighbor agent to form a state vector, and construct a query vector, a value vector and a key vector based on the state vector, and then determine the hard interaction value between the agent and the neighbor agent through the attention mechanism based on the query vector, the value vector and the key vector. Based on this, the process of obtaining the hard interaction value can be represented as:

[0074] ,

[0075] wherein, represents the hard interaction value between the agent and the neighbor agent of the agent , represents the hard attention model, represents the hidden state of the agent , represents the hidden state of the neighbor agent .

[0076] Of course, in actual application, other ways can also be used to obtain the hard interaction value between the agent and the neighbor agent, for example, the distance between the agent and the neighbor agent can be used.

[0077] Further, the interaction intensity is used to reflect the interaction closeness between the agent and the neighbor agent, and through the interaction intensity, the contribution value of the neighbor agent can be more accurate. The interaction intensity is determined based on local observation information, and the determination process can include:

[0078] Obtaining a first embedding representation corresponding to the local observation information of the agent and a second embedding representation corresponding to the local observation information of the neighbor agent;

[0079] Based on the first embedding representation and the second embedding representation, determining the interaction intensity between the agent and the neighbor agent through a bidirectional gated recurrent model.

[0080] Specifically, the first embedding representation and the second embedding representation are obtained by mapping the local observation information of the agent and the local observation information of the neighbor agent to the same vector space, for example, the first embedding representation and the second embedding representation can be determined by encoding the local observation information of the agent and the local observation information of the neighbor agent through a preset encoder. After obtaining the first embedding representation and the second embedding representation, input the first embedding representation and the second embedding representation into the bidirectional gated recurrent model, and determine the interaction intensity between the agent and the neighbor agent based on the bidirectional gated recurrent model through the bidirectional gated recurrent model, wherein the calculation formula of the interaction intensity can be represented as:

[0081] ,

[0082] wherein, represents an agent and the interaction intensity between the neighbor agent of the agent , represents a bidirectional gated recurrent model centered on the neighbor agent , represents the corresponding first embedding representation of the agent , represents the corresponding second embedding representation of the neighbor agent , and each represent a weight matrix.

[0083] The embodiments of the present application simplify the multi-agent learning process by abstracting the game, introduce a graph neural network and an attention mechanism to learn the interaction relationship between agents, and take the local observation information of the agent as the communication content, thereby improving the fairness and efficiency of individual contribution calculation and realizing a better multi-agent cooperation strategy.

[0084] S30, each agent determines the received information contribution of each agent based on the hard interaction value, the interaction intensity, and the hidden state of the neighbor agent.

[0085] Specifically, the received information contribution is used to reflect the information contribution received by the agent from other agents, and through the received information contribution, the agent can learn the information contribution of other agents to itself in a complex multi-agent system, and thus the agent can realize effective cooperation in a complex multi-agent system. Wherein, the received information contribution can be obtained by weighting the hidden state of each neighbor agent with the hard interaction value and the interaction intensity as the weighting coefficients, that is, each agent determines the received information contribution of each agent based on the hard interaction value, the interaction intensity, and the hidden state of the neighbor agent. Specifically, each agent weights the hidden state of each neighbor agent with the product of the hard interaction value and the interaction intensity as the weight to obtain the received information contribution of the agent.

[0086] Based on this, the calculation formula of the received information contribution can be:

[0087] ,

[0088] wherein, represents the corresponding received information contribution of the agent , represents the interaction intensity between the agent and the neighbor agent of the agent , representing the agent and the neighbor agent of the agent hard interaction value between the agent and the neighbor agent, representing the hidden state of the neighbor agent .

[0089] Of course, in actual applications, other ways can also be adopted to determine the received information contribution, such as first calculating the individual received information contribution of the agent with respect to each neighbor agent , then configuring a weighting coefficient for the individual received information contribution based on the local observation information of the neighbor agent, and finally weighting each individual received information contribution based on the weighting coefficient to obtain the received information contribution of the agent.

[0090] S40, constructing the Q value of each agent based on the local observation information, the action and the received information contribution of each agent, and training reinforcement learning based on the Q values of all agents to output a communication cooperation strategy in the combat scene.

[0091] Specifically, the Q value of the agent is a team contribution indicator of the agent to the team in which the agent is located, and the potential impact of different actions can be evaluated through the Q value of the agent, and an action that can maximize the overall performance of the team is selected. Among them, the Q value of the agent is determined based on the local observation information, the action and the received information contribution, and the Q value is constructed by using the received information contribution, so that the agent can effectively assist other agents.

[0092] For example, the construction of the Q value of each agent based on the local observation information, the action and the received information contribution of each agent specifically includes:

[0093] feature extraction is performed on the local observation information and the action of each agent to obtain local observation features and action features;

[0094] The local observation features, action features and received information contribution of each agent are input into a multi-layer perception machine, and the Q value of each agent is output through the multi-layer perception machine.

[0095] Specifically, the local observation features are obtained by mapping the local observation information to a feature space, and the action features are obtained by mapping the action to a feature space. That is, when determining the Q value of the agent, the local observation features of the local observation information and the action features of the action are first extracted, and then the Q value of each agent is determined based on the local observation features, the action features and the received information contribution, wherein the local observation features and the action features can be extracted by a pre-set feature extraction model, such as a feature extraction module in VGG.

[0096] Based on this, the Q-value function corresponding to the agent's Q-value can be expressed as:

[0097] ,

[0098] in, Represents intelligent agents Q-value function, This indicates a preset feature extraction model. This represents a multilayer perceptron. Represents intelligent agents Local observation information, Represents intelligent agents The action, Represents intelligent agents The corresponding contribution of received information.

[0099] Furthermore, after obtaining the Q-value of each agent, a Q-value loss term is constructed based on the Q-values ​​of all agents, and the policy network of each agent is updated based on this Q-value loss term for reinforcement learning. This embodiment of the application optimizes the distributed policy by adopting a centralized training and distributed execution paradigm, enabling agents to quickly adapt and adjust their policies in dynamic environments, thereby enhancing the scalability and robustness of the system.

[0100] S50: Each agent communicates and collaborates based on a communication and collaboration strategy.

[0101] Specifically, after acquiring the communication cooperation strategy, each agent uses the communication cooperation strategy as its corresponding policy network and conducts communication cooperation in the combat scenario based on the communication cooperation strategy.

[0102] To sum up, the embodiment provides a communication cooperation method based on reinforcement learning, which comprises: each agent acquires its own local observation information, and determines a hidden state and an action at a current time based on the local observation information; each agent acquires neighbor agents having a communication relationship with the agent, determines a hard interaction value between the agent and the neighbor agents based on the hidden state, and determines an interaction intensity between the agent and the neighbor agents based on the local observation information; each agent determines a received information contribution of each agent based on the hard interaction value, the interaction intensity, and the hidden state of the neighbor agents; a Q value of each agent is constructed based on the local observation information, the action, and the received information contribution of each agent, and reinforcement learning is trained based on the Q values of all agents to output a communication cooperation strategy in a combat scenario; and the communication cooperation strategy in the combat scenario is determined based on the trained reinforcement learning model. In the application, each agent constructs a Q value of each agent based on its own local observation information, the action, and the received information contribution, which can realize efficient and fair credit allocation between agents, and improve the efficiency of communication cooperation. Meanwhile, in the centralized training and decentralized execution paradigm, the decentralized strategy is optimized, so that the agent can quickly adapt and adjust the strategy in a dynamic environment, and the scalability and robustness of the multi-agent system are enhanced.

[0103] Based on the communication cooperation method based on reinforcement learning, the embodiment provides a communication cooperation system based on reinforcement learning, which comprises a plurality of agents, each agent is deployed with a communication cooperation strategy based on reinforcement learning, and performs communication cooperation in a combat scenario based on the communication cooperation strategy, wherein the determination process of the communication cooperation strategy comprises:

[0104] Each agent acquires its own local observation information, and determines a hidden state and an action at a current time based on the local observation information; each agent acquires neighbor agents having a communication relationship with the agent, determines a hard interaction value between the agent and the neighbor agents based on the hidden state, and determines an interaction intensity between the agent and the neighbor agents based on the local observation information; each agent determines a received information contribution of each agent based on the hard interaction value, the interaction intensity, and the hidden state of the neighbor agents; a Q value of each agent is constructed based on the local observation information, the action, and the received information contribution of each agent, and reinforcement learning is trained based on the Q values of all agents to output a communication cooperation strategy in a combat scenario.

[0105] Further, the determination process of the communication cooperation strategy can be stored as one or more programs in a computer-readable storage medium, and when the one or more programs can be executed by one or more processors to implement the steps in the determination process of the communication cooperation strategy as described in the above embodiment, the communication cooperation strategy can be determined.

[0106] Of course, the determination process of the communication cooperation strategy based on the above can also be implemented by a terminal device, which can include at least one processor 20, a display screen 21, and a memory 22, and can further include a communications interface 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22 and the communications interface 23 can complete the communication between each other through the bus 24. The display screen 21 is configured to display the user guide interface preset in the initial setting mode. The communications interface 23 can transmit information. The processor 20 can call the logical instructions in the memory 22 to execute the steps of the determination process of the communication cooperation strategy described above.

[0107] In addition, the logical instructions in the memory 22 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0108] The memory 22 as a kind of computer readable storage medium can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the method in the embodiments of the present disclosure. The processor 20 executes the functions of the application and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implements the method in the above embodiments.

[0109] The memory 22 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory. For example, a variety of media that can store program codes, such as U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc. It can also be a transient storage medium.

[0110] In addition, the specific process of the memory medium and the plurality of instruction processors in the terminal device loading and executing has been described in detail in the above method, and here will not be stated one by one.

[0111] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for communication cooperation based on reinforcement learning, characterized in that, The communication cooperation method based on reinforcement learning specifically comprises: Each agent obtains its own local observation information, and determines a hidden state and an action at a current time based on the local observation information; Each agent obtains neighbor agents having a communication relationship with the agent, determines a hard interaction value between the agent and the neighbor agents based on the hidden state, and determines an interaction strength between the agent and the neighbor agents based on the local observation information; Each agent determines a received information contribution of each agent based on the hard interaction value, the interaction strength, and the hidden state of the neighbor agents; Q values of each agent are constructed based on the local observation information, the action, and the received information contribution of each agent, and reinforcement learning is trained based on the Q values of all agents to output a communication cooperation strategy in a combat scenario; Each agent performs communication cooperation based on the communication cooperation strategy; The determination of the hidden state and the action at the current time based on the local observation information specifically comprises: A hidden state and output data are determined based on the local observation information through a mamba model; A posterior distribution of the output data is sampled to obtain an uncertainty vector, and an action at the current time is determined based on the output data and the uncertainty vector.

2. The method of claim 1, wherein, The obtaining of the neighbor agents having the communication relationship with the agent by each agent specifically comprises: Each agent determines a communication subgraph centered on each agent based on its own local observation information; Each agent selects neighbor agents having a communication relationship with the agent based on the respective communication subgraph. 3.The method of claim 1, wherein, The determination of the hard interaction value between the agent and the neighbor agents based on the hidden state specifically comprises: The hidden state of the agent and the hidden state of the neighbor agents are input into a hard attention model; A hard interaction value between the agent and the neighbor agents is input through the hard attention model, wherein the hard interaction value is used to reflect whether the agent and the neighbor agents need to interact. 4.The method of claim 1, wherein, The determination of the interaction strength between the agent and the neighbor agents based on the local observation information specifically comprises: A first embedding representation corresponding to the local observation information of the agent and a second embedding representation corresponding to the local observation information of the neighbor agents are obtained; The interaction strength between the agent and the neighbor agents is determined based on the first embedding representation and the second embedding representation through a bidirectional gated recurrent model. 5.The method of claim 1, wherein, The determination of the received information contribution of each agent based on the hard interaction value, the interaction strength, and the hidden state of the neighbor agents specifically comprises: The hidden state of each neighbor agent is weighted by taking the product of the hard interaction value and the interaction strength as a weight to obtain the received information contribution of the agent. 6.The method of claim 1, wherein, The construction of the Q values of each agent based on the local observation information, the action, and the received information contribution of each agent specifically comprises: Local observation features and action features are obtained by performing feature extraction on the local observation information and the action of each agent; The local observation features, the action features, and the received information contribution of each agent are input into a multilayer perceptron, and the Q values of each agent are output through the multilayer perceptron. 7.The method of claim 1, wherein, The local observation information includes position information, life value, shield value, field of view range, surrounding path information, terrain height information, enemy unit information and distance within the field of view range, and friendly unit information and distance within the field of view range. 8.The method of claim 1, wherein, The action space of the combat scene includes a moving direction, an attack unit identifier, a stop and an empty operation, a dead intelligent agent can only take the empty operation, and a surviving intelligent agent takes other actions except the empty operation. 9.A communication cooperation system based on reinforcement learning, characterized in that, The communication cooperation system based on the reinforcement learning includes a plurality of intelligent agents, each of which is deployed with a communication cooperation strategy based on reinforcement learning, and performs communication cooperation in the combat scene based on the communication cooperation strategy, wherein the determination process of the communication cooperation strategy specifically includes: Each intelligent agent obtains its own local observation information, and determines a hidden state and an action at the current time based on the local observation information; Each intelligent agent obtains neighbor intelligent agents having a communication relationship with it, determines a hard interaction value between the intelligent agent and the neighbor intelligent agents based on the hidden state, and determines an interaction strength between the intelligent agent and the neighbor intelligent agents based on the local observation information; Each intelligent agent determines a received information contribution of each intelligent agent based on the hard interaction value, the interaction strength and the hidden state of the neighbor intelligent agents; Q values of each intelligent agent are constructed based on the local observation information, the action and the received information contribution of each intelligent agent, and reinforcement learning is trained based on the Q values of all intelligent agents to output a communication cooperation strategy in the combat scene; Wherein, the determination of the hidden state and the action at the current time based on the local observation information specifically includes: The hidden state and output data are determined based on the local observation information through a mamba model; A posterior distribution of the output data is sampled to obtain an uncertainty vector, and an action at the current time is determined based on the output data and the uncertainty vector.

Images