Multi-agent collaborative confrontation reinforcement learning method based on tacit behavior pre-training

Abstract

The invention discloses a multi-agent collaborative confrontation reinforcement learning method based on tacit behavior pre-training, and the method comprises the steps: a distributed tacit pre-training stage: constructing a situation mechanism and a tacit mechanism which represent the spatial relative position relation between agents, according to the situation mechanism and the tacit mechanism, a tacit reward function is constructed for each agent, and tacit pre-training is performed on each agent based on the tacit reward function of each agent; and a centralized confrontation training stage: performing multi-agent collaborative confrontation training based on a result of tacit pre-training of the plurality of agents. According to the multi-agent collaborative confrontation reinforcement learning method based on tacit behavior pre-training, the purpose of effectively improving the learning efficiency of a multi-agent collaborative confrontation strategy can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and particularly to a multi-agent collaborative adversarial reinforcement learning method based on tacit behavior pre-training. Background Art

[0002] Multi-agent collaborative adversarial algorithms based on reinforcement learning are widely applied to practical tasks such as battlefield confrontation, game AI, and robot control, and have great research significance and value in practical applications. The multi-agent collaborative adversarial problem aims to study how a multi-agent group can defeat other agent groups through optimal collaborative decisions in a complex and changing environment. Since the deduction of the multi-agent collaborative adversarial environment is affected by the action executions of all agents (not under the control of one's own side), a huge policy space is formed, and the lack of algorithmic expressiveness will lead to the problem of difficult convergence. To address the problem of a large policy space in multi-agent collaborative adversarial tasks, existing algorithms have promoted efficient cooperation among multi-agents by introducing insights from how humans solve problems in the real world. However, most of these methods guide policy updates through team rewards and lack the introduction of other common-sense information that is beneficial to collaborative adversarial tasks, restricting the optimized learning of multi-agent collaborative adversarial strategies.

[0003] To address the problem of "large policy space and limited guiding information", the GoMARL method introduces the idea of grouping. By automatically grouping multiple agents, the learning process of reinforcement learning is refined to alleviate the problem of limited learning efficiency of multi-agents caused by a large policy space. Among them, the GoMARL method uses sub-grouping as a bridge to establish connections between some agents and encourages collaborative cooperation among them, thereby improving the learning efficiency of the overall team. This method decomposes the joint action value function into a combination of value functions for each group, guiding multi-agents to optimize their strategies in a more refined manner. At the same time, this method introduces the idea of hierarchical control, enabling agents in the same group to learn similar strategies and making the strategies between different groups diverse. However, in this method, after grouping the multi-agents, the algorithm has limited consideration of the correlation between agents within the group, and each agent lacks more targeted guidance. It is suitable for scenarios where multiple agents are close to each other and can perceive each other, such as having good applicability in StarCraft and robot soccer; while in scenarios where a single agent cannot obtain its global relative coordinates and cannot perceive each other, the learning efficiency of this algorithm is relatively low, and its advantages are not significant enough.

[0004] To address the problem of "large policy space and limited guiding information", the RODE method introduces the idea of roles. By allocating corresponding actions to different agents, it solves the problem of the overly large joint state-action search space of multiple agents, thereby alleviating the problem of limited learning efficiency of multiple agents caused by the large policy space. This method proposes a role-based task decomposition algorithm framework, which mainly consists of three parts: a prediction model, a role selection model, and a role policy model. Among them, the prediction model provides the encoded features of actions, which have clustering characteristics and can assist the role selection model in allocating roles according to the action selections of different agents; the role selection model and the role policy model make hierarchical decisions during the algorithm training execution, that is, first perform the action of allocating roles to agents, and then calculate the actual policy actions according to the roles allocated to the agents. However, this method requires strong prior knowledge and lacks universality for multi-agent collaborative confrontation problems. For different multi-agent collaborative confrontation problems, it is necessary to re-divide roles and set corresponding action spaces. This algorithm is more suitable for relatively complex application scenarios, but has low compatibility for relatively simple application scenarios and has the shortcoming of low learning efficiency.

[0005] The disclosure of the above background technical content is only used to assist in understanding the concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. Without clear evidence indicating that the above content was publicly available on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0006] To solve the above technical problems, the present invention proposes a multi-agent collaborative confrontation reinforcement learning method based on tacit behavior pre-training, which can effectively improve the learning efficiency of multi-agent collaborative confrontation strategies.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention discloses a multi-agent collaborative confrontation reinforcement learning method based on tacit behavior pre-training, including:

[0009] A distributed tacit pre-training stage, including: constructing a situation mechanism and a tacit mechanism for representing the relative spatial position relationship between agents, constructing a tacit reward function for each agent according to the situation mechanism and the tacit mechanism, and respectively performing tacit pre-training on each agent based on the tacit reward function of each agent;

[0010] A centralized confrontation training stage, including: performing multi-agent collaborative confrontation training based on the results of the tacit pre-training of multiple agents.

[0011] Preferably, the policy network for tacit pre-training takes the local observation space and action space of the agent as input variables. The first layer is the first multi-layer perceptron layer, which learns features through multi-layer non-linear transformations. The second layer is the gated recurrent unit layer, which captures time-dependent relationships through a gating mechanism and simultaneously inputs the hidden state of the agent. The third layer is the second multi-layer perceptron layer, and the output variable is the Q-value function, which is trained through the TD loss function.

[0012] Preferably, during the tacit pre-training process, the TD loss function L p is used for training:

[0013]

[0014] In the formula, respectively represent the network parameters of the policy network f p (θ p ) and the target policy network , represents all agents' tacit reward functions at the sampling time step , γ is a numerical parameter, respectively represent the outputs of the policy network f p (θ p ) and the target policy network , represents all agents' trajectory space at the sampling time step , represents all agents' action space at the sampling time step .

[0015] Preferably, the process of tacit pre-training specifically includes:

[0016] A1: Initialize the tacit value T j , the cumulative time steps NT j spent by each agent in the situation p j , the time steps NR j consumed when the value of the tacit reward function is positive. The situation p j represents each situation divided according to the situation mechanism;

[0017] A2: Calculate the situation p i to which agent i belongs at the time step t, the corresponding membership degree λ i of the situation p i , and the corresponding tacit guidance degree according to the situation mechanism and the tacit mechanism, and calculate the tacit reward function according to the membership degree value λ i and the tacit guidance degree , the situation p iRepresents the situation to which agent i belongs;

[0018] A3: Calculate the cumulative time steps NT j spent by each agent in situation p j and the time steps NR consumed when the value of the tacit reward function is positive j , and calculate the tacit value T j = NR j / NT j ;

[0019] A4: Test the tacit value T j to see if the tacit goal is reached. If the tacit goal is reached, stop the tacit pre-training.

[0020] Preferably, the situation mechanism for constructing the relative spatial position relationship between agents includes: using the global state and local observations as inputs to classify the situations of agents, where the situation of an agent represents the relative position of the agent and other agents within a preset time.

[0021] Preferably, the situations of the agents are divided into: fully dispersed situation p fd , partially dispersed situation p pd , fully aggregated situation p fa , partially aggregated situation p pa , and the corresponding membership degrees are: λ fd , λ pd , λ fa , λ pa ; among them, the fully dispersed situation p fd means that the distance between every two agents exceeds their respective sensing ranges; the partially dispersed situation p pd means that the distance between the corresponding agent and all other agents exceeds the sensing range of the corresponding agent, but the distances between the agents other than the corresponding agent and all other agents are within their respective sensing ranges; the fully aggregated situation p fa means that the distance between the corresponding agent and at least two other agents is within their respective sensing ranges, and the distances between the agents other than the corresponding agent and all other agents are within their respective sensing ranges; the partially aggregated situation p pa means that the distance between the corresponding agent and only one other agent is within their respective sensing ranges, and the distances between the agents other than the corresponding agent and all other agents are within their respective sensing ranges.

[0022] Preferably, when all multi-agents in the global state satisfy formula (1), it indicates that each agent belongs to the fully dispersed situation p fd under this global state, and the corresponding membership degree λ fd is as shown in formula (2):

[0023]

[0024] where ||·||2 represents the Euclidean norm between two positions, represents the position of agent i, represents the position of agent k. The positions of the agents are determined by the observation space and state space of the agents, d i represents the sensing distance of agent i, and the set ρ i represents the set of all agents within the sensing range of agent i. The value of the parameter β is greater than 1;

[0025] When the multi - agents in the global state do not satisfy formula (1), but satisfy formula (3), and agent i meets the conditions listed in formula (4), it indicates that agent i belongs to the partially decentralized situation p pd , and the corresponding membership degree λ pd is as shown in formula (5):

[0026]

[0027]

[0028] For agent i that does not satisfy formula (4), but the global state it is in satisfies formula (3), and the agents in this global state can sense each other. The situations that agent i belongs to can be divided into two categories: the partially aggregated situation p pa and the fully aggregated situation p fa :

[0029] When agent i satisfies formula (6), it indicates that agent i belongs to the fully aggregated situation p fa , and the corresponding membership degree λ fa is as shown in formula (7):

[0030]

[0031] where the value of the parameter α is greater than 0 and less than 1;

[0032] When agent i satisfies formula (8), it indicates that agent i belongs to the partially aggregated situation p fa , in the partially aggregated situation p fa , one of the agents is determined as the leading agent, and the other agent is determined as the following agent. Among them, the membership degree of the leading agent and the membership degree of the following agent are as shown in formula (9):

[0033]

[0034] In the formula, represents the position of the leading agent, represents the position of the follower agent, and d l represents the perception distance of the leading agent, and d f represents the perception distance of the follower agent, and the set ρ l represents the set of all agents within the perception range of the leading agent.

[0035] Preferably, constructing a tacit understanding mechanism that characterizes the spatial relative position relationship between agents includes: taking the situation corresponding to each agent, the corresponding membership degree, and the local action-observation history as inputs, and constructing the degree of tacit guidance to guide the agents to learn the corresponding tacit behaviors, where

[0036] The degree of tacit guidance of the fully decentralized situation p fd is: is:

[0037]

[0038] In the formula, represents the position of agent i at time step t, is the coordinate position of the fixed point;

[0039] The degree of tacit guidance of the partially decentralized situation p pd is: is:

[0040]

[0041] In the formula, represents the set of all agents within the perception range of agent i at time step t-1;

[0042] The degree of tacit guidance of the fully aggregated situation p fa is: is:

[0043]

[0044] The degree of tacit guidance of the leading agent and the degree of tacit guidance of the follower agent in the partially aggregated situation p fa are respectively: and are respectively:

[0045]

[0046] In the formula, represents the set of all agents within the perception range of the leading agent at time step t-1, represents the position of the leading agent at time step t, Indicates the position of the agent following time step t.

[0047] Preferably, constructing a tacit reward function for each agent according to the situation mechanism and the tacit understanding mechanism includes: obtaining the tacit reward function of each agent by multiplying the membership degree of the situation to which each agent belongs by the corresponding tacit guidance degree.

[0048] In a second aspect, the present invention discloses a storage medium in which a computer program is stored. Among them, the computer program is set to be run by a processor to execute the multi-agent collaborative confrontation reinforcement learning method based on tacit behavior pre-training described in the first aspect.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows: The multi-agent collaborative confrontation reinforcement learning method based on tacit behavior pre-training proposed by the present invention includes a distributed tacit pre-training stage and a centralized confrontation training stage. By introducing tacit behavior, the distributed tacit pre-training stage and the centralized confrontation training stage are connected. The relative spatial position relationship and its changes between agents are introduced into the algorithm, refining the reinforcement learning process, effectively improving the learning performance of the algorithm in collaborative confrontation tasks, alleviating the problem of low algorithm efficiency caused by too large a policy space, and thus achieving the purpose of effectively improving the learning efficiency of multi-agent collaborative confrontation strategies. In addition, the multi-agent collaborative confrontation reinforcement learning method proposed by the present invention has high generality. The framework proposed by it is compatible with a variety of QMIX (Q-value Mixing) algorithms, and can obtain performance exceeding the original algorithms. In addition, the proposed algorithm does not rely on communication between agents during confrontation training, and can still effectively achieve tacit behavior even without communication, enhancing its generality.

[0050] In a further solution, the present invention proposes a tacit behavior learning method composed of a two-layer structure of a situation mechanism and a tacit understanding mechanism. Among them, the situation mechanism dynamically adjusts the tacit guidance constructed by the tacit understanding mechanism by dividing the situation and determining the membership degree of the corresponding situation, promoting the efficient tacit behavior learning of agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flowchart of the multi-agent collaborative confrontation reinforcement learning method based on tacit behavior pre-training in a preferred embodiment of the present invention;

[0052] Figure 2 is a block diagram of the multi-agent reinforcement learning method based on tacit behavior pre-training;

[0053] Figure 3 is a schematic diagram of the situation mechanism;

[0054] Figure 4It is a flowchart of tacit pre-training. Specific implementation manners

[0055] The following provides a detailed description of the implementation manners of the present invention. It should be emphasized that the following description is merely exemplary and not intended to limit the scope of the present invention and its applications.

[0056] It should be noted that when an element is referred to as "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. Additionally, the connection can be for fixing purposes or for circuit / signal communication purposes.

[0057] It should be understood that the orientation or positional relationship indicated by terms such as "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for facilitating the description of the embodiments of the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be construed as a limitation of the present invention.

[0058] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "a plurality" means two or more unless otherwise specifically defined.

[0059] Regarding the problem of a large policy space in multi-agent collaborative confrontation tasks, existing algorithms promote efficient collaboration among multi-agents by introducing insights from how humans solve problems in the real world. However, most of these methods guide policy updates through team rewards and lack the introduction of other common-sense information that is beneficial to collaborative confrontation tasks, restricting the optimization learning of multi-agent collaborative confrontation strategies. Therefore, the present invention aims to construct a multi-agent reinforcement learning algorithm framework based on tacit behavior pre-training. By introducing the concept of tacit behavior, the purpose is to effectively improve the learning efficiency of multi-agent collaborative confrontation strategies. The present invention enables multi-agents to form a spatial position advantage and achieve the goal of winning efficiently during adversarial training. At the same time, the designed algorithm framework does not rely on communication among multi-agents during the centralized training phase.

[0060] The present invention constructs a multi-agent reinforcement learning algorithm framework based on tacit behavior pre-training, aiming to obtain better learning performance in cooperative confrontation training through the spatial position advantage formed by tacit behavior, without relying on communication between multi-agents.

[0061] The present invention introduces the relative spatial position relationship and its change trend between multi-agents, and quantifies the spatial position relationship and its change between agents as a tacit reward to guide the actions of multi-agents. Based on the above concept, the present invention defines the consensus reached by multiple agents together in the cooperative confrontation problem that is beneficial to victory as tacit understanding; specifically, in the present invention, the spatial position relationship and its change trend between agents are used as the measurement standard of tacit understanding. Accordingly, the present invention proposes that multi-agents rely on tacit understanding to guide individual agents to generate corresponding action behaviors, so that multi-agents form a favorable spatial relationship based on joint actions, form a more favorable advantage in confrontation, and promote the more efficient achievement of the goal of winning in confrontation, and defines the corresponding behavior of each agent as a tacit behavior. At the same time, the present invention defines the spatial position relationship of each agent relative to the global state as the corresponding situation of each agent.

[0062] The present invention assists its own agents to win in the confrontation with the enemy by forming a favorable relative spatial position relationship, which is specifically manifested as individual agents forming tacit behaviors that cooperate with each other, so that the multi-agent system forms an adversarial advantage where its own side is more than the enemy side in the local space.

[0063] As Figure 1 shown, the multi-agent cooperative confrontation reinforcement learning method based on tacit behavior pre-training disclosed in the preferred embodiment of the present invention includes:

[0064] Distributed tacit pre-training stage, including: constructing a situation mechanism and a tacit understanding mechanism that characterize the relative spatial position relationship between agents, constructing a tacit reward function for each agent according to the situation mechanism and the tacit understanding mechanism, and respectively performing tacit pre-training on each agent based on the tacit reward function of each agent;

[0065] Centralized confrontation training stage, including: performing multi-agent cooperative confrontation training based on the results of tacit pre-training of multiple agents.

[0066] Among them, the constructed situation mechanism divides the situation based on the spatial correlation position of multi-agents and its time-varying information, and determines the calculation method of the membership degree of the agent situation. The constructed tacit understanding mechanism determines the degree of tacit guidance for each situation based on the situations classified by the situation mechanism. Then, according to the membership degree of the agent situation and the corresponding degree of tacit guidance, the tacit reward function of the agent can be calculated.

[0067] The multi-agent reinforcement learning method based on tacit behavior pre-training proposed by the present invention includes two training processes, namely, the distributed tacit pre-training stage and the centralized adversarial training stage. By refining the multi-agent reinforcement learning framework, the learning efficiency of the collaborative adversarial behavior strategy is improved. This method introduces the relative positions and time-varying information of multi-agent spaces, and relies on the policy network obtained through pre-training to enable agents to form spatial position advantages in adversarial training, thereby achieving the goal of winning efficiently.

[0068] As Figure 2 shown, the algorithm framework designed by the present invention includes two learning stages: distributed tacit pre-training and centralized adversarial training; for the tacit pre-training stage, a situation mechanism and a tacit mechanism are constructed to represent the relative spatial position relationship between agents, which are used to dynamically guide the action behaviors of agents to obtain a spatial advantage for promoting cooperation. In the subsequent centralized adversarial training stage, the present invention uses the pre-trained network to enhance the formation of favorable relative spatial positions and obtain more efficient learning performance. Among them Figure 2 the direction of the solid arrow represents the data flow, that is, it represents the input and output of data for each module. Figure 2 The superscript t of each parameter in represents the time step t, and the subscript i represents agent i; where i represents any value in 1 to n, so the subscripts 1 and n represent agent 1 and agent n respectively. Among them, s t represents the global state space at time step t, represents the observation space of agent i at time step t, represents the action space of agent i at time step t, represents the trajectory space of agent i at time step t, represents the membership degree corresponding to the situation of agent i at time step t, represents the situation to which agent i belongs at time step t, represents the tacit reward function of agent i at time step t, represents the tacit guidance degree of agent i at time step t, T j represents the tacit value, represents the output Q value of the pre-trained network for agent i. represents the global reward function corresponding to the centralized adversarial training stage at time step t, Q 总 (τ t , u t ) represents the output Q value of the hybrid network. In the centralized adversarial training, the algorithm selected by the present invention is the QMIX (Q-value Mixing) algorithm, and the hybrid network in it comes from this algorithm and is used to non-linearly combine the Q values of each agent through the hybrid network.

[0069] The algorithm framework designed by the present invention includes two learning stages: distributed tacit pre-training and centralized adversarial training. For the tacit pre-training stage, a situation mechanism and a tacit understanding mechanism are constructed to represent the spatial relative position relationship between agents, which are used to dynamically guide the action behavior of agents to obtain a spatial advantage for promoting cooperation. In the subsequent centralized adversarial training stage, the present invention uses the pre-trained network to enhance the formation of favorable spatial relative positions and obtain more efficient learning performance.

[0070] The core technical solution of the present invention will be introduced in detail from the following three aspects: the situation mechanism, the tacit understanding mechanism, and the overall training process.

[0071] (1) Situation mechanism

[0072] The situation mechanism is divided into two parts: situation classification and membership degree calculation. The situation classification part takes the global state s t and the local observation as inputs, and is used to classify the "situation" of the agent. The "situation" is defined as the relative position of the agent and friendly agents (referring to other agents on one's own side) at a given time. Each situation corresponds to a specific tacit behavior, but agents in the same situation need different degrees of guidance to achieve the corresponding tacit behavior. To capture these differences, the membership degree is calculated, and this variable is calculated from s t and .

[0073] In the present invention, four different situations are constructed, as shown in Figure 3 . Each point represents an agent, and points of different shapes represent their corresponding situations. At each time step, the situation to which agent i belongs is classified, denoted as p i . These situations are classified according to the following two dimensions: the spatial relationship between all agents in the global state, and the spatial relationship between agent i and other agents within its local observation.

[0074] Based on the situation classification, the spatial relationship between agent i and other agents is further quantified by the membership degree λ i corresponding to the situation of agent i. Specifically, the position of agent i is represented as where o i represents the observation space of agent i, s represents the global state space, and d i represents the perception distance of agent i. For simplicity of expression, the following will abbreviate as

[0075] The following introduces the four situation classifications of agent i and the calculation methods of their corresponding membership degrees. Among them, the four situations are: Fully Decentralized situation p fd , Partially Decentralized situation p pd , Fully Aggregated situation p fa , and Partially Aggregated situation p pa . The corresponding membership degrees of each situation are: λ fd , λ pd , λ fa , and λ pa .

[0076] (a) Fully Decentralized situation

[0077]

[0078] Among them, ||·||2 represents the Euclidean norm between two positions. When all multi-agents in the global state satisfy formula (1), it means that the distance between each pair of agents (agent i and agent k) exceeds their respective perception ranges.

[0079] Each agent in this global state belongs to the Fully Decentralized situation p fd , and its corresponding membership degree λ fd is:

[0080]

[0081] Among them, the set ρ i represents the set of all agents within the perception range of agent i, and the value of the parameter β is greater than 1, which is used to limit the upper limit value of λ.

[0082] (b) Partially Decentralized situation

[0083]

[0084] When the multi-agents in the global state do not satisfy formula (1), but satisfy formula (3), and agent i meets the conditions listed in formula (4). Intuitively, a scenario where some agents in the global state are aggregated is formed. And if there are no friendly agents within the perception range of agent i, then agent i in this scenario belongs to the Partially Decentralized situation p pd , and the corresponding membership degree λ pd is as shown in formula (5), where formula (5) is the same as formula (2).

[0085]

[0086] (c) Fully aggregated situation

[0087]

[0088] Among them, the symbol |·| represents the number of elements in the set. For agent i that does not satisfy formula (4), but the global state it is in (the global state refers to the global state space s t ) satisfies formula (3), the agents in this global state can perceive each other, and the situation to which agent i belongs is divided into two categories: partially aggregated situation p pa and fully aggregated situation p fa . In the fully aggregated situation p fa , an agent can perceive multiple agents within its perception range. As shown in formula (6), its corresponding membership degree λ fa is:

[0089]

[0090] In the formula, α is a parameter, a quantity parameter adjusted according to the actual situation, and its value range is greater than 0 and less than 1.

[0091] (d) Partially aggregated situation

[0092]

[0093] Similar to the fully aggregated situation p fa , in the partially aggregated situation p pa , an agent can perceive and only perceive one friendly agent, as shown in formula (8). In the present invention, the agent on the left among the two agents is determined as the leading agent. The membership degree of the situation to which this agent belongs is measured by the distance between this agent and the agents outside its perception range, while the membership degree of the situation of its following agent is measured by the distance between it and the leading agent. In the present invention, the corresponding variable subscripts of the leading agent (leader) and the following agent (follow) are respectively marked as l and f; the corresponding membership degree of the leading agent (leader) and the membership degree

[0094]

[0095] In summary, the present invention covers all possible relative position relationships of each agent in the multi-agent space from two different dimensions: the aggregation and dispersion relationships of the global multi-agent and the relative position relationship of the individual agent relative to the global. Moreover, the four situations can be converted into each other and have no intersection. The membership degree of the situation quantifies the spatial relationship among the multi-agents and is used as the parameter value for dynamically adjusting the reward. The classification mechanism of the situation classifies the agents at different spatial positions in the global state, and combines with the membership degree of the situation to form a refined and targeted tacit guidance to quickly achieve local confrontation advantages.

[0096] (2) Tacit mechanism

[0097] The tacit mechanism takes the situation corresponding to each agent, the membership degree of the corresponding situation, and the local action-observation history as inputs to construct a reward to guide the agent to learn the corresponding tacit behavior. Since agents in different situations have significant differences in their local observation spaces, using the same tacit behavior to guide will lead to low learning efficiency and lack of pertinence. Based on the situation to which it belongs, the tacit mechanism designs four targeted degrees of tacit guidance to provide more accurate tacit guidance for agents with different perception situations. The following calculates the degrees of tacit guidance for the four situations.

[0098] Among them, the position of agent i at time step t is represented as For simplicity of expression, hereinafter will be abbreviated as is the coordinate position of the fixed point, and d i represents the perception distance of agent i.

[0099] (a) Fully dispersed situation

[0100] In the fully dispersed situation p fd , when all agents in the global state cannot perceive each other, the goal is to guide the agents to form a tacit behavior of gathering towards the fixed point, and its degree of tacit guidance is:

[0101]

[0102] (b) Partially dispersed situation

[0103] Different from the fully dispersed situation p fd , for agents belonging to the partially dispersed situation p pd , there are some agents in the global state where they are located that can gather with each other. Therefore, the goal in the partially dispersed situation p pd is to make the agent gather towards its nearest agent, and its degree of tacit guidance is:

[0104]

[0105] (c) Fully aggregated situation

[0106] For the agents belonging to the fully aggregated situation p fa it is expected that the agents can always perceive the friendly agents within their observation space and maintain a certain distance, so as to expand the range of the perceivable space of multiple agents, and the degree of tacit guidance is as follows:

[0107]

[0108] (d) Partially aggregated situation

[0109] In the partially aggregated situation p pa it is set that the leading agent moves towards the nearest agent outside its perception range, while the following agent moves towards the direction of the leading agent. The degree of tacit guidance of the corresponding leading agent (leader) and the degree of tacit guidance of the following agent (follow) are respectively as follows:

[0110]

[0111] Among them, the construction of the above-mentioned degree of tacit guidance only considers the spatial changes caused by the actions of a single agent, so as to avoid being interfered by the actions of friendly agents in the guidance learning of tacit behaviors. Based on the classified situations, the present invention designs tacit guidance from two perspectives: whether the agent can perceive the friendly agent, and the spatial position relationship between agents in the global state; this method enables the agent to efficiently master the tacit behaviors corresponding to different situations.

[0112] (3) Overall training process

[0113] In the algorithm framework designed by the present invention, the effectiveness of tacit pre-training determines the formation of the spatial position advantage in the subsequent adversarial training, and further affects the effectiveness of the centralized adversarial training. As Figure 4 shown, in the pre-training stage, a tacit reward function is constructed for the agent by using the situation and the tacit mechanism to learn the tacit behavior strategy. At the same time, the degree of tacit mastery is determined by the tacit mechanism to determine when the pre-training cut-off training goal is reached.

[0114] Among them, the pseudo-code of the tacit pre-training process is as follows:

[0115]

[0116]

[0117] Through Figure 4The network structure of the tacit pre-training part (the network used for this tacit pre-training is called the policy network) is subjected to tacit pre-training. Specifically, the network takes the local observation space of agent i and the action space as input variables. The first layer is an MLP (Multilayer Perceptron) that learns features through multiple non-linear transformations, and the second layer is a GRU (Gated Recurrent Unit) that captures time-dependent relationships through a gating mechanism. At the same time is the hidden state (such as the historical state and action sequence of agent i in the environment). The third layer is an MLP layer, and the output variable of the network is the Q-value function, which is trained through the TD loss function L in line 19 of the pseudo-code p . Among them, lines 15, 17, 18, and 20 in the pseudo-code represent the standard training process of the reinforcement learning part, collecting data samples and extracting data samples for training.

[0118] Among them, the TD loss function L p is:

[0119]

[0120] In the formula, respectively represent the network parameters of the policy network f p (θ p ) and the target policy network ; is the product of the tacit guidance degree in the tacit mechanism and the membership function in the situation mechanism, as shown in line 14 of the pseudo-code, representing the tacit reward function of all agents at the sampling time step . γ is a numerical parameter, respectively represent the outputs of the policy network and the target policy network, represents the trajectory space of all agents at the sampling time step , represents the action space of all agents at the sampling time step .

[0121] Among them, Figure 4 compared with Figure 2 refines the description of the entire tacit training process, mainly describing how to judge whether the tacit behavior is mastered during the training process and when to end the pre-training process, which also exactly corresponds to what is described in the above pseudo-code. First, for the tacit value T j , the index NR j (the number of time steps consumed when the reward size is positive), the index NT j (for each agent in the situation p jThe cumulative number of time steps spent is initialized to 0. The subscripts of these three metrics correspond to situation j, where j ∈ {fd, pd, fa, pa}, corresponding to line 3 of the pseudocode. According to the aforementioned situation mechanism and tacit understanding mechanism, the observation space of the agent obtained from the environment at time step t (at this time and the global state space s t-1 , s t ) are used to calculate the situation p i to which agent i belongs, the corresponding membership value λ i of this situation, and the tacit understanding guidance degree corresponding to the current action (select the action that maximizes the Q value according to the policy network), corresponding to lines 6 - 8 of the pseudocode. According to situation p i , the corresponding membership value λ i of the situation, and the corresponding tacit understanding guidance degree , calculate the metrics NR j and metric NT j (the calculation method is shown in lines 10 and 11 of the pseudocode). Based on metrics NR j and metric NT j , calculate the tacit understanding value T j = NR j / NT j . After cumulative calculation over t tac time steps (mainly cumulative calculation of metric NR j , metric NT j and the corresponding changing tacit understanding value T j ), at this time, test whether the tacit understanding value [T fd , T pd , T fa , T pa reaches the tacit understanding target [TO fd , TO pd , TO fa , TO pa , TO j . If the metric is reached, it means that the tacit understanding behavior has been mastered, and the tacit understanding pre-training process can be stopped.

[0122] Among them, D in line 17 of the pseudocode represents the replay buffer. A set of samples are randomly sampled from D for the training process of reinforcement learning. Since it involves the reinforcement problem of multiple agents, i is used as the subscript to represent agent i; in line 18, represents the historical trajectory composed of all agents, represents the collection of reward values of all agents, Represents the set of all agent actions. Line 19 represents the calculation formula of the TD (Temporal Difference) loss function. Line 20 is to update and optimize the parameters of the two policy networks based on minimizing the loss function.

[0123] The above pseudocode shows the learning process of tacit pre-training and the conditions for determining the end of learning in a round of multi-agent collaborative adversarial training. Based on the membership degree and tacit guidance of the situation generated by the tacit mechanism and the situation mechanism, a corresponding tacit reward function is constructed for each agent, and tacit pre-training is carried out based on the constructed reward (such as lines 7 and 14 of the pseudocode). Among them, each agent shares network parameters and uses the network to learn the corresponding tacit behaviors under four different situations to explore a more efficient and flexible pre-training network model.

[0124] In addition, to determine when the pre-training network reaches the expected training goal, the network performance is evaluated after each round. Specifically, the present invention quantifies the cumulative time steps spent by each agent in situation p j and records it as NT j , and records the time steps consumed when the reward size is positive as NR j (lines 8 to 11 of the pseudocode). To evaluate the mastery degree of the tacit behavior TB j , the present invention defines the tacit value T j and calculates it as shown in line 12 of the pseudocode. After t time steps, T j is compared with the corresponding tacit goal TO j to evaluate whether the agent has mastered TB j .

[0125] After the tacit pre-training network (the parameters of the pre-training network are θ p ) finishes training, multi-agent collaborative adversarial training is carried out based on the pre-training network using the global reward (the parameters of the adversarial training network are ξ f ). The following formula defines the temporal difference (TD) loss function of the adversarial training:

[0126]

[0127] In the formula, is the target adversarial policy network, and the expected value E is estimated through the multi-round game samples stored in the replay buffer B; θ p represents the network parameters of the policy network f p (θ p ), ξ f is the adversarial training network parameter, r global is the global reward function (generated from the simulation environment according to the state change caused by the action selection), Qtot For the adversarial policy network, τ is the trajectories of all agents globally, and u is the action space of all agents globally.

[0128] The reinforcement learning method proposed by the present invention has a relatively wide application prospect, and is particularly suitable for complex scenarios with cooperative adversarial characteristics. In the civilian field, this method can significantly improve the adversarial level of game AI, and through the cooperative decision-making and policy optimization among agents, provide players with a more challenging and realistic game experience. In the military field, this method has important strategic value in the cooperative combat of unmanned aerial vehicle clusters, can realize the intelligent cooperation and tactical coordination of multiple unmanned aerial vehicles, improve the task execution efficiency and combat effectiveness in the air combat scenario, and provide technical support for decision-making support in complex battlefield environments. In addition, this method can also be extended and applied to multiple fields such as robot collaboration, intelligent transportation scheduling, and financial investment strategy optimization, providing an innovative technical solution for solving the cooperative decision-making problem in complex systems.

[0129] Another preferred embodiment of the present invention discloses a storage medium, in which a computer program is stored, and the computer program is configured to be run by a processor to execute the multi-agent cooperative adversarial reinforcement learning method based on tacit behavior pre-training in the above embodiment.

[0130] Optionally, the above storage medium may include, but is not limited to: USB flash drive, read-only memory (ROM for short), random access memory (RAM for short), mobile hard disk, magnetic disk or optical disc and other various media that can store computer programs.

[0131] The background part of the present invention may include background information about the problems or environment of the present invention, rather than the description of the prior art by others. Therefore, the content included in the background art section is not an admission by the applicant of the prior art.

[0132] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several alternatives or modifications can be made to these described embodiments, and these alternative or modified forms should all be regarded as belonging to the protection scope of the present invention. In the description of this specification, the descriptions referring to terms such as "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the scope defined by the appended claims.

Claims

1. A multi-agent collaborative adversarial reinforcement learning method based on tacit behavior pre-training, characterized in that, Including: A distributed tacit pre-training stage, including: constructing a situation mechanism and a tacit understanding mechanism for representing the spatial relative position relationship between agents, constructing a tacit understanding reward function for each agent according to the situation mechanism and the tacit understanding mechanism, and respectively performing tacit understanding pre-training on each agent based on the tacit understanding reward function of each agent; A centralized adversarial training stage, including: performing multi-agent collaborative adversarial training based on the results of the tacit understanding pre-training of multiple agents.

2. The multi-agent collaborative adversarial reinforcement learning method based on tacit behavior pre-training according to claim 1, wherein The policy network for performing tacit understanding pre-training takes the local observation space and action space of the agent as input variables. The first layer is the first multi-layer perceptron layer that learns features through multi-layer non-linear transformation. The second layer is the gated recurrent unit layer that captures temporal dependencies through a gating mechanism and simultaneously inputs the hidden state of the agent. The third layer is the second multi-layer perceptron layer, and the output variable is the Q-value function, which is trained through the TD loss function.

3. The multi-agent collaborative adversarial reinforcement learning method based on tacit behavior pre-training according to claim 1, characterized in that During the tacit pre-training process, training is carried out through the TD loss function L p : where θ p and represent the network parameters of the policy network f p (θ p ) and the target policy network respectively, represents the coordination reward function of all agents at the sampling time step , γ is a numerical parameter, represent the outputs of the policy network f p (θ p ) and the target policy network respectively, represents the trajectory space of all agents at the sampling time step , represents the action space of all agents at the sampling time step .

4. The multi-agent cooperative adversarial reinforcement learning method based on pre-training of tacit behavior according to claim 1, characterized in that The process of tacit understanding pre-training specifically includes: A1: Initialize the default value T j , each agent in the situation p j The cumulative number of time steps spent in j 、The number of time steps NR when the value of the tacit reward function is positive j , situation p j Indicates the various situations divided according to the situation mechanism described; A2: Calculate the situation p to which agent i belongs at time step t according to the situation mechanism and the tacit understanding mechanism i and the corresponding membership degree λ i of the situation p i , as well as the corresponding tacit understanding guiding degree and calculate the tacit understanding reward function according to the membership degree value λ i and the tacit understanding guiding degree . The situation p i represents the situation to which agent i belongs; A3: Calculate each agent’s position in situation p j The cumulative number of time steps spent in j The number of time steps NR when the value of the tacit reward function is positive j , and calculate the tacit understanding value T j =NR j / NT j ; A4: Test the tacit understanding value T j Whether the tacit understanding goal is achieved. If the tacit understanding goal is achieved, stop the tacit understanding pre-training.

5. The multi-agent collaborative adversarial reinforcement learning method based on tacit behavior pre-training according to claim 1, characterized in that Constructing a situation mechanism for representing the spatial relative position relationship between agents includes: using the global state and local observation as inputs to classify the situation of the agent, where the situation of the agent represents the relative position of the agent and other agents within a preset time.

6. The multi-agent collaborative adversarial reinforcement learning method based on pre-training of tacit behavior according to claim 5, wherein The situations of the agent are divided into: fully decentralized situation p fd , partially decentralized situation p pd , fully aggregated situation p fa , partially aggregated situation p pa , and the membership degrees corresponding to each situation are: λ fd , λ pd , λ fa , λ pa ; among them, the fully decentralized situation p fd means that the distance between every two agents exceeds their respective sensing ranges; the partially decentralized situation p pd means that the distance between the corresponding agent and all other agents exceeds the sensing range of this agent, but the distances between the agents other than the corresponding agent and all other agents are within their respective sensing ranges; the fully aggregated situation p fa means that the distance between the corresponding agent and at least two other agents is within their respective sensing ranges, and the distances between the agents other than the corresponding agent and all other agents are within their respective sensing ranges; the partially aggregated situation p pa means that the distance between the corresponding agent and only one other agent is within their respective sensing ranges, and the distances between the agents other than the corresponding agent and all other agents are within their respective sensing ranges.

7. The multi-agent collaborative adversarial reinforcement learning method based on tacit behavior pre-training according to claim 5, characterized in that When all multi - agents in the global state satisfy formula (1), it indicates that each agent belongs to the fully decentralized situation p in this global state fd , and the corresponding membership degree λ fd As shown in formula (2): where, ||·||2 represents the Euclidean norm between two positions, represents the position of agent i, represents the position of agent k, and the position of the agent is determined by the observation space and state space of the agent, d i represents the perception distance of agent i, and the set ρ i represents the set of all agents within the perception range of agent i, and the value of the parameter β is greater than 1; When the multi-agent in the global state does not satisfy formula (1), but satisfies formula (3), and agent i meets the conditions listed in formula (4), it indicates that agent i belongs to the partially decentralized situation p in this global state pd , and the corresponding membership degree λ pd As shown in formula (5): For agent i that does not satisfy formula (4), but the global state it is in satisfies formula (3), the agents in this global state can perceive each other. The situations to which agent i belongs are divided into two categories: the partially aggregated situation p pa and the fully aggregated situation p fa : When the agent i satisfies formula (6), it indicates that the agent i belongs to the full aggregation situation p fa , and the corresponding membership degree λ fa is shown in formula (7): wherein the value of the parameter α is greater than 0 and less than 1; When the agent i satisfies formula (8), it indicates that the agent i belongs to the partial aggregation situation p fa , in the partial aggregation situation p fa , one of the agents is determined as the leading agent, and the other agent is determined as the following agent, where the membership degree of the leading agent and the membership degree of the following agent are as shown in formula (9): In the formula, represents the position of the leading agent, represents the position of the follower agent, d l represents the perception distance of the leading agent, d f represents the perception distance of the follower agent, and the set ρ l represents the set of all agents within the perception range of the leading agent.

8. The multi-agent cooperative adversarial reinforcement learning method based on tacit behavior pre-training according to claim 7, characterized in that, Constructing a tacit understanding mechanism for representing the spatial relative position relationship between agents includes: using the situation corresponding to each agent, the corresponding membership degree, and the local action-observation history as inputs to construct a tacit understanding guiding degree to guide the agent to learn the corresponding tacit behavior, where Full dispersion situation p fd Degree of tacit guidance is as follows: wherein, represents the position of agent i at time step t, is the coordinate position of the fixed point; Partial dispersion situation p pd Degree of tacit guidance is as follows: wherein, denotes the set of all agents within the perception range of agent i at time step t - 1; Full aggregation situation p fa Degree of tacit guidance is as follows: Partial aggregation situation p fa Tacit guidance degree of the leading agent And tacit guidance degree of the follower agent Are respectively: In the formula, represents the set of all agents within the perception range of the leading agent at time step t-1, represents the position of the leading agent at time step t, represents the position of the following agent at time step t.

9. The multi-agent collaborative adversarial reinforcement learning method based on tacit behavior pre-training according to claim 8, wherein, constructing the tacit understanding reward function for each agent according to the situation mechanism and the tacit understanding mechanism includes: multiplying the membership degree of the situation to which each agent belongs by the corresponding tacit understanding guiding degree to obtain the tacit understanding reward function of each agent.

10. A storage medium, characterized in that, A computer program is stored in the storage medium, wherein the computer program is configured to be run by a processor to execute the multi-agent collaborative adversarial reinforcement learning method based on tacit behavior pre-training according to any one of claims 1 to 9.

Images