A multi-agent transfer reinforcement learning method based on graph attention network

Abstract

The application discloses a kind of multi-agent migration reinforcement learning method based on graph attention network.The application calculates the track feature of the interaction of intelligent agent and environment, constructs the knowledge transfer relationship graph of intelligent agent according to the similarity of intelligent agent track feature;Hard attention mechanism is applied on the relationship graph to cut off irrelevant transfer relationship, and further application soft attention mechanism selects the intelligent agent with the maximum attention weight as the knowledge transfer object.In the way of judging knowledge transfer time, a binary classification neural network is designed, the action and value of the intelligent agent and the knowledge transfer object are input into the network, and whether the knowledge transfer is executed this time is output;By designing the loss function based on the variance of action value to guide the training of classification network, adaptive learning and judgment of knowledge transfer time are realized;This method fully reduces unnecessary knowledge transfer, increases the efficiency of effective knowledge transfer, and promotes the intelligent agent to obtain a faster convergence speed and better solution on the task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-agent reinforcement learning, and relates to a multi-agent transfer reinforcement learning method based on a graph attention network. BACKGROUND

[0002] With the progress of science and technology, multi-agent systems have become a hot topic in the study of complex dynamic systems, attracting widespread attention from scientists in the fields of computer science, artificial intelligence, biological ecology, communication control, and many others. Multi-agent systems are widely used in various industries, including wireless communication systems, power networks, robots, and game AI. With the increasing labor costs in current society and the rapid development of artificial intelligence algorithms, compared with traditional methods that require experts to design multi-agent system task strategies, multi-agent systems trained using reinforcement learning technology can learn how to complete tasks on their own without expert guidance, which makes more and more researchers tend to use reinforcement learning technology to design multi-agent system task strategies. The complex task environment and large number of agents in multi-agent systems pose higher requirements on the task solving ability and learning efficiency of the system. Therefore, it is crucial to design an efficient multi-agent reinforcement learning algorithm for specific multi-agent task scenarios.

[0003] Multi-agent reinforcement learning usually establishes a corresponding policy learning network for agents in the environment for specific task scenarios. Further design of reinforcement learning algorithms enables agents to adjust their network parameters based on the reward feedback from the environment during interaction with the environment, in order to better complete the preset tasks of the multi-agent system. In actual work scenarios, agents are not aware of environmental information, task information, reward information, and how to complete the task at the beginning of the task. A random exploration strategy is usually designed for agents to help them understand the environment and task, so that learning agents from scratch need a long time of training to better adapt to the task environment and learn how to complete the task. For example, in a football game scenario, agents (players) do not know in advance that scoring a goal by shooting the football into the goal will result in a score, so agents move randomly on the field and need a long time of training to learn how to score. A good agent strategy usually requires tens of thousands of rounds of training, consuming a large amount of training time and computing resources. Therefore, how to design an efficient reinforcement learning algorithm to accelerate learning in multi-agent systems is a problem worth studying.

[0004] Transfer learning is an effective means to accelerate learning by transferring knowledge from experts or other processes to the current task. The traditional multi-agent reinforcement learning has the problems of slow training speed and low learning efficiency, which is largely due to the lack of effective communication and knowledge sharing among the agents in the environment. For example, in a navigation task, each agent learns how to reach the target independently, which is equivalent to training multiple identical strategies simultaneously, resulting in repeated training and resource waste. If an agent learns how to reach the target, sharing its knowledge with other agents can significantly speed up the learning efficiency of the system. Therefore, multi-agent transfer reinforcement learning, which combines transfer learning and reinforcement learning, is an effective means to accelerate the learning of multi-agent systems.

[0005] The core idea of multi-agent transfer reinforcement learning is to help agents learn the target task by drawing on the knowledge of similar tasks, in order to improve the learning efficiency on the target task. Specifically, agents need to transfer knowledge with other agents in the environment, and select appropriate agents for knowledge recommendation to help themselves learn better. In a multi-agent system, agents try to explore the environment to learn appropriate strategies to solve problems, but the differences in exploration by different agents lead to different strategies and knowledge. By finding well-performing agents for knowledge transfer, poorly performing agents can be helped to learn, thereby speeding up the learning speed of the system.

[0006] Existing multi-agent transfer reinforcement learning algorithms include: traditional methods relying on human experts; knowledge transfer methods based on recommendations, including Ad-Hoc, recommendation exchange, bidirectional imitation, etc.; multi-agent transfer reinforcement learning algorithm based on recommendation budget (Teachona Budget); and RCMP (Relaxed Confidence Method with Cautious) which applies the idea of multi-head network to DQN (Deep Q Network). However, the above methods require the addition of human experts or polling all agents to select the transfer target, which is only suitable for simple scenarios with a small number of agents. In complex task scenarios with a large number of agents, it is difficult for human experts to design task strategies, and polling all agents for recommendations will result in high computational cost, low efficiency, and slow convergence speed. In addition, existing algorithms still cannot effectively solve the following three problems:

[0007] (1) What knowledge to transfer. Current multi-agent transfer reinforcement learning algorithms usually take states, actions, action values, network parameters, abstract high-level knowledge, etc. as transfer content. When the transferred knowledge is too much, it will increase the communication cost; when the transferred knowledge is too little, it is difficult for the knowledge to help learning, so it is necessary to design appropriate knowledge transfer content.

[0008] (2) Who: There are a large number of agents in the multi-agent system, each agent has different knowledge and task completion ability, how to select the appropriate agent for knowledge transfer is a key problem. Only selecting the best elite agent for knowledge transfer ignores the possibility of obtaining effective knowledge transfer from other agents (with similar trajectories, etc.). The polling method will lead to resource waste and increase the migration cost.

[0009] (3) When: After determining the knowledge transfer object, the agent needs to decide whether to accept the knowledge transferred by the transfer object, and blind acceptance of knowledge transfer may lead to negative transfer. If the agent itself can handle the task well, the redundant knowledge transfer process will also waste communication resources. SUMMARY

[0010] In view of the problems such as the deficiency of the current algorithm in selecting the knowledge transfer object and judging the knowledge transfer time, the application provides a multi-agent transfer reinforcement learning method based on a graph attention network.

[0011] In the application, the selection of the knowledge transfer object is different from the way of selecting the knowledge transfer object by polling between agents in the traditional multi-agent transfer reinforcement learning algorithm. The trajectory features of the interaction between the agent and the environment are calculated, and the knowledge transfer relationship graph of the agent is constructed according to the similarity of the trajectory features of the agent. The hard attention mechanism is applied on the relationship graph to cut off irrelevant transfer relationships, and the soft attention mechanism is further applied to select the agent with the maximum attention weight as the knowledge transfer object. This way not only effectively increases the accuracy of the selection of the knowledge transfer object, but also reduces the calculation cost, and is better suitable for the scenario with a large number of agents. In the way of judging the knowledge transfer time, a binary classification neural network is designed, the action and value of the agent and the knowledge transfer object are input into the network, and whether the knowledge transfer is executed this time is output. The loss function based on the action value variance is designed to guide the training of the classification network, and the knowledge transfer time is learned and judged adaptively. This way fully reduces unnecessary knowledge transfer, increases the efficiency of effective knowledge transfer, and promotes the agent to get a faster convergence speed and better solution in the task.

[0012] In the application, the task of the agent is to complete the task target set by the multi-agent system, and the solution is the policy network parameter of the agent to complete the task. On this basis, the method of the application is roughly divided into three stages:

[0013] (1) Selecting knowledge transfer object stage: define the agent in the multi-agent system as a point, and the knowledge transfer relationship between the agents as an edge, that is, construct the multi-agent system as a fully connected graph. The observation state of each agent is taken as the feature of its point, and the hidden feature representation of each node is calculated by a long short-term memory unit, that is, the agent trajectory feature. The trajectory features of all agents are input into a hard attention network, and the transfer relationship weight between the agents is output as 0 or 1 (0 means cutting off the edge that does not need to be transferred, and 1 means retaining the potential transfer relationship edge). Further, through a soft attention mechanism, the importance weight of the retained potential transfer relationship (the value is between 0 and 1) is obtained, and the agent with the largest attention weight is selected as the transfer object. That is, the most similar agent is selected as the transfer object through the similarity of the trajectory, because the agents with similar trajectories have similar task strategies.

[0014] (2) Judgment of knowledge transfer time stage: the knowledge of the agent is defined as what action should be taken in a certain environment state, so the content of knowledge transfer is the action. That is, the agent passes the current state to the transfer object in a certain state, and the transfer object gives what action should be taken in this state according to its own strategy knowledge, and passes it to the agent as an action suggestion. The variance of all action estimation values of the agent in the state is taken as the confidence of the state. The smaller the variance, the more uniform the distribution of the action estimation value in the state, and the agent cannot clearly take what action; and the larger the variance, the more discrete the distribution of the action estimation value in the state, and the agent has significant confidence for a certain action, so there is no need for knowledge transfer. By inputting the action estimation values of the two agents into a neural network, whether to perform knowledge transfer is output under the guidance of the state confidence, that is, the transfer time judgment is completed.

[0015] (3) Multi-agent reinforcement learning stage: the learning process of the agent in the environment is defined as a Markov decision process, that is, the agent learns the strategy as a state-action to reward mapping relationship. The agent stores the state, action, reward and the like obtained by interacting with the environment into an experience pool during the learning process, and trains the agent network by using importance sampling and the like to complete strategy updating. During the interaction between the agent and the environment, the agent first determines whether to perform knowledge transfer through the above-mentioned selection of knowledge transfer object and judgment of knowledge transfer time process. If knowledge transfer is performed, the action suggested in the knowledge transfer process is taken; if knowledge transfer is not performed, the action is selected by the strategy of the agent itself.

[0016] The present application adopts a traditional deep reinforcement learning algorithm (for example, a DQN algorithm, a DDQN algorithm and the like) to solve the strategy of a single agent in a multi-agent environment, and accelerates the task convergence speed through the transfer operation of the agent strategy knowledge.

[0017] To achieve the above objectives, the technical solution adopted by the present invention is as follows.

[0018] A multi-agent transfer reinforcement learning method based on graph attention networks, the specific steps of which are as follows:

[0019] Step 1: Set up the agent reinforcement learning model.

[0020] A multi-agent reinforcement learning task is represented by a tuple. <S,A i O i ,R i ,T,γ,n>. Where S represents the state space of the environment. a i ∈A i o i ∈O i and r i ∈R i Let represent the action space, observation space, and reward function of agent i, respectively. T is the state transition function of the environment, γ is the reward discount factor, and n is the number of agents in the environment. All agents are represented by neural networks with identical structures, and each agent can obtain its own observation o. i Due to the limitations of the task environment, if the state of the environment can be fully observed by every agent, it is a globally observable environment; otherwise, it is a partially observable environment. Partially observable environments increase the difficulty of learning for agents.

[0021] The actions of all agents constitute a joint action, denoted as . Each agent's policy π i It can be modeled as an agent in state s i Take action a i The probability, the joint policy of the agents is defined as π = (π / 2) 1 , π 2 ..., π n ),Right now

[0022] π = p(s, a) = p(a|s) = p((a) 1 a 2 , ..., a n )|s) (1)

[0023] Each agent's reward function is influenced by the environmental state and the actions of other agents, i.e., r i =R i (s, a, s′). The reward function is set so that only the agent itself can obtain it. The state transitions of the environment are determined by the previous state of the environment and the joint action of the agent.

[0024] For multi-agent systems, the goal of multi-agent reinforcement learning is to find a set of policy parameters that maximize the expected cumulative reward. The specific steps are as follows:

[0025] (1.1) Randomly initialize the agent Q-network parameters w i , and empty the experience pool D.

[0026] (1.2) The agent interacts with the environment iteratively.

[0027] (1.2.1) Observe the state si of the agent i, and calculate its feature

[0028] (1.2.2) Input the feature into the agent Q-network to obtain the estimated value of all actions, and use the ε-greedy method to select the executed action a i .

[0029] (1.2.3) Execute the action a i , obtain the environmental reward r i and the new environmental state s′ i , and store (s i , a i , r i , s′ i ) in the experience pool.

[0030] (1.2.4) Collect m samples (s j , a j , r j , s′ j ) from the experience pool D, j = 1,..., m. Calculate the target value y j :

[0031]

[0032] where a′ j is the action that estimates the agent will take to obtain the maximum reward at state s′ j , and the task is completed to indicate that the agent has reached the termination state.

[0033] (1.2.5) Update all parameters w i of the Q-network using the mean square loss error function by gradient backpropagation of the neural network:

[0034]

[0035] where is the value estimated by the policy network when taking action a j at state s j , and y j represents the value of the state sj the true reward obtained.

[0036] Step 2, selecting knowledge transfer objects

[0037] (2.1) Establishing an agent transfer relationship graph: the purpose of this step is to build the similarity relationship between agents. The state observed by the agent is represented as the feature of the point, and the hidden features of the agent trajectory are extracted by using the long short-term memory network (LSTM). The similarity between the hidden features of the agents is calculated to determine the transfer objects. If there is an edge between the agents, it means that there is a transfer relationship between the agents. Initially, the edges between the agents are all present by default, that is, a fully connected graph is constructed.

[0038] (2.2) Hard attention mechanism: input the fully connected graph established in step (2.1) into the hard attention mechanism. The output is the relationship graph after disconnecting unnecessary knowledge transfer edges, which realizes the quantitative reduction of knowledge transfer relationships. When outputting, a subset of all input elements is selected, only important elements are focused on, and other irrelevant elements are completely ignored. The traditional hard attention mechanism uses the Softmax function for normalization processing at the output end of the network. After obtaining the probability value corresponding to each output, sampling is performed. The sampling process causes the gradient to be unable to normally backpropagate. This method uses the Gumbel-softmax function to solve this problem. Gumbel noise information g i is added before the output of the above Softmax function, so that sampling samples y i with randomness can be output without the need for direct sampling.

[0039]

[0040] where x i and x j are the outputs of the last layer of the hard attention network; g i and g j are random noises; τ is a coefficient for controlling the output form of the Gumbel-softmax function. The smaller the value, the closer the output result is to the one-hot vector; K is the dimension of the input; exp(·) is the exponential function with e as the base; y i is the sampling sample after adding noise.

[0041] The specific steps are as follows:

[0042] (2.2.1) Input the observation of agent i into the LSTM network to obtain the hidden feature h i .

[0043] (2.2.2) The feature h i is sequentially combined with the features hj concatenated into a hard attention mechanism implemented by bi-LSTM:

[0044]

[0045] where, is the hard attention weight between agent i and agent j (its value is 0 or 1). 0 means that there is no migration relationship between i and j, then the edge between agent i and agent j is disconnected; 1 means that i and j have potential migration relationship, then the edge between agent i and agent j is retained. gum is Gumbel-softmax function, Bi-LSTM is bidirectional LSTM, and f is full connection layer.

[0046] (2.3) Soft attention mechanism: the irrelevant edges are deleted and only the potential migration relationship is retained in the agent migration relationship graph processed in step (2.2). The weight of the retained edge is further calculated through the soft attention mechanism, so as to select the migration target. The soft attention mechanism uses softmax function to calculate the importance distribution between agents.

[0047]

[0048] where, i and h k are the hidden features of agent i and k; f(·) is full connection layer; exp(·) is the exponential function with e as the base; K is the dimension of input hidden feature, w ik is the attention weight between agent i and j.

[0049] The specific steps are as follows:

[0050] (2.3.1) Initialize query vector W q and key-value vector W k .

[0051] (2.3.2) Input the feature h i and the features h j of other agents into the soft attention mechanism in turn:

[0052]

[0053] where, is the transpose of h j , is the transpose of W k , is the soft attention weight between agent i and agent j (its value belongs to 0 to 1).

[0054] (2.4) Calculate the migration target: the migration target is calculated by the agent migration relationship graph obtained in step (2.2) and step (2.3) with fusion, and obtain the final attention weight.

[0055]

[0056] wherein, W i，j is the attention weight between agent i and agent j, the greater the value, the more similar agent i and agent j are. Therefore, for agent i, the agent j with the maximum W i，j is selected as the migration target.

[0057] (2.5) Update the attention network weight: agents with similar trajectories tend to learn similar strategies, so knowledge transfer between agents with similar trajectories will usually speed up the learning of agents. The JS divergence between the hidden features of the agents is used as a similarity measure between the agents.

[0058]

[0059] wherein, n is the hidden feature dimension of the agent; x i is the i-th feature; JS(p||q) is the JS divergence of agent p and agent q.

[0060] (2.5.1) Calculate the similarity label: use the JS divergence to calculate the similarity index between agent i and agent j as the label.

[0061]

[0062] wherein, JS i，k is the JS divergence of the hidden features between agent i and k, and l i，j is the label of the similarity degree between agent i and k, which is used to guide the attention network training.

[0063] (2.5.2) Gradient descent update parameters: use the similarity label calculated in (2.5.1) as the true label of the attention weight calculated in step (2.4), and use the mean square error loss function to train the hard and soft attention mechanism.

[0064]

[0065] Step 3, judge the knowledge transfer time

[0066] On the basis of selecting knowledge transfer objects through the graph attention mechanism, a binary classification neural network is applied to determine whether the agent needs to accept the action recommended by the transfer object in the current state. The variance of the agent's action estimate value is used as the state confidence of the agent. If the variance of the action estimate value of the agent is small in a certain state, the agent estimates similar values for all possible actions, and the agent lacks confidence in processing the state (all action values are similar, and the agent does not know which action should be taken); if the agent estimates that the action values are not similar for all executable actions, i.e., the value of a certain action is significantly greater than the values of other actions, the agent has sufficient confidence in processing the state (the agent clearly knows to select the action with the maximum value).

[0067]

[0068]

[0069] wherein Q(s, a) is the estimated value of the agent taking action a in state s, |A| is the number of all possible actions that the agent can take, μ(s) is the value expectation of the agent in state s, and V(s) is the variance of the action value of the agent in state s, i.e., the state confidence.

[0070] The specific steps are as follows:

[0071] (3.1) Migration time judgment network input: In state s, input all action estimate values of the agent i and the migration target agent j and the values of the actions decided to be taken into the binary classification neural network.

[0072] (3.2) Obtain the migration time judgment result: According to the output of the neural network, determine whether to migrate this time. The output of the network is 0 or 1 (0 represents that no knowledge transfer is performed this time; 1 represents that knowledge transfer is performed).

[0073] (3.3) Neural network gradient update: After each forward propagation of the neural network, the label is calculated according to the rules designed by the state confidence and the action value, to guide the back propagation of the network parameters.

[0074]

[0075] wherein V i (s) and V j (s) are the action value variances of the agent i and the agent j in state s, Q i (s, a i ) and Q j (s, a j ) are the values of the respective actions taken by the agent i and the agent j in state s. Step 4, multi-agent transfer reinforcement learning

[0076] The process of selecting an action by an agent according to its own strategy in the multi-agent reinforcement learning of step 1 is modified as follows: first, the knowledge transfer object is selected through step 2, and then the knowledge transfer time is determined through step 3 to determine whether knowledge transfer is performed; if knowledge transfer is performed, the transferred action is used to complete the subsequent reinforcement learning process, otherwise the agent's own strategy is used to complete the reinforcement learning process; the agent performs necessary knowledge transfer in the iterative reinforcement learning process, optimizes the agent's strategy, and until the preset task completion capability index or the number of training times is reached.

[0077] The beneficial effects of the present application are as follows:

[0078] The present application adopts a new knowledge transfer method in the training process of multi-agent reinforcement learning. According to the similarity of the trajectories between agents, first, the hard attention mechanism is used to remove irrelevant agents, realizing the preliminary reduction of agent transfer relationship and saving computing resources. Secondly, the soft attention mechanism is used to calculate the weight of the agent transfer relationship, and the agent with the largest weight is selected for knowledge transfer, realizing the accurate selection of the transfer target and improving the accuracy of knowledge transfer; after selecting the knowledge transfer target, the two-class neural network of agent state information and action value training is used to adaptively determine whether to perform knowledge transfer. Not only does it prevent the occurrence of negative transfer, but also saves the communication cost of the transfer process. Finally, the learning ability and learning speed of multi-agent reinforcement learning are accelerated. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 The framework flowchart of the present application.

[0080] Figure 2 The agent reinforcement learning model used in the present application.

[0081] Fig. 3(a) is the knowledge transfer object algorithm model designed by the present application, and Fig. 3(b) is a schematic diagram of the knowledge transfer time judgment algorithm model.

[0082] Fig. 4(a) and Fig. 4(b) are schematic diagrams of the knowledge transfer object selection and knowledge transfer time judgment process and results, respectively. DETAILED EMBODIMENTS

[0083] The specific embodiments of the present application will be further described in combination with the drawings and technical solutions.

[0084] The method of the present application can be used to accelerate the training process of agents in a multi-agent environment, and the method flow of the present application is as shown in Figure 1 .

[0085] The DQN network is used to construct the agent in the present application, and its reinforcement learning model is as shown in Figure 2The knowledge transfer object algorithm and the knowledge transfer time judgment algorithm model are shown in FIG. 3(a) and FIG. 3(b).

[0086] The embodiments of the present application are described in detail as follows (as shown in FIG. 1), which specifically include the following steps: Figure 1

[0087] (1) Initialization process: first, initialize the DQN network parameters of the agent, and empty the experience pool.

[0088] (2) Main loop process:

[0089] (2.1) Observe the environment state: obtain the current environment state through the sensor of the agent.

[0090] (2.2) Calculate the action value: input the observed state features into the DQN network to obtain the estimated action value of all actions taken by the agent.

[0091] (2.3) Select the knowledge transfer target: select a knowledge transfer target for each agent through the hard attention mechanism and the soft attention mechanism.

[0092] (2.4) Judge the knowledge transfer time: input the action value and other information of the agent and the knowledge transfer target into the binary classification neural network to judge whether this knowledge transfer is needed.

[0093] (2.5) Select and execute the action: according to the knowledge transfer result, select the agent's own action or the transferred action, and execute the action to obtain the environment feedback reward and the next state features.

[0094] (2.6) Store the experience: store the agent state, action, reward, and next state information into the experience pool.

[0095] (2.7) Network parameter update: sample the sample data in the experience pool, calculate the loss, and use the gradient descent method to update the DQN network parameters reversely.

[0096] When the loop condition is not met, that is, the loop reaches the pre-set training number or the pre-set index, the agent strategy scheme that performs well in the multi-agent system can be obtained.

[0097] ​The knowledge transfer object selection and knowledge transfer time judgment process of the embodiment is shown in FIG. 4(a) and FIG. 4(b). As can be seen from the figure, in the navigation task of 6 unmanned vehicles, all unmanned vehicles need to navigate to the target, and in the method, the unmanned vehicles can be abstracted as agents as individuals participating in system training. The state of the agent (unmanned vehicle) is obtained through the sensor, and the "hollow circle" represents the agent, the "solid line" represents the transfer relationship between the agents, and the thicker the "solid line", the greater the transfer relationship weight between the agents. FIG. 4(a) shows the process of agent 1 selecting a transfer object in a system with 6 agents (unmanned vehicles). After hard attention reduction, agent 1 only has potential transfer relationships with agents 2, 5 and 6. Through the soft attention mechanism, the attention weight is obtained, and the final transfer object is agent 6, which accurately selects the transfer object. The process shown in FIG. 4(b) shows that the agent judges whether there is a transfer relationship between agent 1 and agent 6 through a binary classification neural network.

Claims

1. A multi-agent transfer reinforcement learning method based on a graph attention network, characterized in that, The specific steps are as follows: Step 1, set up the intelligent agent reinforcement learning model; A multi-agent reinforcement learning task is represented by a tuple ; wherein, represents a state space of an environment; , and represent an action space, an observation space and a reward function of an agent respectively; is a state transition function of the environment, is a reward discount factor, is a number of agents in the environment; all the agents are represented by a same structure of neural network, each agent can obtain its own observation , the agent is an unmanned vehicle; The actions of all agents constitute a joint action, denoted as . ; each agent's policy Modeling as an agent in state Take action below The probability, the joint policy of the agents is defined as ,Right now ； The reward function of each agent is affected by the state of the environment and the actions of other agents, i.e. The reward function is set to be only available to the agent itself, and the state transition of the environment is determined by the joint action of the previous state of the environment and the agent. For multi-agent system, the goal of multi-agent reinforcement learning is to find a set of policy parameters that maximize the expected cumulative reward; The specific steps are as follows: (1.1) initializing the agent randomly network parameters , empty the experience pool ; (1.2) Iterative interaction between agent and environment; (1.2.1) observing the state of the agent ;​​ (1.2.2) the features input agent network, obtaining an estimate of the value of all actions, using - the greedy method selects the action to perform ; (1.2.3) performing an action , obtaining an environment reward and a new environment state , storing into the experience pool; (1.2.4) collecting a sample from the experience pool ; computing a target value , i.e. the true reward obtained at the state ​​​ ； wherein, to estimate the action that the agent will take to obtain the maximum reward in state The task ends when the agent reaches a terminal state. (1.2.5) updating all parameters of the network using gradient backpropagation through the neural network using a mean square loss error function all parameters of the network : ； wherein, is the value obtained by the policy network estimate at state when the policy network estimate. Step 2, select the knowledge transfer object (2.1) Establish the agent migration relationship diagram: the purpose is to build the similarity relationship between agents; The state observed by the agent is used as the feature representation of the point, and the long short-term memory network LSTM is used to extract the hidden features of the agent trajectory, and the similarity between the hidden features of the agents is calculated to determine the migration object; If there is an edge between the agents, it means that there is a migration relationship between the agents; At the beginning, the edges between the agents are all present by default, that is, a fully connected graph is constructed; (2.2) Hard attention mechanism: input the fully connected graph established in step (2.1) into the hard attention mechanism, and the output is the relationship graph after disconnecting the unnecessary knowledge transfer edges, realizing the quantitative reduction of knowledge transfer relationship; before using the Softmax function to normalize the output end of the hard attention mechanism, Gumbel noise information is added using the Gumbel-softmax function Output samples with randomness, solve the problem that direct sampling leads to gradient unable to backpropagation; ； in, and This is the output of the last layer of the hard attention network; and It is random noise; The smaller the value of the coefficient that controls the form of the output of the Gumbel-softmax function, the closer the output is to a one-hot vector; The dimension of the hidden feature is used as input; For An exponential function with base 0; These are the sampled samples after noise has been added. The specific steps are as follows: (2.2.1) inputting the observation of the intelligent agent to the LSTM network to obtain hidden features ; (2.2.2) Hidden features concatenated with the features of other agents are concatenated and input into a hard attention mechanism implemented by a bi-LSTM: ； wherein, is the agent is the agent is the hard attention weight between the agent and the agent , which is 0 or 1; 0 means that there is no migration relationship between the agent and the agent ; 1 means that there is a potential migration relationship between the agent and the agent ; and the edge between the agent and the agent is retained. is the Gumbel-softmax function, is the bidirectional LSTM, is the full connection layer. (2.3) Soft attention mechanism: the agent migration relationship diagram processed in step (2.2) deletes irrelevant edges and only retains potential migration relationships; Further, the weight of the retained edge is calculated through the soft attention mechanism to select the migration target; The soft attention mechanism uses the softmax function to calculate the importance distribution between agents; ； wherein, and are hidden features of agents i and k; is a fully connected layer; is an exponential function with as base; is the dimension of the input hidden features, is the attention weight between agents i and j; The specific steps are as follows: (2.3.1) initializing the query vector and the key value vector ; (2.3.2) the hidden features of the other agents are sequentially input into the soft attention mechanism: ； wherein, is the transpose of , is the transpose of , is the soft attention weight between the agent and the agent , whose value belongs to 0 to 1. (2.4) Calculate the migration target: obtain the final attention weight by fusing the attention weight obtained in step (2.2) and the attention weight obtained in step (2.3) with fusion, obtain the final attention weight; ； in, For intelligent agents With intelligent agents The larger the attention weight between agents, the higher the value. With intelligent agents The more similar they are; therefore, for intelligent agents In terms of choice The largest intelligent agent As a migration target; (2.5) Update the attention network weight: use the JS divergence between the agent hidden features as the similarity measure between the agents; ； wherein, is the hidden feature dimension of the agent; is the i-th feature; is the agent and the agent JS divergence; (2.5.1) Compute similarity label: Use JS divergence to compute the similarity metric between the agent and the agent as the label; ； wherein, is an agent with JS divergence of hidden features between, is an agent with a label of similarity degree, for guiding attention network training; (2.5.2) Gradient descent update parameters: use the similarity label calculated in (2.5.1) as the true label of the attention weight calculated in step (2.4), and use the mean square error loss function to train the hard and soft attention mechanisms; ； Step 3, determine the knowledge transfer time Based on the selection of the knowledge transfer object through the graph attention mechanism, a binary classification neural network is applied to determine whether the agent needs to accept the action recommended by the migration object in the current state; The variance of the agent's action estimate value is used as the state confidence of the agent; If the variance of the agent's action estimate value is small in a certain state, the agent estimates similar values for all possible actions, and the agent lacks confidence in handling the state, i.e. all action values are similar, and the agent does not know which action to take; If the agent estimates the action value for all executable actions is not similar, i.e. the value of a certain action is significantly greater than the value of other actions, the agent has enough confidence to handle the state, i.e. the agent knows explicitly to choose the action with the maximum value; ； ； in, For the agent in state Take action below The estimated value, This represents the total number of possible actions that the agent can take. For the agent in state Value expectations below For the agent in state The variance of the value of the next action, i.e., state confidence; The specific steps are as follows: (3.1) Migration time judgment network input: in the state , the agent and all action estimates of the migration target agent and the action value of the respective decision to take; (3.2) Obtain the migration time judgment result: according to the output of the neural network, determine whether to migrate this time; The output of the network is 0 or 1, 0 means not to perform knowledge transfer, and 1 means to perform knowledge transfer; (3.3) Neural network gradient update: after each forward propagation of the neural network, the label is calculated according to the rules designed by the state confidence and the action value, which guides the back propagation of the network parameters; ； wherein, and are the action-value variance of the agent and the agent at state , and are the value of each of the performed actions of the agent and the agent at state . Step 4, multi-agent migration reinforcement learning The process of selecting actions by the agent according to its own strategy in the multi-agent reinforcement learning of step 1 is modified as follows: firstly, the knowledge transfer object is selected through step 2, and secondly, the knowledge transfer time is judged through step 3 to determine whether knowledge transfer is performed; if knowledge transfer is performed, the transferred action is used to complete the subsequent reinforcement learning process, otherwise the reinforcement learning process is completed using the agent's own strategy; the agent performs necessary knowledge transfer in the iterative reinforcement learning process, optimizes the agent's strategy, and until the preset task completion capability index or the number of training times is reached.

Images