A method for generating multi-agent adaptive sampling strategies

Abstract

The present invention discloses a method for generating a multi-agent adaptive sampling strategy. The method defines an agent and its policy network, evaluation network and experience according to an application environment, and realizes the automatic generation of a multi-agent collaborative strategy. The algorithm adopted by the method extracts the experience pool experience according to the priority probability by using the experience priority on the basis of the MADDPG algorithm: after each M step, for each agent, the policy loss of the current agent's recent M-step experience is calculated, and the priority of the current experience is calculated, and the experience of a minibatch size is extracted with the priority as the probability, and the minimum loss of the expected return is calculated through the target evaluation network, and the policy network parameters and the evaluation network parameters are updated; after each certain number of steps, the policy network parameters θ of all agents are updated πi The method of the present invention can accelerate the convergence of the model to improve the learning efficiency of the intelligent agent and reduce the exploration time of the intelligent agent, thereby improving the automatic generation and evaluation efficiency of the multi-agent collaboration strategy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for generating a multi-agent adaptive sampling strategy. Background Art

[0002] Reinforcement Learning (RL), also known as reinforcement learning, evaluation learning or enhanced learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how an agent can maximize rewards or achieve specific goals through learning strategies during its interaction with the environment.

[0003] Many scenarios in reinforcement learning involve the interaction of multiple intelligent agents, such as the control of multiple robots, language communication, multi-player games, etc.

[0004] MADDPG is a multi-agent reinforcement learning framework based on a deep deterministic policy gradient algorithm, which can be used to automatically generate multi-agent collaboration strategies.

[0005] In a multi-agent system, each agent obtains rewards by interacting with the environment to learn and improve its own strategy, thereby obtaining the optimal strategy in the environment. This process is called multi-agent reinforcement learning.

[0006] In single-agent reinforcement learning, the environment in which the agent is located is stable and unchanging, but in multi-agent reinforcement learning, the environment is complex and dynamic, which brings great difficulties to the learning process.

[0007] Dimension explosion: In single-agent reinforcement learning, it is necessary to store state-value functions or action-state-value functions. In multi-agent reinforcement learning, the state space becomes larger, and the connection action space (the connection action refers to the action of the multi-agent system at the current moment composed of the current actions of each agent) grows exponentially with the number of agents. Therefore, the multi-agent system has a very large dimension and is computationally complex.

[0008] Difficulty in determining target rewards: In a multi-agent system, each agent may have different tasks, but they are coupled and affect each other. The quality of reward design directly affects the quality of the learned strategy.

[0009] Instability: In a multi-agent system, multiple agents learn simultaneously. When the strategies of their peers change, the optimal strategy of each agent may also change, which will affect the convergence of the algorithm.

[0010] Exploration-Exploitation: Exploration not only considers the exploration of the environment itself, but also explores the changes in the strategies of peers, which may break the equilibrium state of the strategies of peers. The exploration of each agent may affect the strategies of peers, which will make it difficult for the algorithm to stabilize and slow down the learning speed. Summary of the invention

[0011] In view of the above-mentioned defects or shortcomings of the prior art, the purpose of the present invention is to provide a multi-agent adaptive sampling strategy generation method, which can accelerate the convergence of the MADDPG model, greatly improve the learning efficiency of the agent, reduce the exploration time of the agent, and improve the automatic generation efficiency of the multi-agent collaboration strategy.

[0012] To achieve the above object, the present invention provides a method for generating a multi-agent adaptive sampling strategy, comprising the following steps:

[0013] Step S1: Construct the neural network model of the agent and initialize the multi-agent collaborative strategy network π, target strategy network π', evaluation network Q, target evaluation network Q', and their respective network parameters θ π ,θ π' ,θ Q ,θ Q' ; Initialize the agent’s experience buffer pool and action exploration noise;

[0014] Step S2: Execute a set number of rounds, each round including the following steps:

[0015] Step S21: Initialize the environment and all agent state sets;

[0016] Step S22: For each step in the round, each agent learns experience and saves the experience learned by each agent into its own experience cache pool;

[0017] Step S23: Every M steps, where M is an integer not less than 2, for each agent, train its own neural network and execute:

[0018] Step S23-1: Calculate the strategy loss Loss of the current agent in the past M steps of experience;

[0019] Step S23-2: Calculate the priority Pr(i) of the current M-step experience according to the strategy loss Loss of the recent M-step experience;

[0020] Step S23-3: Determine whether the probability of the priority Pr(i) of the current M-step experience meets the threshold, and if so, extract the experience of minibatch size;

[0021] Step S23-4: learning the policy network and the evaluation network based on the experience of the extracted minibatch size;

[0022] Step S24: Loop through steps S22 and S23 until all experiences are traversed and the multi-agent collaboration strategy evaluation results are output.

[0023] Furthermore, the step S22 specifically includes:

[0024] 1) According to the current agent i The observation status Policy Network π i and exploration noise R a Select current action

[0025] 2) Agent i Execute current action Get the next state and rewards And gather experience Store in experience cache pool R i .

[0026] Furthermore, the current action The formula is:

[0027]

[0028] in, Represents the agent i strategy network.

[0029] Furthermore, the step S23-1 calculates the strategy loss Loss of the current agent's recent M-step experience, including the following formula:

[0030] y=r+γQ π' (s',a1',a'2^a' N )| ai'＝πi'(oi)

[0031] Loss=(yQ π (s,a1,a2^a N )) 2

[0032] Among them, y represents the expected return of the current action in the current experience; γ represents the reward discount coefficient; r represents the reward value of the target evaluation network.

[0033] Furthermore, in step S23-2: the priority Pr(i) of the current M-step experience is calculated according to the strategy loss Loss of the recent M-step experience, including the following formula:

[0034] p(i)=rank(rank(Loss(i))+rank reverse(T))

[0035]

[0036] Among them: p(i) represents the probability of the current experience, β represents the hyperparameter of adaptive sampling, ranging from 0 to 1, α is used to adjust the priority; rank represents the ranking method for sorting its incoming elements; rank reverse Represents the inverted order method for sorting its incoming elements; T is the corresponding time difference error; Loss represents the value returned by the loss function.

[0037] Furthermore, the step S23-4 specifically includes:

[0038] (1) Calculate each experience action The expected return y j :y j =r j +γQ'(s j+1 ,a1',a'2^a' N ,θ Q' ), where γ represents the reward discount factor; r j Represents the reward value of the target evaluation network;

[0039] (2) Calculate the minimized loss to update the evaluation network parameters: Where L represents minimizing loss and K represents the number of agents;

[0040] (3) Calculate the total reward of the policy network parameters through the gradient formula and update the policy network parameters of the current agent: in represents the gradient; J represents the total reward; K represents the number of agents;

[0041] (4) Every N steps, the target policy network and target evaluation network parameters of all agents are updated by the weighting factor τ: θ Q' =τθ Q +(1-τ)θ Q' ,θ π' =τθ π +(1-τ)θ π' .

[0042] Furthermore, the multi-agent adaptive sampling strategy generation method is used for the collaborative control of multiple traffic signals, wherein the agent is represented by the traffic signal; the deterministic behavior strategy is the traffic light control strategy; the strategy network is represented by the traffic signal's control strategy for the traffic light each time, and the evaluation network is represented by the traffic signal's evaluation of the traffic light control strategy. In the experience, the observation value of the current state represents the real-time information of the traffic environment observed by the traffic signal; the next state represents the traffic signal's predicted value of the traffic environment; the action represents the traffic control action of the traffic signal; and the reward represents the delay time saved for the vehicle.

[0043] The present invention achieves the following technical effects:

[0044] 1. The multi-agent adaptive sampling strategy generation method proposed in the present invention defines the agent and its strategy network, evaluation network and experience according to the application environment, and realizes the automatic generation of multi-agent collaboration strategy. The ET-MADDPG algorithm adopted by the present invention uses the experience priority to extract the experience from the experience pool according to the priority probability on the basis of the MADDPG algorithm, which accelerates the convergence of the model, greatly improves the learning efficiency of the agent, reduces the exploration time of the agent, and improves the efficiency of the effect evaluation of multi-agent collaboration.

[0045] 2. By taking advantage of experience extraction by using experience priority, we can obtain strategies for the agent to learn to obtain the maximum reward faster. We can also limit the agent's exploration space by setting a priority probability threshold, effectively avoiding falling into the local optimum. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The following is a flow chart of an algorithm provided in an embodiment of the present invention.

[0047] Figure 2 This is the experimental result of using MADDPG algorithm;

[0048] Figure 3 The experimental results of ET-MADDPG algorithm are shown in Figure 2. DETAILED DESCRIPTION

[0049] To further illustrate various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.

[0050] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0051] Example 1

[0052] The present invention discloses a multi-agent adaptive sampling strategy generation method, which relates to a multi-agent collaborative strategy automatic generation algorithm, hereinafter referred to as the ET-MADDPG algorithm. The ET-MADDPG algorithm is optimized on the traditional MADDPG algorithm, including: firstly initializing the system model, initializing the strategy network π of each agent i and the target policy network π' i and the evaluation network Q i And the target evaluation network Q i ' and its network parameters θ πi ,θ πi '、θ Qi ,θ πi' , initialize each agent's experience cache pool R i and the motion exploration noise R a . Then, for each action round, initialize the state set S of all agents, and for each step in the round, for each agent in the environment i , perform the following steps: According to the current agent i The observation status Policy Network π i and the motion exploration noise R a Select current action Then the current action As the next step And rewards Experience Store in experience cache pool R i ; Every M steps, for each agent i , calculate the current agent i The strategy loss of the recent M-step experience and calculate the priority P of the current experience r , with priority P r Extract a minibatch of experience for probability and calculate the expected return y through the target evaluation network j Minimize the loss to update the evaluation network parameters and strategy network parameters; after executing a certain number of steps, update the target strategy and evaluation network parameters of all agents. Figure 1 shown.

[0053] First, define the abbreviations of the input and output characters required by the system, so that it can be described later: the policy network π of the agent i , the target policy network π i ', evaluate the network Q i , target evaluation network Q i ', parameter θ πi and θ Qi , experience cache pool R iand the motion exploration noise R a , the state set S of all agents, the experience cache pool R i ,experience Priority P r , the number of experience sampling times T, T is the number of times the experience is sampled and then trained. In this example, the multi-agent adaptive sampling strategy generation method specifically includes:

[0054] Step S1: Build the neural network model of the agent; initialize the policy network π of each agent i (o,θ πi ) and the evaluation network Q i (s,a1,a2^a N ,θ Qi ) and the network parameters θ of the policy network and the evaluation network πi and θ Qi ; Initialize the target policy network π for each agent i '(o,θ πi' ) and the target evaluation network Q i '(s,a1,a2^a N ,θ Qi' ) and the corresponding network parameters θ of the target strategy network and the target evaluation network πi' and θ Qi' The policy network and target policy network are the actor network and targetactor network in the MADDPG algorithm; the evaluation network and target evaluation network are the critic network and targetcritic network in the MADDPG algorithm; initialize each agent i Experience cache pool R i and the motion exploration noise R a .

[0055] Step S2: Execute a set number of rounds. For each round (episode), the neural network model is constructed and simulated to store the experience of all agents, and the experience of each agent is added to its own experience cache pool. The specific steps of the algorithm are as follows:

[0056] For each episode, loop through the following steps:

[0057] Step S21: Initialize the environment and all agent state sets S;

[0058] Step S22: For each step in the round, for each agent in the environment i , perform the following steps:

[0059] 1) According to the current observation state of the agent Policy Network π i and exploration noise R a Select current action

[0060] 2) Agent i Execute current action Get the next state and rewards And the current agent's observation state Current Action Next state and rewards Composition Experience Store in experience cache pool R i .

[0061] Step S23: After each M steps (i.e., experience pool R i The experience of the recent M steps has been stored), and for each agent, the respective neural network is trained: the policy loss of the recent M steps of experience of the current agent and the current experience priority are calculated, and the experience of the set number of samples (minibatch size) is extracted according to the current experience priority as the high-quality experience finally selected; the expected return of each experience action is calculated through the target evaluation network using the extracted high-quality experience of the minibatch size, and the evaluation network parameters and the policy network parameters are updated. In this example, step S23 specifically includes:

[0062] Step S23-1: After executing M steps, for each agent, perform the following steps:

[0063] According to the formula Loss = (yQ π (s,a1,a2^a N )) 2 and y = r + γQ π' (s',a1',a'2^a' N )| ai'＝πi'(oi) Calculate the strategy loss Loss of the current agent's recent M-step experience, where y represents the expected return; γ represents the reward discount coefficient; r j Represents the reward value of the target evaluation network.

[0064] Step S23-2: According to the formula Calculate the priority of the current M-step experience. Where: p(i) represents the probability of the current experience, β represents the hyperparameter of adaptive sampling, ranging from 0 to 1, and α is used to adjust the priority; the calculation formula of p(i) is: p(i) = rank(rank(Loss(i)) + rank reverse (T)), where rank represents the ranking method for sorting the incoming elements; rankreverse Represents the inverted order method for sorting its incoming elements; T is the corresponding time difference error; Loss represents the value returned by the loss function.

[0065] Step S23-3: Determine whether the probability of the priority Pr(i) of the current experience meets the threshold. If it does, extract the experience of the set number of samples (i.e., the minibatch size). This threshold can be generated by the random function rand(0,1) during training and adjusted according to the training evaluation effect. Figure 1 The judgment formula shown in rand(0,1)>Pr(i).

[0066] Step S23-4: Use minibatch experience to learn the policy network and evaluation network.

[0067] In this example, step S23-4 specifically includes:

[0068] (1) Calculate each experience action The expected return y j :y j =r j +γQ'(s j+1 ,a1',a'2^a' N ,θ Q' ), where γ represents the reward discount factor; r j Represents the reward value of the target evaluation network;

[0069] (2) Calculate the minimized loss to update the evaluation network parameters: Where L represents minimizing loss and K represents the number of agents;

[0070] (3) Calculate the total reward of the policy network parameters through the gradient formula and update the policy network parameters of the current agent: in represents the gradient; J represents the total reward; K represents the number of agents;

[0071] (4) Every N steps, the target policy network and target evaluation network parameters of all agents are updated by the weighting factor τ: θ Q' =τθ Q +(1-τ)θ Q' ,θ π' =τθ π +(1-τ)θ π' N is a positive integer, and the value of N can be changed according to the speed of environmental changes to adjust the update frequency of the weighting factor τ.

[0072] Steps S22 and S23 are executed in a loop until the experience of all agents is traversed, the loop ends, and then the multi-agent collaboration strategy and / or evaluation results are output.

[0073] Example 2

[0074] The multi-intelligent adaptive sampling strategy generation method of the present invention can be applied to multiple fields such as intelligent robots, traffic control, flexible manufacturing, and scheduling of manufacturing systems.

[0075] In this embodiment, the multi-agent adaptive sampling strategy generation method is used for the collaborative control of multiple traffic signals, wherein the agent is represented by a traffic signal; the deterministic behavior strategy is a traffic light control strategy; the strategy network is represented by the traffic signal's control strategy for the traffic light each time, and the evaluation network is represented by the traffic signal's evaluation of the traffic light control strategy. In the experience, the observation value of the current state represents the real-time information of the traffic environment observed by the traffic signal; the next state represents the traffic signal's predicted value of the traffic environment; the action represents the traffic control action of the traffic signal; and the reward represents the delay time saved for the vehicle.

[0076] The specific steps include:

[0077] Step S1: define the neural network model of the traffic signal, and initialize the strategy network, target strategy network, evaluation network, target evaluation network, and their respective network parameters for the collaboration of multiple traffic signals; initialize the experience buffer pool and action exploration noise of the agent;

[0078] Step S2: Execute a set number of rounds, each round including the following steps:

[0079] Step S21: Initialize the state set of the environment and all traffic signals;

[0080] Step S22: Each traffic signal learns experience every time a step is executed, and saves the experience learned by each traffic signal into its own experience buffer pool;

[0081] Step S23: After executing M steps, for each traffic signal, train the respective neural network, calculate the policy loss of the current traffic signal in the past M steps, calculate the priority of the current experience, and extract the experience of a set number of samples based on whether the priority meets the threshold as a condition, calculate the minimum loss of the expected return through the target evaluation network, and update the policy network parameters and the evaluation network parameters; after executing N steps, update the target policy network and the target evaluation network through the weighting factor;

[0082] Step S24: loop through steps S22 and S23 until all experiences are traversed and a multi-traffic signal coordination strategy and / or evaluation result is output.

[0083] In other technical fields such as intelligent robots, flexible manufacturing, and scheduling of manufacturing systems, for similar application conditions, the multi-agent adaptive sampling strategy generation method can be referred to to perform automatic generation of multi-agent collaboration strategies.

[0084] Example 3

[0085] In order to verify the implementation and effect of the multi-agent collaborative strategy automatic generation algorithm (hereinafter referred to as the ET-MADDPG algorithm) in the multi-agent adaptive sampling strategy generation method, the following implementation was carried out.

[0086] Experimental content:

[0087] (1) Environment: This experiment is based on the Starcraft2 game;

[0088] (2) Game settings: In the specific game, four Terran agents fight against four Terran agents built into the game Starcraft 2 at level 7 difficulty (the highest difficulty);

[0089] (3) Training settings: In this experiment, the MADDPG algorithm and the ET-MADDPG algorithm were used for experiments. The number of epochs for training was set to 170 times. The battle win rate battle_won_mean (equal to: number of winning games / total number of games) and the loss value loss of the model were statistically analyzed.

[0090] The experimental results data can be found in Figure 2 and Figure 3 .

[0091] in Figure 2 The experimental results using the MADDPG algorithm are given; Figure 3 The experimental results of the ET-MADDPG algorithm are given. In the figure, curve ① is the loss function (LOSS); curve ② is the battle win rate (battle_WON_mean).

[0092] According to the experimental results, the ET-MADDPG algorithm greatly improves the learning efficiency of the intelligent agent compared with the MADDPG algorithm, which randomly extracts experience from the experience pool for learning. After 50 epochs of learning, the ET-MADDPG algorithm is basically in a stable state and has maintained a high combat winning rate. After 170 epochs of the MADDPG algorithm, the intelligent agent is still learning and the winning rate is rising.

[0093] It can be seen that the multi-agent adaptive sampling strategy generation method proposed in the present invention defines the agent and its strategy network, evaluation network and experience according to the application environment, and realizes the automatic generation of multi-agent cooperation strategy. The ET-MADDPG algorithm adopted by the invention uses the experience priority to extract the experience from the experience pool according to the priority probability on the basis of the MADDPG algorithm, accelerates the convergence of the model, greatly improves the learning efficiency of the agent, and reduces the exploration time of the agent. At the same time, by taking advantage of the experience priority for experience extraction, it is possible to obtain a strategy for the agent to learn to obtain the maximum reward faster, and it is also possible to limit the exploration space of the agent by setting the priority probability threshold, effectively avoiding falling into the local optimum.

[0094] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A method for generating a multi-agent adaptive sampling strategy, characterized in that: include: Step S1: Construct the neural network model of the agent and initialize the multi-agent collaborative strategy network π, target strategy network π', evaluation network Q, target evaluation network Q', and their respective network parameters θ π ,θ π' ,θ Q ,θ Q' ; Initialize the agent’s experience buffer pool and action exploration noise R a ; Step S2: Execute a set number of rounds, each round including the following steps: Step S21: Initialize the environment and all agent state sets; Step S22: For each step in the round, each agent learns experience and saves the experience learned by each agent into its own experience cache pool; Step S23: Every M steps, where M is an integer not less than 2, for each agent, train its own neural network and execute: Step S23-1: Calculate the strategy loss Loss of the current agent in the past M steps of experience; Step S23-2: Calculate the priority Pr(i) of the current M-step experience according to the strategy loss Loss of the recent M-step experience; Step S23-3: Determine whether the probability of the priority Pr(i) of the current M-step experience meets the threshold, and if so, extract the experience of minibatch size; Step S23-4: learning the policy network and the evaluation network based on the experience of the extracted minibatch size; Step S24: loop through steps S22 and S23 until all experiences are traversed and the multi-agent collaboration strategy evaluation results are output; The step S23-4 specifically includes: (1) Calculate each experience action The expected return y j :y j =r j +γQ'(s j+1 ,a1',a'2^a' N ,θ Q' ), where γ represents the reward discount factor; r j Represents the reward value of the target evaluation network; (2) Calculate the minimized loss to update the evaluation network parameters: Where L represents minimizing loss and K represents the number of agents; (3) Calculate the total reward of the policy network parameters through the gradient formula and update the policy network parameters of the current agent: in represents the gradient; J represents the total reward; K represents the number of agents; (4) Every N steps, the target policy network and target evaluation network parameters of all agents are updated by the weighting factor τ: θ Q' =τθ Q +(1-τ)θ Q' ,θ π' =τθ π +(1-τ)θ π' .

2. The method for generating a multi-agent adaptive sampling strategy according to claim 1, characterized in that: The step S22 specifically includes: 1) According to the current agent i The observation status Policy Network π i and exploration noise R a Select current action 2) Agent i Execute current action Get the next state and reward r i j , and combine experience Store in experience cache pool R i .

3. The method for generating a multi-agent adaptive sampling strategy according to claim 2, characterized in that: The current action The formula is expressed as: in, Represents the agent i strategy network.

4. The method for generating a multi-agent adaptive sampling strategy according to claim 1, wherein: The step S23-1 calculates the strategy loss Loss of the current agent's recent M-step experience, including the following formula: y=r+γQ π' (s',a1',a'2^a' N )| ai'＝πi'(oi) Loss=(y-Q π (s,a1,a2^a N )) 2 Among them, y represents the expected return of the current action in the current experience; γ represents the reward discount coefficient; r represents the reward value of the target evaluation network.

5. The method for generating a multi-agent adaptive sampling strategy according to claim 4, characterized in that: In step S23-2: the priority Pr(i) of the current M-step experience is calculated according to the strategy loss Loss of the recent M-step experience, including the following formula: p(i)=rank(rank(Loss(i))+rank reverse (T)) Among them: p(i) represents the probability of the current experience, β represents the hyperparameter of adaptive sampling, ranging from 0 to 1, α is used to adjust the priority; rank represents the ranking method for sorting its incoming elements; rank reverse Represents the inverted order method for sorting its incoming elements; T is the corresponding time difference error; Loss represents the value returned by the loss function.

6. The method for generating a multi-agent adaptive sampling strategy according to any one of claims 1 to 5, characterized in that: Used for collaborative control of multiple traffic signals, wherein the intelligent agent is represented by the traffic signal; the deterministic behavior strategy is the traffic light control strategy; the strategy network is represented by the traffic signal's control strategy for the traffic light each time, and the evaluation network is represented by the traffic signal's evaluation of the traffic light control strategy. In the experience, the observation value of the current state represents the real-time information of the traffic environment observed by the traffic signal; the next state represents the traffic environment prediction value of the traffic signal; the action represents the traffic control action of the traffic signal; and the reward represents the delay time saved for the vehicle.

Images