Multi-agent communication information prediction method under ACML model based on neural network
By adopting the ACML model of neural network in multi-agent communication, the communication information prediction of agents is optimized, and the problem of low information utilization efficiency caused by limited communication budget is solved, and the efficient completion of multi-agent tasks in small ball environments is achieved. and team rewards.
Patent Information
- Application Number
- CN202411165159.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-23
AI Technical Summary
The prior art In multi-agent communication, due to the limited communication budget, the information utilization efficiency is low, which affects the strategic learning of the agent, and the existing methods fail to make full use of communication information to alleviate bandwidth limitations.
Adopting an ACML model based on neural networks, including Actor network, information generation network, Critic network, information coordination network and information prediction network, the model is trained to optimize the communication of the agent by defining the action and reward mechanism of the agent in the small ball environment. Information prediction to achieve efficient use of information.
Without limiting the communication budget, the task completion time and team rewards of multiple agents in the ball environment are improved, significantly improving the effectiveness of input data of the information prediction network and the strategic learning effect of the agent.
Smart Images

Figure CN119047509A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of multi-agent communication, and in particular relates to a multi-agent communication information prediction method under an ACML model based on a neural network. Background Art
[0002] Communication information processing among multiple agents is an important part of the field of multi-agent communication, and is of great significance for solving communication bandwidth limitations and achieving efficient use of information.
[0003] Through communication, agents can pass information to reduce the difficulty of finding good strategies. For example, agents exploring different parts of the environment can share observations to alleviate partial observability, and share their intentions to predict each other's actions to deal with non-stationarity. However, in the real world, communication budgets are often limited. For example, in wired packet routing systems, links have limited transmission capacity; in wireless IoT, sensors have limited battery capacity. SchedNet considers the case of shared channels and limited bandwidth. According to the importance of information, only a subset of agents are selected to convey their information, and the total amount of communication information is reduced by limiting the number of agents to communicate; Gated-ACML uses gating units between each agent and the centralized information coordination module to reduce the transmitted communication information. The above methods alleviate the limitation of communication bandwidth by increasing the importance of information that can be communicated, but the insufficient amount of information still has an adverse effect on the agent's strategy learning. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention provides a multi-agent communication information prediction method under the ACML model based on a neural network, which improves the efficiency of multi-agents in utilizing the information obtained through communication when the communication budget is limited, and realizes the efficient utilization of communication information in a small ball environment.
[0005] The method for predicting multi-agent communication information under the ACML model based on a neural network includes the following steps:
[0006] S1: Establish a small ball environment; the small ball environment includes a number of intelligent agents, a number of fixed obstacles and a target landmark, and each intelligent agent performs a simultaneous reaching task, that is, all intelligent agents reach the target landmark at the same time;
[0007] S2: Defines the actions of the agents in the ball environment, local observations, and joint rewards of the agent team;
[0008] The actions of the agent include two types of discrete actions: one is displacement action a i={south,east,north,west,none}, where south, east, north, and west represent the accelerations applied to the agent in the south, east, north, and west directions, respectively, and none represents no acceleration. One is the communication action c i = {communicate, slient}, where communicate means the agent communicates at this time step, and slient means the agent does not communicate at this time step; the local observation of the i-th agent Where p = [p i,x ,p i,y ] represents the position of the agent, p i,x The horizontal coordinate of the agent's position, p i,y The vertical coordinate representing the position of the agent, g = [g x ,g y ] represents the target landmark position, g x Indicates the horizontal coordinate of the target landmark position, g y represents the ordinate of the target landmark location, l represents the communication budget, Indicates the location of the obstacle. The horizontal coordinate representing the obstacle position, The vertical coordinate representing the position of obstacles. The number of obstacles is less than the number of agents. The number of included elements depends on the number of obstacles;
[0009] The joint reward r of the agent team is as shown in formulas (1) to (4):
[0010] r=-λ·r1-r2 (1)
[0011]
[0012] Among them, r1 represents the sum of the distances between the agent and the target landmark, r2 represents the sum of the distances between all agents and the target landmark, λ represents the weight coefficient and λ∈(0,1), d(p i ,g) represents the distance between the i-th agent and the target landmark, d(p j ,g) represents the distance between the jth agent and the target landmark, p i represents the position of the ith agent, p j The position of the jth agent;
[0013] S3: Building ACML model;
[0014] The ACML model includes several Actor networks, several information generation networks, a Critic network, an information coordination network and an information prediction network. One intelligent agent corresponds to one Actor network and one information generation network.
[0015] The information generation network is used to obtain a local observation o of an intelligent agent i , and based on local observations o i Generate the communication information m of the agent i , when the communication action of the agent is slient, the communication information When the agent's communication action is communicate, the communication information m i is a one-dimensional array consisting of two elements;
[0016] The working process of the information generation network is shown in formula (5):
[0017] m i =m(o i ) (5)
[0018] Among them, m represents the information generation network mapping function, m i represents the communication information of the ith agent, o i represents the local observation of the i-th agent;
[0019] The information prediction network is used to generate the communication information m of each intelligent agent based on all information generation networks. i , get the predicted communication information of each agent
[0020] The working process of the information prediction network is shown in formula (6):
[0021]
[0022] Among them, m hh represents the information prediction network mapping function, represents the predicted communication information of the i-th agent generated by the information prediction network, i = 1, 2, ..., N, N represents the number of agents, m h represents the historical communication information of the intelligent agent, m -i Represents the communication information of other agents except the i-th agent;
[0023] The information coordination network is used to communicate the predicted information of each agent generated by the information prediction network. Generate the global information M of each agent i ;
[0024] The working process of the information coordination network is shown in formula (7):
[0025]
[0026] Where M represents the information coordination network mapping function, M i represents the global information of the i-th agent generated by the information coordination network, i = 1, 2, ..., N, Represents the predicted communication information of other agents except the i-th agent;
[0027] The Actor network is used to observe the local observation o of the i-th agent. i With global information M i , generate the agent displacement action a i ;
[0028] The working mode of the Actor network is shown in formula (8):
[0029]
[0030] in, represents the mapping function of the Actor network of the i-th agent, θ i Represents the parameters of the Actor network of the i-th agent;
[0031] The Critic network is used to generate the agent action value Q based on the local observations and displacement actions of all agents a ;
[0032] The working mode of the Critic network is shown in formula (9):
[0033]
[0034] Among them, Q represents the mapping function of the Critic network, ω represents the parameters of the Critic network, is the set of local observations of other agents excluding the ith agent, is the set of displacement actions of other agents excluding the i-th agent;
[0035] The networks used by the Actor network, Critic network, information coordination network, information prediction network and information generation network are all DNN neural networks;
[0036] S4: Unlimited communication budget, so that all agents can fully communicate, train the Actor network, Critic network, information coordination network and information generation network in the ACML model, and obtain the trained Actor network, Critic network, information coordination network and information generation network;
[0037] S4.1: Initialize the Actor network, Critic network, information coordination network and information generation network;
[0038] S4.2: Initialize the experience replay buffer D and set the current number of iterations to 0;
[0039] D is the experience replay buffer containing the most recent experience tuple (o,a,r,o′), o′ i is the i-th agent at o i The local observation at the next time step after , is the set of local observations of the next time step of other agents that do not include the i-th agent;
[0040] S4.3: Determine whether the current number of iterations reaches the maximum number of iterations. If yes, execute S4.13; otherwise, execute S4.4.
[0041] S4.4: Initialize the ball environment;
[0042] S4.5: Determine whether the displacement of the agent in the current ball environment has reached the maximum number of steps set. If yes, increase the number of iterations by 1 and return to S4.3. Otherwise, execute S4.6.
[0043] S4.6: The information generation network corresponding to the i-th agent obtains the local observation o of the i-th agent i , generate the local information m i and transmitted to the information coordination network;
[0044] S4.7: The information coordination network generates the local information m of all agents generated by the network based on the acquired information i , generating the global information M of each agent i And send it to the corresponding agent's Actor network;
[0045] The working process of the information coordination network is shown in formula (10):
[0046] M i =M(m i ,m -i ) (10)
[0047] S4.8: The Actor network of the i-th agent obtains the local observation o of the i-th agent i With global information M i , generate the agent displacement action a i ;
[0048] S4.9: All agents execute the displacement actions generated by their Actor network according to S4.8, and obtain the new local observation o′ and the joint reward r of the agent team;
[0049] S4.10: Store the experience (o, a, r, o′) into the experience replay buffer D;
[0050] S4.11: Extract experience from the experience replay buffer D and update the parameters of the Actor network, Critic network, information coordination network, and information generation network;
[0051] The update of the critic network parameter ω is shown in equations (11)-(15):
[0052]
[0053]
[0054] Among them, L(ω) represents the loss function for updating the parameters of the Critic network. represents the expected function, δ represents the loss function, γ represents the discount rate, is the displacement action value of the agent output by the Critic network, a′ represents the displacement action of the next time step after the displacement action a, μ θ (o′) represents the action function of the local observation o′ in the Actor network at the next time step after the local observation o, ω - Represents the parameters of the target network. The target network structure is the same as the Critic network, but the parameters are different. represents the displacement action value of the next time step after displacement action a, ω new represents the parameters of the updated Critic network, τ represents the hyperparameter, and α represents the step size of the gradient descent. represents the gradient of ω, represents the parameters of the updated target network, a′ i is the i-th agent in a i The displacement action of the next time step after ;
[0055] The Actor network parameters θ of the i-th agent i The update of is shown in equations (15)-(16):
[0056]
[0057] in, Represents the i The gradient of J(θ i ) represents the value function, Expresses expectation and o i Get from D, Represents local observation o i In the action function of the Actor network, Indicates a i The gradient of θ new is the updated Actor network parameter, β is the step size for gradient ascent, represents the gradient of θ, and J(θ) represents the value function;
[0058] Parameters θ of the information coordination network and the information generation network pc Updated by the chain rule, as shown in equations (17)-(18):
[0059]
[0060] in, Represents the pc The gradient of J(θ pc ) represents the value function, Expressing expectations, M i (m1,m2,…,m N θ pc ) represents global information, Indicates M i expectations, is the updated parameter, η is the step size of gradient ascent;
[0061] S4.12: Add 1 to the current time step and return to S4.5;
[0062] S4.13: Record the parameters of the current Actor network, Critic network, information coordination network and information generation network, and obtain the trained Actor network, Critic network, information coordination network and information generation network;
[0063] S5: Without limiting the communication budget, we use the trained Actor network, Critic network, information coordination network and information generation network to obtain several multi-agent communication records, and then establish an information prediction network dataset with communication budget constraints.
[0064] The specific form of the agent communication record is 2×N columns of data, and any row of data is the communication information of all agents at the same time step;
[0065] The method for establishing an information prediction network dataset about communication budget constraints is as follows: by default, communication is performed at the first time step of each multi-agent communication record, and n communication information is retained at a fixed frequency, where n is the communication budget; the time step where the random number k is located is taken as the communication moment that needs to be predicted for this multi-agent communication record, and the communication information at and after time k in this multi-agent communication record is cleared, and the processed communication record is used as the input feature of the dataset, and the real communication information at time k is used as the label, to obtain the information prediction network dataset about communication budget constraints, and the multi-agent communication record in the dataset is recorded as m h,b , b is the number of the multi-agent communication record in the dataset;
[0066] S6: using the information prediction network dataset about the communication budget constraint to train the information prediction network in the ACML model, and obtaining the parameters of the trained information prediction network;
[0067] S7: impose communication budget constraints, and the communication between multiple agents is carried out at a set fixed frequency. The parameters of the trained information prediction network, actor network, critic network, information coordination network, and information generation network are loaded into the ACML model. The ACML model is executed to obtain the agent running trajectory and team reward R.
[0068] The calculation method of the team reward R is shown in formula (19):
[0069]
[0070] Among them, Z represents the time step of this running trajectory, γ z-1 represents the discount rate γ to the z-1th power, r z represents the joint reward of the team of agents at the zth time step.
[0071] The beneficial effects of the multi-agent communication information prediction method under the ACML model based on a neural network proposed by the present invention are as follows:
[0072] 1. The present invention establishes a data set based on the real communication information of the intelligent agent without limiting the communication budget, thereby ensuring the validity of the information prediction network input data during training.
[0073] 2. The present invention achieves the reduction of multi-agent task completion time and significant improvement of team rewards in a small ball environment by predicting the agent's information at the moment when communication is prohibited. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] Figure 1 Flow chart of the multi-agent communication information prediction method under the ACML model based on neural network in this embodiment;
[0075] Figure 2 This is a schematic diagram of the ball environment in this embodiment;
[0076] Figure 3 This is an ACML model architecture diagram in which an information prediction network is added in this embodiment;
[0077] Figure 4 This is an ACML model architecture diagram that does not include an information prediction network in this embodiment;
[0078] Figure 5 This is a flow chart of ACML model training that does not include an information prediction network in this embodiment;
[0079] Figure 6 This is an iterative curve diagram of information prediction network loss in this embodiment;
[0080] Figure 7 This is a running trajectory diagram of the intelligent agent containing the information prediction network in the small ball environment of this embodiment. DETAILED DESCRIPTION
[0081] In order to explain the technical solution of the present invention more clearly, the present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0082] The multi-agent communication information prediction method under the ACML model based on neural network, such as Figure 1 As shown, the following steps are included:
[0083] S1: Establish a small ball environment; the small ball environment includes a number of intelligent agents, a number of fixed obstacles and a target landmark, and each intelligent agent performs a simultaneous reaching task, that is, all intelligent agents reach the target landmark at the same time;
[0084] In this embodiment, taking two agents as an example, the ball environment established is as follows: Figure 2 As shown in Figure 1, the initial positions of the agent and the target landmark are random, the obstacle is always located near the line from a certain agent to the target landmark, and the communication budget represents the remaining number of communications.
[0085] S2: Defines the actions of the agents in the ball environment, local observations, and joint rewards of the agent team;
[0086] The actions of the agent include two types of discrete actions: one is displacement action a i ={south,east,north,west,none}, where south, east, north, and west represent the accelerations applied to the agent in the south, east, north, and west directions, respectively, and none represents no acceleration. One is the communication action c i= {communicate, slient}, where communicate means the agent communicates at this time step, and slient means the agent does not communicate at this time step; the local observation of the i-th agent Where p = [p i,x ,p i,y ] represents the position of the agent, p i,x The horizontal coordinate of the agent's position, p i,y The vertical coordinate representing the position of the agent, g = [g x ,g y ] represents the target landmark position, g x Indicates the horizontal coordinate of the target landmark position, g y represents the ordinate of the target landmark location, l represents the communication budget, Indicates the location of the obstacle. The horizontal coordinate representing the obstacle position, The vertical coordinate representing the position of obstacles. The number of obstacles is less than the number of agents. The number of included elements depends on the number of obstacles.
[0087] In this embodiment, the simultaneous arrival task requires two agents to arrive at the target landmark at the same time, and the joint reward r of the agent team is shown in formulas (1) to (4):
[0088] r=-λ·r1-r2 (1)
[0089]
[0090]
[0091] Among them, r1 represents the sum of the distances between the agent and the target landmark, r2 represents the sum of the distances between all agents and the target landmark, λ represents the weight coefficient and λ∈(0,1), d(p i ,g) represents the distance between the i-th agent and the target landmark, d(p j ,g) represents the distance between the jth agent and the target landmark, p i represents the position of the ith agent, p j The position of the jth agent, the team joint reward encourages the agents to reach the target landmark at the same time through r2 and adjustment λ;
[0092] S3: Building ACML model;
[0093] like Figure 3As shown, the ACML model includes several Actor networks, several information generation networks, a Critic network, an information coordination network and an information prediction network. One intelligent agent corresponds to one Actor network and one information generation network.
[0094] The information generation network is used to obtain a local observation o of an intelligent agent i , and based on local observations o i Generate the communication information m of the agent i , when the communication action of the agent is slient, the communication information When the agent's communication action is communicate, the communication information m i is a one-dimensional array consisting of two elements;
[0095] The working process of the information generation network is shown in formula (5):
[0096] m i =m(o i ) (5)
[0097] Among them, m represents the information generation network mapping function, m i represents the communication information of the ith agent, o i represents the local observation of the i-th agent;
[0098] The information prediction network is used to generate the communication information m of each intelligent agent based on all information generation networks. i , get the predicted communication information of each agent
[0099] The working process of the information prediction network is shown in formula (6):
[0100]
[0101] Among them, m hh represents the information prediction network mapping function, represents the predicted communication information of the i-th agent generated by the information prediction network, i = 1, 2, ..., N, N represents the number of agents, m h represents the historical communication information of the intelligent agent, m -i Represents the communication information of other agents except the i-th agent;
[0102] The information coordination network is used to communicate the predicted information of each agent generated by the information prediction network. Generate the global information M of each agent i ;
[0103] The working process of the information coordination network is shown in formula (7):
[0104]
[0105] Where M represents the information coordination network mapping function, M i represents the global information of the i-th agent generated by the information coordination network, i = 1, 2, ..., N, Represents the predicted communication information of other agents except the i-th agent;
[0106] The Actor network is used to observe the local observation o of the i-th agent. i With global information M i , generate the agent displacement action a i ;
[0107] The working mode of the Actor network is shown in formula (8):
[0108]
[0109] in, represents the mapping function of the Actor network of the i-th agent, θ i Represents the parameters of the Actor network of the i-th agent;
[0110] The Critic network is used to generate the agent action value Q based on the local observations and displacement actions of all agents a ;
[0111] The working mode of the Critic network is shown in formula (9):
[0112]
[0113] Among them, Q represents the mapping function of the Critic network, ω represents the parameters of the Critic network, is the set of local observations of other agents excluding the ith agent, is the set of displacement actions of other agents excluding the i-th agent;
[0114] The networks used by the Actor network, Critic network, information coordination network, information prediction network and information generation network are all DNN neural networks;
[0115] ,The network structure of DNN neural network includes an input layer, two hidden layers and an output layer connected in sequence, and the activation function is ReLU;
[0116] S4: Unlimited communication budget, so that all agents can fully communicate, train the Actor network, Critic network, information coordination network and information generation network in the ACML model, and obtain the trained Actor network, Critic network, information coordination network and information generation network;
[0117] In this embodiment, the ACML model that does not include the information prediction network is as follows Figure 4 As shown in the figure, the dotted line indicates the flow of communication information. The specific training process is as follows Figure 5 As shown:
[0118] S4.1: Initialize the Actor network, Critic network, information coordination network and information generation network;
[0119] S4.2: Initialize the experience replay buffer D and set the current number of iterations to 0;
[0120] In this embodiment, D is the experience replay buffer containing the most recent experience tuple (o, a, r, o′), o i ′ is the i-th agent at o i The local observation at the next time step after , is the set of local observations of the next time step of other agents that do not include the i-th agent;
[0121] S4.3: Determine whether the current number of iterations reaches the maximum number of iterations. If yes, execute S4.13; otherwise, execute S4.4.
[0122] S4.4: Initialize the ball environment;
[0123] In this implementation, the positions of the agents and the target landmarks are randomly distributed, the obstacles are always located near the line from a certain agent to the target landmark, and the number of obstacles is less than the number of agents;
[0124] S4.5: Determine whether the displacement of the agent in the current ball environment has reached the maximum number of steps set. If yes, increase the number of iterations by 1 and return to S4.3. Otherwise, execute S4.6.
[0125] In this embodiment, the maximum number of steps of the agent's displacement is set to 25;
[0126] S4.6: The information generation network corresponding to the i-th agent obtains the local observation o of the i-th agent i , generate the local information m i and transmitted to the information coordination network;
[0127] S4.7: The information coordination network generates the local information m of all agents generated by the network based on the acquired informationi , generating the global information M of each agent i And send it to the corresponding agent's Actor network;
[0128] In this embodiment, the working process of the information coordination network is shown in formula (10):
[0129] M i =M(m i ,m -i ) (10)
[0130] S4.8: The Actor network of the i-th agent obtains the local observation o of the i-th agent i With global information M i , generate the agent displacement action a i , the number of Actor networks is equal to the number of agents;
[0131] S4.9: All agents execute the displacement actions generated by their Actor network according to S4.8, and obtain the new local observation o′ and the joint reward r of the agent team;
[0132] S4.10: Store the experience (o, a, r, o′) into the experience replay buffer D;
[0133] S4.11: Extract experience from the experience replay buffer D and update the parameters of the Actor network, Critic network, information coordination network, and information generation network;
[0134] In this embodiment, the update of the critic network parameter ω is shown in equations (11)-(15):
[0135]
[0136]
[0137] Among them, L(ω) represents the loss function for updating the parameters of the Critic network. represents the expected function, δ represents the loss function, γ represents the discount rate, is the displacement action value of the agent output by the Critic network, a′ represents the displacement action of the next time step after the displacement action a, μ θ (o′) represents the action function of the local observation o′ in the Actor network at the next time step after the local observation o, ω - Represents the parameters of the target network. The target network structure is the same as the Critic network, but the parameters are different. The purpose is to alleviate the deviation caused by the bootstrapping of the Critic network. represents the displacement action value of the next time step after displacement action a, ω newrepresents the parameters of the updated Critic network, τ represents the hyperparameter, which is used to weighted update the parameters of the target network, and α represents the step size of the gradient descent. represents the gradient of ω, represents the parameters of the updated target network, a i ′ is the i-th agent in a i The displacement action of the next time step after ;
[0138] In this embodiment, the Actor network parameter θ of the i-th agent i The update of is shown in equations (15)-(16):
[0139]
[0140] in, Represents the i The gradient of J(θ i ) represents the value function, Expresses expectation and o i Get from D, Represents local observation o i In the action function of the Actor network, Indicates a i The gradient of θ new is the updated Actor network parameter, β is the step size for gradient ascent, represents the gradient of θ, and J(θ) represents the value function;
[0141] In this embodiment, the parameters θ of the information coordination network and the information generation network are pc Updated by the chain rule, as shown in equations (17)-(18):
[0142]
[0143] in, Represents the pc The gradient of J(θ pc ) represents the value function, Expressing expectations, M i (m1,m2,…,m N θ pc ) represents global information, Indicates M i expectations, is the updated parameter, η is the step size of gradient ascent;
[0144] S4.12: Add 1 to the current time step and return to S4.5;
[0145] S4.13: Record the parameters of the current Actor network, Critic network, information coordination network and information generation network, and obtain the trained Actor network, Critic network, information coordination network and information generation network;
[0146] S5: Without limiting the communication budget, we use the trained Actor network, Critic network, information coordination network and information generation network to obtain several multi-agent communication records, and then establish an information prediction network dataset with communication budget constraints.
[0147] In this embodiment, the specific form of the agent communication record is 2×N columns and no more than 25 rows of data. Any row of data is the communication information of all agents at the same time step. Finally, the trained Actor network, Critic network, information coordination network and information generation network are executed to obtain 10,000 multi-agent communication records.
[0148] In this embodiment, the process of executing the trained Actor network, Critic network, information coordination network and information generation network is as follows: (1) The information generation network of the ith agent obtains the local observation o of the ith agent i , generate communication information m i and transmit it to the information coordination network; (2) the information coordination network predicts all the communication information m generated by the network based on the acquired information i , generate global information M i And send it to the corresponding agent (3) The Actor network of the i-th agent obtains the local observation o of the i-th agent i With global information M i , generate the agent displacement action a i ; (4) Repeat steps 1-3 until the maximum number of time steps is reached, and obtain the agent communication information m1, m2, ..., m N trajectories;
[0149] In this embodiment, each multi-agent communication record is processed to establish an information prediction network data set about the communication budget limit. The method is as follows: the first time step of each multi-agent communication record is defaulted to communication, and n communication information is retained at a fixed frequency, where n is the communication budget; the time step where the random number k is located is taken as the communication moment to be predicted for this record, and the communication data at and after time k in this record is cleared, k∈(2,…,25), and the processed record is used as the input feature of the data set, and the real communication information at time k is used as the label to obtain the information prediction network data set about the communication budget limit. The multi-agent communication record in the data set is recorded as m h,b , b is the number of the multi-agent communication record in the dataset, b∈(1,…,10000).
[0150] S6: using the information prediction network dataset about the communication budget constraint to train the information prediction network in the ACML model, and obtaining the parameters of the trained information prediction network;
[0151] In this embodiment, the information prediction network uses a DNN neural network, including an input layer, two hidden layers, and an output layer, and the activation function is ReLU. The Adam optimizer is used, the learning rate is set to 0.01, the data batch input each time is 512, and the loss function is the mean square error loss function. After reaching the maximum number of iterations, the information prediction network parameters and the loss function iteration curve are output. The loss function iteration curve is as follows: Figure 6 shown.
[0152] S7: Imposing communication budget constraints, communication between multiple agents is carried out at a set fixed frequency, loading the parameters of the trained information prediction network, actor network, critic network, information coordination network and information generation network into the ACML model, and executing the ACML model to obtain the agent operation trajectory and team reward R.
[0153] In this embodiment, the calculation method of the team reward R is shown in formula (19):
[0154]
[0155] Among them, Z represents the time step of this running trajectory, γ z-1 represents the discount rate γ to the z-1th power, r z represents the joint reward of the agent team at the zth time step;
[0156] In this embodiment, after adding the information prediction network, the multi-agent running trajectory is as follows: Figure 7 As shown in Figure 2, it can be seen that when r2 is large, the agent close to the target landmark moves forward in a roundabout way while avoiding obstacles; the agent far away from the target landmark always moves forward in a straight line.
[0157] Table 1 shows the comparison of the ACML model with and without the information prediction module in the case of communication constraints, when the communication budget is 5 and 10 respectively, on the simultaneous completion of the task by multiple agents in the small ball environment, including team rewards and the number of steps to complete the task. The data is the average of ten experiments.
[0158] Table 1 Comparison of the results of the ACML model of the present invention and the model without information prediction module
[0159]
[0160] As shown in Table 1, when the communication budget is 5 and 10, the multi-agents can complete the simultaneous arrival task before the maximum number of steps in the ACML model with or without the information prediction module. After adding the information prediction module, the multi-agents perform better in terms of team rewards and the number of steps to complete the task, and the effect is more obvious when the communication budget is reduced. This shows that adding the information prediction module can enable the multi-agents to make better use of the known communication information.
Claims
1. A method for predicting multi-agent communication information under the ACML model based on a neural network, characterized in that: The following steps are involved: S1: Establish a small ball environment; the small ball environment includes a number of intelligent agents, a number of fixed obstacles and a target landmark, and each intelligent agent performs a simultaneous reaching task, that is, all intelligent agents reach the target landmark at the same time; S2: Defines the actions of the agents in the ball environment, local observations, and joint rewards of the agent team; S3: Building ACML model; S4: Unlimited communication budget, so that all agents can fully communicate, train the Actor network, Critic network, information coordination network and information generation network in the ACML model, and obtain the trained Actor network, Critic network, information coordination network and information generation network; S5: Without limiting the communication budget, we use the trained Actor network, Critic network, information coordination network and information generation network to obtain several multi-agent communication records, and then establish an information prediction network dataset with communication budget constraints. S6: using the information prediction network dataset about the communication budget constraint to train the information prediction network in the ACML model, and obtaining the parameters of the trained information prediction network; S7: Imposing communication budget constraints, communication between multiple agents is carried out at a set fixed frequency, loading the parameters of the trained information prediction network, actor network, critic network, information coordination network and information generation network into the ACML model, and executing the ACML model to obtain the agent operation trajectory and team rewards.
2. The method for predicting multi-agent communication information under the ACML model based on a neural network according to claim 1, characterized in that: The actions of the agent described in S2 include two types of discrete actions: one is the displacement action a i ={south,east,north,west,none}, where south, east, north, and west represent the accelerations applied to the agent in the south, east, north, and west directions, respectively, and none represents no acceleration. One is the communication action c i = {communicate, slient}, where communicate means the agent communicates at this time step, and slient means the agent does not communicate at this time step; the local observation of the i-th agent Where p = [p i,x ,p i,y ] represents the position of the agent, p i,x The horizontal coordinate of the agent's position, p i,y The vertical coordinate representing the position of the agent, g = [g x ,g y ] represents the target landmark position, g x Indicates the horizontal coordinate of the target landmark position, g y represents the ordinate of the target landmark location, l represents the communication budget, Indicates the location of the obstacle. The horizontal coordinate representing the obstacle position, The vertical coordinate representing the position of obstacles. The number of obstacles is less than the number of agents. The number of included elements depends on the number of obstacles.
3. The method for predicting multi-agent communication information under the ACML model based on a neural network according to claim 2, characterized in that: The joint reward r of the agent team in S2 is shown in equations (1) to (4): r=-λ·r1-r2 (1) Among them, r1 represents the sum of the distances between the agent and the target landmark, r2 represents the sum of the distances between all agents and the target landmark, λ represents the weight coefficient and λ∈(0,1), d(p i ,g) represents the distance between the i-th agent and the target landmark, d(p j ,g) represents the distance between the jth agent and the target landmark, p i represents the position of the ith agent, p j The position of the jth agent.
4. The method for predicting multi-agent communication information under the ACML model based on a neural network according to claim 3, characterized in that: The ACML model described in S3 includes several Actor networks, several information generation networks, a Critic network, an information coordination network and an information prediction network. One agent corresponds to one Actor network and one information generation network. The information generation network is used to obtain a local observation o of an intelligent agent i , and based on local observations o i Generate the communication information m of the agent i , when the communication action of the agent is slient, the communication information When the agent's communication action is communicate, the communication information m i is a one-dimensional array consisting of two elements; The working process of the information generation network is shown in formula (5): m i =m(o i ) (5) Among them, m represents the information generation network mapping function, m i represents the communication information of the ith agent, o i represents the local observation of the i-th agent; The information prediction network is used to generate the communication information m of each intelligent agent based on all information generation networks. i , get the predicted communication information of each agent The working process of the information prediction network is shown in formula (6): Among them, m hh represents the information prediction network mapping function, represents the predicted communication information of the i-th agent generated by the information prediction network, i = 1, 2, ..., N, N represents the number of agents, m h represents the historical communication information of the intelligent agent, m -i Represents the communication information of other agents except the i-th agent; The information coordination network is used to communicate the predicted information of each agent generated by the information prediction network. Generate the global information M of each agent i ; The working process of the information coordination network is shown in formula (7): Where M represents the information coordination network mapping function, M i represents the global information of the i-th agent generated by the information coordination network, i = 1, 2, ..., N, Represents the predicted communication information of other agents except the i-th agent; The Actor network is used to observe the local observation o of the i-th agent. i With global information M i , generate the agent displacement action a i ; The working mode of the Actor network is shown in formula (8): in, represents the mapping function of the Actor network of the i-th agent, θ i Represents the parameters of the Actor network of the i-th agent; The Critic network is used to generate the agent action value Q based on the local observations and displacement actions of all agents a ; The working mode of the Critic network is shown in formula (9): Among them, Q represents the mapping function of the Critic network, ω represents the parameters of the Critic network, is the set of local observations of other agents excluding the ith agent, is the set of displacement actions of other agents excluding the i-th agent; The networks used by the Actor network, Critic network, information coordination network, information prediction network and information generation network are all DNN neural networks.
5. The method for predicting multi-agent communication information under the ACML model based on a neural network according to claim 4, characterized in that: S4 specifically includes: S4.1: Initialize the Actor network, Critic network, information coordination network and information generation network; S4.2: Initialize the experience replay buffer D and set the current number of iterations to 0; D is the experience replay buffer containing the most recent experience tuple (o,a,r,o′), o i ′ is the i-th agent at o i The local observation at the next time step after , is the set of local observations of the next time step of other agents that do not include the i-th agent; S4.3: Determine whether the current number of iterations reaches the maximum number of iterations. If yes, execute S4.13; otherwise, execute S4.
4. S4.4: Initialize the ball environment; S4.5: Determine whether the displacement of the agent in the current ball environment has reached the maximum number of steps set. If yes, increase the number of iterations by 1 and return to S4.
3. Otherwise, execute S4.
6. S4.6: The information generation network corresponding to the i-th agent obtains the local observation o of the i-th agent i , generate the local information m i and transmitted to the information coordination network; S4.7: The information coordination network generates the local information m of all agents generated by the network based on the acquired information i , generating the global information M of each agent i And send it to the corresponding agent's Actor network; The working process of the information coordination network is shown in formula (10): M i =M(m i ,m -i ) (10) S4.8: The Actor network of the i-th agent obtains the local observation o of the i-th agent i With global information M i , generate the agent displacement action a i ; S4.9: All agents execute the displacement actions generated by their Actor network according to S4.8, and obtain the new local observation o′ and the joint reward r of the agent team; S4.10: Store the experience (o, a, r, o′) into the experience replay buffer D; S4.11: Extract experience from the experience replay buffer D and update the parameters of the Actor network, Critic network, information coordination network, and information generation network; S4.12: Add 1 to the current time step and return to S4.5; S4.13: Record the parameters of the current Actor network, Critic network, information coordination network, and information generation network, and obtain the trained Actor network, Critic network, information coordination network, and information generation network.
6. The method for predicting multi-agent communication information under the ACML model based on a neural network according to claim 5, characterized in that: The update of the critic network parameter ω in S4.11 is shown in equations (11)-(15): Among them, L(ω) represents the loss function for updating the parameters of the Critic network. represents the expected function, δ represents the loss function, γ represents the discount rate, is the displacement action value of the agent output by the Critic network, a′ represents the displacement action of the next time step after the displacement action a, μ θ (o′) represents the action function of the local observation o′ in the Actor network at the next time step after the local observation o, ω - Represents the parameters of the target network. The target network structure is the same as the Critic network, but the parameters are different. represents the displacement action value of the next time step after displacement action a, ω new represents the parameters of the updated Critic network, τ represents the hyperparameter, and α represents the step size of the gradient descent. represents the gradient of ω, represents the parameters of the updated target network, a i ′ is the i-th agent in a i The displacement action of the next time step after ; The Actor network parameters θ of the i-th agent i The update of is shown in equations (15)-(16): in, Represents the i The gradient of J(θ i ) represents the value function, Expresses expectation and o i Get from D, Represents local observation o i In the action function of the Actor network, Indicates a i The gradient of θ new is the updated Actor network parameter, β is the step size for gradient ascent, ▽ θ represents the gradient of θ, and J(θ) represents the value function; Parameters θ of the information coordination network and the information generation network pc Updated by the chain rule, as shown in equations (17)-(18): in, Represents the pc The gradient of J(θ pc ) represents the value function, Expressing expectations, M i (m1,m2,…,m N θ pc ) represents global information, Indicates M i expectations, is the updated parameter, and η is the step size of gradient ascent.
7. The method for predicting multi-agent communication information under the ACML model based on a neural network according to claim 4, characterized in that: The specific form of the agent communication record in S5 is 2×N columns of data, and any row of data is the communication information of all agents at the same time step.
8. The method for predicting multi-agent communication information under the ACML model based on a neural network according to claim 7, characterized in that: The method for establishing the information prediction network dataset about the communication budget limit described in S5 is as follows: by default, communication is performed at the first time step of each multi-agent communication record, and n pieces of communication information are retained at a fixed frequency, where n is the communication budget; the time step where the random number k is located is taken as the communication moment to be predicted for this multi-agent communication record, and the communication information at and after time k in this multi-agent communication record is cleared, and the processed communication record is used as the input feature of the dataset, and the real communication information at time k is used as the label, so as to obtain the information prediction network dataset about the communication budget limit, and the multi-agent communication record in the dataset is recorded as m h,b , b is the number of the multi-agent communication record in the dataset.
9. The method for predicting multi-agent communication information under the ACML model based on a neural network according to claim 2, characterized in that: The calculation method of the team reward R described in S7 is shown in formula (19): Among them, Z represents the time step of this running trajectory, γ z-1 represents the discount rate γ to the z-1th power, r z represents the joint reward of the team of agents at the zth time step.
Citation Information
Patent Citations
Cargo transportation system based on multi-agent reinforcement learning
CN110991972A
Mixed-experience multi-agent reinforcement learning motion planning method
CN113341958A
Multi-agent cooperative communication strategy training system and method based on teammate perception
CN114757092A
Heterogeneous cluster zero communication target allocation method based on multi-agent reinforcement learning
CN116340737A
Multi-agent collaborative decision reinforcement learning method, system and device
CN117151206A
