Multi-agent reinforcement learning systems

Abstract

The present approach involves a method of training a first machine learning agent in a multi-agent reinforcement learning framework. The framework comprises a plurality of other machine learning agents each implementing their own respective policy. The method comprises determining a plurality of joint actions, each comprising a different combination of respective actions to be taken by each of the plurality of machine learning agents based on their respective policies, and calculating, based on the determined plurality of joint actions and an action of the first machine learning agent, the mean value function of the action of the first machine learning agent.

Description

[0001] MULTI-AGENT REINFORCEMENT LEARNING SYSTEMS

[0002] TECHNICAL FIELD

[0003] This invention relates to multi-agent reinforcement learning (MARL).

[0004] BACKGROUND

[0005] One approach to multi-agent reinforcement learning is to perform centralised training with decentralised execution (CT-DE). This approach can provide certain benefits of the centralised training method for solving multi-agent systems, while allowing actors to execute their actions using only local observations. Multi-agent systems (MASs) modelled in this way may involve coordination among agents. Centralised training has typically demonstrated particularly good performance in multi-agent reinforcement learning. However, CT relies on agents learning from one-off observations of other agents' actions at a given state. Since MARL agents explore and progressively update their policies, the one-off observations may provide little information about the typical behaviour of other agents and consequently, little information about the expected return for a given action. As such, CT methods may have high variance and can provide error-prone estimates which harm learning.

[0006] Further, centralised learning is typically computationally expensive and can often require large numbers of samples to complete training. Scaling such CT methods therefore typically requires value function (VF) factorisations which in turn require strong assumptions (e.g. monotonicity in QMIX). This can lead to poor performance when violated. Recent results have indicated that in some settings centralised learning can hinder learning. There are cases in which centralised learning is greatly outperformed by independent learning (IL) algorithms. IL algorithms typically learn quickly and are easy to tune.

[0007] Independent learning algorithms have other disadvantages, however. First, they do not explicitly account for the actions of other agents. Unlike single agent reinforcement learning (RL), in MASs the collective of agents is typically required to coordinate to successfully solve the task. For independent learning methods, the presence of other RL agents that are not accounted for produces the appearance of a non-stationary environment from the perspective of a given agent. This degrades the strength of the algorithm because it causes convergence guarantees to be lost.

[0008] A second issue for IL algorithms is that they are unable to distinguish between (i) randomness due to the stochasticity of the modelled environment and (ii) randomness produced by the explorative actions of RL agents. This hinders the ability of independent learning methods to learn optimal policies in some environments.

[0009] One difficulty in implementing MARL algorithms is their high sample complexity. Early MARL algorithms such as QMIX (see “QMIX: Monotonic Value Function Factorisation for Deep MultiAgent Reinforcement Learning”, Rashid et al., 2018) have been out-performed recently by algorithms such as MAPPO (see “Is Independent Learning All You Need in the StarCraft MultiAgent Challenge?”, de Witt et al., 2021). QMIX typically imposes strong conditions on the structure of the system rewards which guide learning. However, these conditions are often violated, which in turn causes QMIX to fail. This can be interpreted as QMIX simply underperforming in comparison to MAPPO. More generally, QMIX has a number of aspects that differ to MAPPO which may cause differences in performance.

[0010] Another difficulty in implementing MARL algorithms is that they can exhibit sub-optimal convergence. This has been documented the prior art cited above and in “Multi-agent reinforcement learning: Independent vs. cooperative agents”, Tan et al., 1993. These papers indicate that centralised training - decentralised execution MARL algorithms can fail in some multiagent settings where IL MARL algorithms succeed. Conversely, there are some settings where the opposite is true.

[0011] There is a need for an improved approach to MARL that computes accurate VF estimates which incorporate the exact behaviour of other agents while maintaining a decentralised critic. It would also be beneficial to provide a method for planned exploration with reduced VF variance. Furthermore, it would be advantageous to have an approach to MARL that addressed the typically low sample efficiency of MARL and / or the risk of MARL converging on suboptimal solutions.

[0012] SUMMARY

[0013] According to one aspect there is provided a method of training a first machine learning agent in a multi-agent reinforcement learning framework, the framework comprising a plurality of other machine learning agents each implementing their own respective policy, the method comprising: determining a plurality of joint actions each comprising a different combination of respective actions to be taken by each of the plurality of machine learning agents based on their respective policies; and calculating, based on the determined plurality of joint actions and an action of the first machine learning agent, the mean value function of the action of the first machine learning agent. The method may comprise receiving current state information about each of the plurality of other machine learning agents; and evaluating each respective policy given the current state information to provide a respective action of the other machine learning agent, where each different possible combination of actions provides a different joint action. This can assist in identifying a preferred solution.

[0014] The value function may be a real number which represents the expected sum of rewards as a result of the action of the first machine learning agent in the current state while accounting for one of the plurality of joint actions of the plurality of other machine learning agents. This can assist in identifying an optimal solution.

[0015] The mean value function may be a mean of the total value function of a plurality of joint actions being considered over the number of the plurality of joint actions being considered. This can assist in identifying an optimal solution.

[0016] The method steps of determining and calculating may be repeated for a different action of the first machine learning agent. This can assist in providing an optimised solution for training the MARL algorithm.

[0017] The value function may be calculated using a value function equation arranged to take one of the determined plurality of joint actions as an input. The value function equation may be written as: Vi’. S x A±x ... x ^ R ; where Vtis the value function, S is the current state, A±- ANrepresent each action of a joint action for N other machine learning agents, and IR is the real number output which represents the expected sum of rewards as a result of the action i of the first machine learning agent. This can assist in providing a solution which takes into account an exact action of multiple other agents.

[0018] The value function equation may be implemented as a multi-layer perceptron with the joint action appended to the current state term. This can assist in implementing the proposed approach to identify an optimal solution.

[0019] The method may comprise updating the first machine learning agent and optimising the policy of the first machine learning agent based on the calculated mean value function. This can assist in training the first machine learning agent to identify an optimal solution.

[0020] According to another aspect there is provided a device for implementing a policy of a first machine learning agent, the policy optimised by calculating a mean value function of one or more actions of the first machine learning agent within a multi-agent reinforcement learning framework comprising a plurality of other machine learning agents, where the mean value function is calculated to account for a plurality of joint actions of the plurality of other machine learning agents. The device may implement the policy optimised according to the method of any of the preceding claims.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0023] Figure 1 shows a simple example of a situation where the present invention may be advantageous.

[0024] Figure 2 shows a schematic diagram of the combined joint action and state in a network. Figure 3 shows an example algorithm.

[0025] Figure 4 shows a computer system suitable for implementing the present invention.

[0026] DETAILED DESCRIPTION

[0027] The proposed approach succeeds in computing accurate value function, VF, estimates that incorporate the exact behaviour of other agents while maintaining a decentralised critic. At its core, the proposed approach comprises a method of embedding shared knowledge in the agents' individual VFs. The proposed method dramatically reduces the variance of the VFs during training and enables agents to quickly determine their optimal joint policy given the influence of other agents.

[0028] The proposed approach is placed within the context of Multi-agent Reinforcement Learning, MARL. MARL is a setting in which a collection of decision-making agents observe a state of the environment and attempt to jointly take actions to maximise their utility. Concretely, at timestep t, N agents each observe a state of the environment, st. Given this observation, each agent i samples its respective policy to select an action atl~nl(- |st). The collection of each agent’s action forms a joint-action at. Actuating this joint-action in the environment results in the environment emitting a joint-reward rt~R(st, at) and transitioning to a new state st+1~P(st,at,st+1). The goal of each agent is to optimise the expected long-term sum of rewards, EE“=0t+ / c]. for all time-steps t. Crucially, achieving optimal outcomes with respect to this goal may require agents to coordinate; a central component of MARL. In summary, the settings consists of an environment E (which has a set of states S, allowed actions A, a reward function r, and transition dynamics P), and a collection of decision making agents with jointpolicy 7T.

[0029] For illustration of these concepts, a simple example of a MARL environment 101 is shown in Figure 1. This is a traffic junction in which agents 102 are represented by squares. The squares can move on orthogonal roads 103. The state of the environment consists of the locations of the agent squares 102 on the roads 103 at any given time. Each agent’s 102 action at an iteration is to either move forward or stand still. The so-called joint-action consists of the collection of these actions. For example, a set of actions where each agent 102 takes a respective action. A typical reward function in this MARL example might reward forward movement of an agent 102 and severely penalise accidents where the roads 103 intersect 104.

[0030] Many contemporary MARL algorithms fall within the Centralised Training -- Decentralised Execution, CT-DE, framework. For example, VDN, QMIX, MAPPO, and HATRPO algorithms. A central feature of this CT-DE framework is a centralised value function, also called a centralised critic, that is used to aid coordination between agents. The critic is privy to extra information which aids in learning, for example the global state of the environment, while agents still execute their policies independently. That is, agents still execute their policies based on their own local observations without consultation regarding other agents’ actions.

[0031] The proposed approach consists of two core components: (1) a value function which takes the joint action as input and (2) marginalisation of the actions of all agents except that of the tth agent. That is, when updating the parameters of the tth agent. The proposed approach focuses on training a first machine learning agent in a multi-agent reinforcement learning framework. The framework comprises a plurality of other machine learning agents, each implementing their own respective policy. The method comprises determining a plurality of joint actions, each comprising a different combination of respective actions to be taken by each of the plurality of machine learning agents based on their respective policies. The method also comprises calculating, based on the determined plurality of joint actions and an action of the first machine learning agent, the mean value function of the action of the first machine learning agent. Thus, the proposed method is applicable to multi-agent reinforcement learning algorithms. The steps of determining and calculating may also be repeated for a different action of the first machine learning agent. In what follows, there is described the implementation of the two aforementioned components within an actor-critic style MARL learning algorithm. Typically, given a state observation r and given access to an observation of the joint action a e c / l any agent i e JV is able to compute Q (T, a) which estimates the agent’s expected return using its policy given the behaviour of all other agents N I { / } given that the agent takes an action a e c / £j. Therefore, Q (T, a) seeks to provide an estimate of the agent’s own action while accounting for the actions of other agents.

[0032] However, since the agents use stochastic policies and repeatedly update their policies, the action a_j provides little information about the usefulness of taking action a in state s and may also be an action tuple that is unlikely to be played again. This can occur if at least one of the agents -i samples an action to be executed which is a low probability action under either its current policy or under its policies after subsequent updates. This procedure involves making a single observation (alt. . . , aN) ~ , nN) of other agents’ actions sampled from their individual policies. This is not very useful in providing information about other agents’ behaviour. Learning requires many samples and many revisitations of the same state. Including global observations is computationally expensive as the set of possible actions explodes with increasing numbers of agents. This necessitates Q value factorisations methods which impose restrictions e.g. QMIX with monotonic factorisation. The proposed method comprises receiving current state information about each of the plurality of other machine learning agents in the environment and evaluating each respective policy given the current state information to provide a respective action of the other machine learning agent, where each different possible combination of actions provides a different joint action.

[0033] In typical implementations of actor-critic style MARL algorithms, the value function (also called the critic) of the tth agent is a function or equation defined as follows:

[0034] Vf.S ]R

[0035] That is, the value function outputs a real number representing the expected sum of discounted rewards of policy nlfrom state st.

[0036] The proposed implementation augments the value function equation to also take the joint action at~n:(- 1 ■) as input:

[0037] 1^: 5 X A1X ... X ANIR

[0038] The value function now outputs a real-number representing the expected sum of discounted rewards of policy nlfrom state stgiven joint-action at. That is, the expected sum of rewards is as a result of the action of the first machine learning agent in the current state while accounting for one of the plurality of joint actions of the plurality of other machine learning agents. The value function, that is the real number output of the function called the value function shown above, is calculated using a value function equation arranged to take one of the determined plurality of joint actions as an input. Vtis the value function, S is the current state, A±- ANrepresent each action of a joint action for N other machine learning agents, and IR is the real number output which represents the expected sum of rewards as a result of the action i of the first machine learning agent.

[0039] In contemporary MARL algorithms, the value function is typically implemented as a multi-layer perceptron (MLP). For the proposed augmentation, the joint action atis appended to stand passed through the network as depicted in Figure 2.

[0040] Figure 2 shows a schematic diagram of the combined joint action and state 201 passing through the network 200. As the combined input 201 passed through various layers 202 and 203 of the perceptron network 200, the output 204 is generated. The output 204 is a real number representing the expected sum of discounted rewards of policy nlfrom state stwhich also takes into account the joint action at.

[0041] A key object in the proposed framework is the jointly marginalised action-value function Q which is defined via the following expression:

[0042] 2i(s, cii)

[0043] As with Qt (T, a), the function Qtseeks to estimate the expected return following agent i taking action a;. However, unlike Qi tQtbuilds in the distribution of the actions played by other agents under their current policy. Therefore, giving less weight to low probability actions and conversely giving more weight to high probability actions. Another key aspect to note is that Q depends only on the agent’s own action and state, which acquires the benefit of IL (decentralised learning), while factoring in the behaviour of other agents. This avoids the need for rigid factorisations of the value functions that generally requires imposing restrictive assumptions on the reward structure.

[0044] To approximate the expectation, we construct the following object Q

[0045] It is noted that for large enough k we have Qt(s, a) « Qt(s, a) and by the law of large numbers we therefore have that Qt-> Qtas k co. In standard implementations of the actor-critic style MARL algorithms, the critic’s parameters are updated with respect to the gradient of the following loss:

[0046] L = rt+ y7(st+1) - 7(st)

[0047] In the proposed approach, the critic now takes as input atas well.

[0048] K samples of are generated and marginalised out as follows:

[0049] Similarly,

[0050] The critic’s parameters are updated with respect to the gradient of the following loss:

[0051] L = rt+ y7'(st+1) - 7'(st)

[0052] The two aspects of the proposed approach as described above are incorporated into standard MARL algorithms as shown in the schema of Figure 3. Stages 302, 304, and 305 explicitly indicate components of the proposed method and stages 301 , 303, and 306 indicate algorithmic components of generic MARL algorithms which benefit from the other proposed components.

[0053] The procedure 300 as illustrated in Figure 3, the first stage 301 starts by initialising the environment alongside a MARL learning algorithm with value function or critic and a policy where said learning algorithm uses a policy. At stage 302 the value function or critic is augmented to take the joint action into account by appending the joint action vector to the current state. As a result, the output value function takes into account the actions of the other agents without expensive computational processes and in doing so reduces the variance of the value function compared to other computationally inexpensive solutions.

[0054] The next stage 303 comprises a typical MARL step where policies of the other agents in the environment whose actions are being considered are rolled out. As a result, action data is obtained for each of those agents. D is a data set defined by a set of observations o - oT, joint actions ar- aT, and reward tuples rr- rT. Here T is the time horizon of the problem, for example an appropriate integer or T=oo.

[0055] The next stage 304 involves repeating the evaluation of the augmented value function for each of the possible joint actions. That is, one joint action per agent to be considered within the environment. The value function or critic can then provide a real number output which accounts for that specific joint action. In this notation o; is the observation or local observation for agent i.

[0056] In the next stage 305 the mean of all of the value function outputs or critics’ values can be calculated by averaging them over the number of joint actions considered which is the same as the number of agents considered within the environment. The mean value function is a mean of the total value function of a plurality of joint actions being considered over the number of the plurality of joint actions being considered.

[0057] In the final stage 306 the value function output or critic value is used to update the value function or critic and the corresponding policy. That is, the MARL algorithm is trained by subsequently updating the policy based on the calculated mean value function. Thus, the method may comprise updating the first machine learning agent and optimising the policy of the first machine learning agent based on the calculated mean value function.

[0058] Figure 4 shows a computing system 401 comprising a processing entity 402 and a program store 403. The processing entity 402 comprises one or more processors. The program store 403 stores in non-transient form program code executable by the processing entity 402 to perform the algorithmic functions described herein. The computing system 401 may be a single computer or multiple computers. It may be in a single location or distributed between multiple locations.

[0059] The computing system 401 may be contained within one or more devices. As such there is described herein a device for implementing a policy of a first machine learning agent. The policy being optimised by calculating a mean value function of one or more actions of the first machine learning agent within a multi-agent reinforcement learning framework comprising a plurality of other machine learning agents. The mean value function is calculated to account for a plurality of joint actions of the plurality of other machine learning agents. Therefore, there is provided a device configured to operate according to the optimised training of the policy it implements. The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A method of training a first machine learning agent (102) in a multi-agent reinforcement learning framework (101), the framework comprising a plurality of other machine learning agents each implementing their own respective policy, the method comprising: determining (304) a plurality of joint actions each comprising a different combination of respective actions to be taken by each of the plurality of machine learning agents based on their respective policies; and calculating (305), based on the determined plurality of joint actions and an action of the first machine learning agent, the mean value function of the action of the first machine learning agent.

2. The method of claim 1 , wherein the method comprises receiving current state information about each of the plurality of other machine learning agents; and evaluating (303) each respective policy given the current state information to provide a respective action of the other machine learning agent, where each different possible combination of actions provides a different joint action.

3. The method of claim 1 or 2, wherein the value function is a real number which represents the expected sum of rewards as a result of the action of the first machine learning agent in the current state while accounting for one of the plurality of joint actions of the plurality of other machine learning agents.

4. The method of any preceding claim, wherein the mean value function is a mean of the total value function of a plurality of joint actions being considered over the number of the plurality of joint actions being considered.

5. The method of any preceding claim, wherein the steps of determining and calculating are repeated for a different action of the first machine learning agent.

6. The method of any preceding claim, wherein the value function is calculated (302) using a value function equation arranged to take one of the determined plurality of joint actions as an input.

7. The method of claim 6, wherein the value function equation is written as:where Vtis the value function, S is the current state, A±- ANrepresent each action of a joint action for N other machine learning agents, and IR is the real number output which represents the expected sum of rewards as a result of the action i of the first machine learning agent.

8. The method of claim 6 or 7, wherein the value function equation is implemented as a multilayer perceptron (200) with the joint action appended to the current state term.

9. The method of any preceding claim, wherein the method comprises updating (306) the first machine learning agent and optimising the policy of the first machine learning agent based on the calculated mean value function.

10. A device for implementing a policy of a first machine learning agent, the policy optimised by calculating a mean value function of one or more actions of the first machine learning agent within a multi-agent reinforcement learning framework comprising a plurality of other machine learning agents, where the mean value function is calculated to account for a plurality of joint actions of the plurality of other machine learning agents.

11. The device of claim 10, the device implementing the policy optimised according to the method of any of the preceding claims.

Images