System, method and apparatus for multi-agent reinforcement learning

Abstract

To provide a system, method and apparatus for efficient asynchronous multi-agent reinforcement learning.SOLUTION: A multi-agent reinforcement learning system includes a multi-agent including a first agent and a second agent, history encoders 310-330 provided for each agent of the multi-agent, a memory configured to store one or more commands, and at least one processor configured to execute the one or more commands stored in the memory. The at least one processor, by executing the one or more commands, is configured to generate first history information of the first agent by inputting observation data of the first agent into a first history encoder, and to generate second history information of the second agent by inputting observation data of the second agent into a second history encoder.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to systems, methods, and apparatus for performing multi-agent reinforcement learning. [Background technology]

[0002] In recent years, reinforcement learning has been primarily implemented in single-agent environments with static tasks. However, many real-world problems manifest as multi-agent problems, where each agent must make independent decisions based on imperfect observations, often taking into account the other agents, in order to maximize a collective or individual reward. Multi-agent reinforcement learning is a framework for solving such problems, capable of learning effective policies.

[0003] However, because such multi-agents execute their actions asynchronously with each other, much research has been conducted on asynchronous behavior. While asynchronous multi-agent reinforcement learning has been studied based on deep Q-networks, due to its structural limitations, methods based on deep Q-networks cannot simultaneously achieve centralized learning and distributed execution. Therefore, research on more efficient asynchronous multi-agent reinforcement learning methods is needed. Summary of the Invention [Problem to be solved by the invention]

[0004] One embodiment of the present disclosure provides a system, method, and apparatus for performing multi-agent reinforcement learning. [Means for solving the problem]

[0005] An embodiment of the present disclosure may provide a method and apparatus for deriving optimized results using reinforcement learning.

[0006] A multi-agent reinforcement learning system according to one embodiment of the present disclosure includes a multi-agent including a first agent and a second agent; a history encoder provided in each agent of the multi-agent, the history encoder including a first history encoder corresponding to the first agent and a second history encoder corresponding to the second agent; a memory for storing one or more instructions; and at least one processor for executing the one or more instructions stored in the memory, wherein the at least one processor, by executing the one or more instructions, can input observation data of the first agent to the first history encoder to generate first history information for the first agent, and input observation data of the second agent to the second history encoder to generate second history information for the second agent.

[0007] In one embodiment, the first history encoder may encode the observation information of the first agent after including time information associated with the observation information of the first agent.

[0008] In one embodiment, each history encoder provided in each agent of the multi-agent may include a multi-layer perceptron (MLP) and a gated recurrent unit (GRU).

[0009] In one embodiment, the system may further include an aggregation module that receives and processes one or more pieces of history information corresponding to the output of a history encoder provided in each agent of the multi-agent.

[0010] In one embodiment, the output value of the aggregation module can be processed by a Multi-Layer Perceptron (MLP).

[0011] An embodiment of the present disclosure may provide a multi-agent reinforcement learning method including a first agent and a second agent, the method being executed by at least one processor. The method may include inputting observation data of the first agent to a first history encoder to generate first history information for the first agent, and inputting observation data of the second agent to a second history encoder to generate second history information for the second agent, where a history encoder including the first history encoder and the second history encoder may be provided for each agent of the multi-agent.

[0012] In one embodiment, the first history encoder may encode the observation information of the first agent after including time information associated with the observation information of the first agent.

[0013] In one embodiment, each history encoder provided in each agent of the multi-agent may include a multi-layer perceptron (MLP) and a gated recurrent unit (GRU).

[0014] In one embodiment, the method may process, by an aggregation module, one or more pieces of history information corresponding to the output of a history encoder provided in each agent of the multi-agent.

[0015] In one embodiment, the method may further include processing the output value of the aggregation module by a Multi-Layer Perceptron (MLP).

[0016] One embodiment of the present disclosure may provide one or more non-transitory computer-readable recording media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations, the operations including inputting observation data of a first agent into a first history encoder to generate first history information for the first agent, and inputting observation data of a second agent into a second history encoder to generate second history information for the second agent, wherein a history encoder including the first history encoder and the second history encoder is provided for each agent of a multi-agent system.

[0017] In one embodiment, the first history encoder may encode the observation information of the first agent after including time information associated with the observation information of the first agent.

[0018] In one embodiment, each history encoder provided in each agent of the multi-agent may include a multi-layer perceptron (MLP) and a gated recurrent unit (GRU).

[0019] In one embodiment, the operations may further include processing, by an aggregation module, one or more pieces of history information corresponding to the output of a history encoder provided for each agent of the multi-agent.

[0020] In one embodiment, the operations may further include processing the output value of the aggregation module through a Multi-Layer Perceptron (MLP).

[0021] An embodiment of the present disclosure includes a program stored on a recording medium for causing a computer to execute a method according to an embodiment of the present disclosure.

[0022] An embodiment of the present disclosure includes a computer-readable recording medium having a program recorded thereon for causing a computer to execute a method according to an embodiment of the present disclosure.

[0023] One embodiment of the present disclosure includes a computer-readable recording medium on which a database used in one embodiment of the present disclosure is recorded. [Effects of the Invention]

[0024] According to one embodiment of the present disclosure, more efficient and accurate reinforcement learning training can be achieved. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a diagram for explaining a training buffer and the like in a macro-action decentralized partially observable Markov decision process (MacDec-POMDP) ​​according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating a centralized critic using a joint history encoder according to one embodiment of the present disclosure. [Figure 3] FIG. 3 is a diagram illustrating a collaborative agent-oriented centralized critic according to one embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating a reinforcement learning system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0026] To clarify the technical concept of the present disclosure, embodiments will be described in detail with reference to the accompanying drawings. Note that, in describing the present disclosure, if a detailed description of related publicly known functions or components is deemed to unnecessarily obscure the gist of the present disclosure, such detailed description will be omitted. In the drawings, components having substantially the same functions or configurations will be assigned the same reference numerals and symbols whenever possible, even if they are shown in different drawings. For convenience of description, both an apparatus and a method may be described together as necessary. Note that the operations described in the present disclosure do not necessarily have to be performed in the order described, and may be performed in parallel, selectively, or individually.

[0027] The terms used in the embodiments are currently commonly used and general terms that are selected as much as possible while taking into consideration the functions in the present disclosure, but these terms may change depending on the intentions or precedents of engineers in the relevant technical field, or the emergence of new technologies. In addition, in certain cases, the applicant may use terms arbitrarily selected, and in such cases, the meaning of the terms will be explained in detail in the explanation of the relevant embodiments. Therefore, the terms used in the present disclosure should be interpreted not as mere words and phrases, but based on the meaning of the terms and the content of the entire disclosure.

[0028] Throughout this specification, expressions in the singular are to be construed as including the plural unless the context clearly dictates otherwise. Furthermore, terms such as "comprise" and "have" should be understood to mean the presence of stated features, numerical values, steps, operations, components, parts, or combinations thereof, and do not preclude the possibility of the presence or addition of one or more other features, numerical values, steps, operations, components, parts, or combinations thereof in addition to those stated. In other words, when a certain component is described as "comprising" in this specification, unless otherwise specified, it does not mean that only that component is included, but that other components may also be included.

[0029] The phrase "at least one" refers to the components as a whole, and not to these components individually. For example, the phrases "at least one of A, B, and C" and "at least one of A, B, or C" can encompass "A alone," "B alone," "C alone," "either A and B," "either B and C," "either A and C," "all of A, B, and C," or combinations thereof.

[0030] Furthermore, terms such as "... unit" and "... module" used in this disclosure refer to a unit that processes at least one function or operation, and the unit may be embodied as hardware, software, or a combination of hardware and software.

[0031] Throughout this specification, when a part is said to be "connected" to another part, this includes not only when the part is "directly connected" to another part, but also when the part is "electrically connected" to another part via another material. Furthermore, when a part is said to "comprise" a particular component, this does not exclude other components, but means that the part may further include other components, unless otherwise specified.

[0032] Throughout this specification, the phrase "configured to" may be interpreted as meaning "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of," depending on the context and circumstances. The term "configured to" does not necessarily mean "specifically designed to" in terms of hardware. Instead, in some circumstances, the phrase "a system configured to" may mean that the system is "capable of" performing something in combination with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or it may refer to a general-purpose processor (e.g., a CPU or application processor) capable of performing those operations by executing one or more software programs stored in memory.

[0033] In one embodiment, a reinforcement learning system receives observations that characterize the current state of the environment and, based on those observations, selects actions for an agent to perform. When the agent performs the selected action, the environment transitions to a new state accordingly, and the reinforcement learning system receives a reward. This reward may be a numerical value that is a function of the environment state. During the course of interacting with the environment, the reinforcement learning system may attempt to maximize the long-term reward obtained for the actions performed by the agent.

[0034] The MacDec-POMDP (Macro-Action Decentralized Partially Observable Markov Decision Process) can integrate an options framework into the Dec-POMDP by defining a set of macro-actions for each agent. In one embodiment, the MacDec-POMDP can be represented as the following tuple: JPEG2025138618000002.jpg867

[0035] where: JPEG2025138618000003.jpg134150

[0036] Throughout this specification: JPEG2025138618000004.jpg86150

[0037] JPEG2025138618000005.jpg51150

number

[0038] FIG. 1 is a diagram for explaining a training buffer and the like in MacDec-POMDP according to an embodiment of the present disclosure.

[0039] Referring to Figure 1, the training buffer for Mac-CERTs (Macro-Action Concurrent Experience Replay Trajectories) 110 and the training buffer for Mac-JERTs (Macro-Action Joint Experience Replay Trajectories) 120 are shown. i and m i may denote the macro-observation and macro-action of agent i, respectively.

[0040] Referring to the example of FIG. 1 , the multi-agent system may include Agent 1, Agent 2, and Agent 3. Agent 1 may act at time step 4 and time step 6, and Agent 2 may act at time step 1 and time step 6, resulting in a state transition for Agent 5. In one embodiment, Mac-CERTs 110 may generate a training buffer by collecting the time steps at which state transitions occurred for each agent. Accordingly, as shown in FIG. 1 , a buffer containing information at time points t=0, 4, and 6 may be generated for Agent 1, a buffer containing information at time points t=0, 1, and 6 may be generated for Agent 2, and a buffer containing information at time points t=0 and 5 may be generated for Agent 3. In contrast, Mac-JERTs 120 may generate a training buffer by collecting the time steps at which state transitions occurred for any one of the agents. In this case, a buffer containing information at time points t=0, 1, 4, 5, and 6 may be generated. This will be described in more detail below.

[0041] In one embodiment of Mac-CERTs, the experience of each agent i regarding state transitions is represented by the following tuple: JPEG2025138618000007.jpg939

[0042] JPEG2025138618000008.jpg54150

number

[0043] In one embodiment of Mac-JERTs, an experience regarding a state transition can be expressed as a tuple such as: JPEG2025138618000010.jpg1036

[0044] JPEG2025138618000011.jpg105150

[0045] Multi-agent reinforcement learning (MRL) is being actively developed and applied in various fields. Multi-agent MRL enables distributed agents to cooperate effectively and complete assigned tasks. Currently, commonly used MRL models employ a method in which all agents simultaneously decide on their own actions based on the same duration. However, in real-world scenarios, agents often decide and execute actions at different times, resulting in unmotivated selection and execution of actions among agents. One embodiment of the present disclosure provides a framework for modeling multi-agent decision-making using MacDec-POMDP, in which the decision-making and execution of actions among agents occur asynchronously over time. In one embodiment, MacDec-POMDP allows for the initiation time and duration of each agent's action to differ from one another.

[0046] FIG. 2 is a diagram illustrating a centralized critic using a joint history encoder, etc., according to one embodiment of the present disclosure.

[0047] JPEG2025138618000012.jpg207166JPEG2025138618000013.jpg170166

[0048] In one embodiment, the joint history encoder 210 may include a first multi-layer perceptron (MLP) 230 and a gated recurrent unit (GRU) 240. The first MLP 230 may include an input layer, one or more hidden layers, and an output layer. The first MLP 230 receives training data 220 as input and determines output information. The output information of the first MLP 230 may be input to the GRU 240. The GRU 240 is a model used to process time series data and may operate using a reset gate and an update gate. The reset gate determines how much of the past state to forget, and the update gate determines how much information from the past state to retain. The output information of the GRU 240 may be input to the second MLP 250.

[0049] In this way, a centralized critic focused on combining the histories of all agents based on the time axis can use a combined history encoder to abstract joint macro-observations. However, continuous joint macro-observations may contain overlapping observations at time steps when some agents have begun to execute macro-actions and other agents have not yet performed their actions. As a result, overlapping observations may lead to inaccurate inferences. For example, at t=5 in Figure 2, overlapping observations for Agents 1 and 2 are included, even though only Agent 3 has undergone a state transition. Such overlapping observations may lead to inappropriate inferences.

[0050] In one embodiment, in the example shown in FIG. 2 of MacDec-PDMDP, the joint macro observation history abstracted by the joint history encoder at time step t=6 is defined by Equation 3 below:

number

[0051] JPEG2025138618000015.jpg44150

[0052] Therefore, because common centralized critic structures for multi-agent reinforcement learning focus on history along a common time axis, the centralized critic introduces as much overlapping macro-observations as possible from each agent, even at time steps when one agent initiates a macro-action and other agents are yet to execute their actions. However, such overlapping macro-observations can potentially impair the accuracy of the centralized critic's evaluation. To overcome the limitations of this approach and enable more efficient asynchronous learning, a specialized architecture is required that can effectively collect individual agent histories and facilitate inference between agents. In contrast, one embodiment of the present disclosure provides an agent-oriented centralized critic. This will be described in more detail below with reference to FIG. 3.

[0053] FIG. 3 is a diagram illustrating a collaborative agent-oriented centralized critic according to one embodiment of the present disclosure.

[0054] Referring to FIG. 3, an agent-oriented centralized critic (AOCC) method for MacDec-POMDP is illustrated. In one embodiment, the agent-oriented reinforcement learning architecture includes an agent-oriented history encoder 310, a history encoder 320, and a history encoder 330 for each agent, which perform positional encoding, and an aggregation module 340 for aggregating the histories of all agents. The agent-oriented history encoder 310, a history encoder 320, and a history encoder 330 for each agent can receive the latest local macro observations of the corresponding agent as input and output the local macro observation history. Unlike the centralized encoder shown in FIG. 2, such an agent-oriented history encoder does not duplicate the same observations. According to one embodiment, the agent-oriented history encoder can capture the local history more accurately by focusing only on the history of each agent.

[0055] In one embodiment, GRUs can be used in the agent-oriented history encoders without parameter sharing between agents, i.e., a first agent-oriented history encoder 310 corresponding to a first agent can include a first GRU, a second agent-oriented history encoder 320 corresponding to a second agent can include a second GRU, and a third agent-oriented history encoder 330 corresponding to a third agent can include a third GRU.

[0056] In one embodiment, the history of agent i is defined by Equation 4:

number

[0057] JPEG2025138618000017.jpg23150

[0058] In one embodiment, the agent-oriented history encoder encodes only each agent's local macro-observation, not the joint observation, which may lead to a loss of temporal consistency between agents. In one embodiment of the present disclosure, to take temporal order into account, a time marker corresponding to each agent's macro-observation may be introduced, and a sinusoidal positional encoding may be performed. The time step of the local macro-observation is represented by the positional encoding vector p i The encoded vector is the local macro observation z i By concatenating these, the time sequence can be taken into account. Accordingly, the history for agent i is defined by the following Equation 5:

number

[0059] In one embodiment, positional encoding can be such that the temporal ordering and duration of macro-behaviors across all agents is taken into account.

[0060] In one embodiment, the agent-oriented history derived from the agent-oriented history encoder can be further processed by an aggregation module to approximate a value function. The aggregation module can include an attention network, an MLP followed by concatenation, an adder, etc. Alternatively, for simplicity, only concatenation with the MLP layer may be performed.

[0061] In one embodiment, the input of the aggregation module consists of a concatenation of the most recent local histories, defined by Equation 6:

number

[0062] JPEG2025138618000020.jpg21150

[0063] According to one embodiment of the present disclosure, efficient centralized critic learning is possible even in an asynchronous setting by independently encoding the observation history and time information of each agent through positional encoding and then aggregating them. That is, based on the timestep information at which each agent begins to execute a macro-action, the observation history of each agent is encoded, and the time information is also separately encoded, allowing for explicit association in agent-oriented critic learning.

[0064] Furthermore, according to one embodiment of the present disclosure, reinforcement learning can be performed to derive more accurate results by removing duplicate macro-observations and the like.

[0065] FIG. 4 is a diagram illustrating a reinforcement learning system according to an embodiment of the present disclosure.

[0066] 4, as an example of a multi-agent reinforcement learning system, a reinforcement learning system executed by three agents, a first agent 410, a second agent 420, and a third agent 430, is shown. However, this is merely an example, and unlike in FIG. 3, reinforcement learning may be executed by two agents, or may be executed by more than three agents. For convenience of explanation, the following description will be given using an example with three agents.

[0067] In one embodiment, the environment 510 may change based on a first action 470 determined by the first agent 410, a second action 480 determined by the second agent 420, and a third action 490 determined by the third agent 430. In response, the first agent 410 may obtain a first observation 440 as the environment transitions, the second agent 420 may obtain a second observation 450, and the third agent 430 may obtain a third observation 460.

[0068] In one embodiment, a multi-agent reinforcement learning system may include a memory storing one or more instructions and at least one processor executing the one or more instructions stored in the memory. By executing the one or more instructions, the at least one processor may input observation information corresponding to a first observation 440 of a first agent 410 to a first history encoder to generate first history information for reinforcement learning. The at least one processor may also input observation information corresponding to a second observation 450 of a second agent 420 to a second history encoder to generate second history information. The at least one processor may also input observation information corresponding to a third observation 460 of a third agent 430 to a third history encoder to generate third history information. Such a history encoder may be provided for each agent in the multi-agent system. That is, if there are n agents, n history encoders may be provided.

[0069] In one embodiment, the first history encoder, the second history encoder, and the third history encoder can encode the observation information of the corresponding agent after including time information associated with the observation information. If the time information is not included, a separate history encoder would be required for each agent, which could result in inaccurate information being generated because the time of state transitions would not be reflected. However, according to one embodiment of the present disclosure, more accurate training is possible by encoding the observation information of each agent by associating it with time information. Furthermore, the first history encoder, the second history encoder, and the third history encoder can include a multi-layer perceptron (MLP) and a gated recurrent unit (GRU).

[0070] In one embodiment, a multi-agent reinforcement learning system may include an aggregation module that receives and processes one or more pieces of history information corresponding to the output of a history encoder provided in each agent of the multi-agent system. That is, one aggregation module may be provided that processes multiple pieces of history information. In addition, the output value of the aggregation module may be processed by a multi-layer perceptron (MLP).

[0071] In one embodiment, an environment 510 may transition and a reward 500 may be determined based on a first action 470 determined by a first agent 410, a second action 480 determined by a second agent 420, and a third action 490 determined by a third agent 430. The agents may cooperate with each other to maximize the value of the reward.

[0072] An embodiment of the present disclosure may also be realized in the form of a recording medium containing computer-executable instructions, such as a program module executed by a computer. Computer-readable media are any available media that can be accessed by a computer, and include volatile or nonvolatile media, removable or non-removable media. Computer-readable media may also include both computer recording media and communication media. Computer recording media include volatile or nonvolatile, removable or non-removable media implemented by any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media generally include computer-readable instructions, data structures, or program modules and may include any information transmission media.

[0073] The above description of the present disclosure is for illustrative purposes only, and it should be understood that those skilled in the art to which the present disclosure pertains can easily modify the present disclosure into other specific forms without departing from the technical spirit and essential characteristics of the present disclosure. Therefore, the above-described embodiments are illustrative in all respects and should not be construed as limiting. For example, each component described as a single configuration may be implemented as a distributed configuration, and similarly, each component described as a distributed configuration may be implemented as a combined configuration.

[0074] The scope of the present disclosure is not limited to the above detailed description, but is defined by the claims. All changes or modifications derived from the language and meaning of the claims, and the range of equivalents of such language, should be construed as being within the scope of the present disclosure.

Claims

1. A multi-agent reinforcement learning system, comprising: a multi-agent including a first agent and a second agent; a history encoder provided in each agent of the multi-agent, the history encoder including a first history encoder corresponding to the first agent and a second history encoder corresponding to the second agent; a memory for storing one or more instructions; at least one processor that executes the one or more instructions stored in the memory; The at least one processor executes the one or more instructions to: a system inputting observation data of the first agent into the first history encoder to generate first history information for the first agent, and inputting observation data of the second agent into the second history encoder to generate second history information for the second agent.

2. The first history encoder comprises: The system of claim 1 , wherein the encoding is performed after including time information associated with the first agent's observation information in the first agent's observation information.

3. Each history encoder provided in each agent of the multi-agent The system of claim 1, comprising a Multi-Layer Perceptron (MLP) and a Gated Recurrent Unit (GRU).

4. The system comprises: The system of claim 1, further comprising an aggregation module that receives and processes one or more pieces of history information corresponding to an output of a history encoder provided in each agent of the multi-agent.

5. The output value of the aggregation module is: The system of claim 4, wherein the processing is performed by a Multi-Layer Perceptron (MLP).

6. 1. A multi-agent reinforcement learning method executed by at least one processor, the multi-agent reinforcement learning method including a first agent and a second agent, the method comprising: inputting observation data of the first agent into a first history encoder to generate first history information of the first agent; inputting the observation data of the second agent into a second history encoder to generate second history information of the second agent; A method, characterized in that a history encoder including the first history encoder and the second history encoder is provided for each agent of a multi-agent.

7. The first history encoder comprises:

7. The method of claim 6, further comprising encoding the first agent's observation information after including time information associated with the first agent's observation information.

8. Each history encoder provided in each agent of the multi-agent The method of claim 6, comprising a Multi-Layer Perceptron (MLP) and a Gated Recurrent Unit (GRU).

9. The method comprises: The method of claim 6, further comprising the step of processing, by an aggregation module, one or more pieces of history information corresponding to the output of a history encoder provided in each agent of the multi-agent.

10. The method comprises: The method of claim 9, further comprising processing the output value of the aggregation module by a Multi-Layer Perceptron (MLP).

11. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations; The operation is inputting observation data of a first agent into a first history encoder to generate first history information of the first agent; inputting the observation data of a second agent into a second history encoder to generate second history information of the second agent; One or more non-transitory computer-readable recording media, wherein a history encoder including the first history encoder and the second history encoder is provided in each agent of a multi-agent.

12. The first history encoder comprises:

12. The one or more non-transitory computer-readable storage media of claim 11, wherein the encoding is performed after including time information associated with the observation information of the first agent in the observation information of the first agent.

13. Each history encoder provided in each agent of the multi-agent 12. The one or more non-transitory computer-readable media of claim 11, comprising a Multi-Layer Perceptron (MLP) and a Gated Recurrent Unit (GRU).

14. The operation is The one or more non-transitory computer-readable storage media of claim 11, further comprising an operation of processing, by an aggregation module, one or more pieces of history information corresponding to the output of a history encoder provided in each agent of the multi-agent.

15. The operation is The one or more non-transitory computer-readable media of claim 14 , further comprising an operation of processing the output value of the aggregation module through a multi-layer perceptron (MLP).

Images