Multi-agent reinforcement learning program, information processing device, and multi-agent reinforcement learning method
By calculating and sharing constraint influences among agents, the method ensures stable multi-agent reinforcement learning that adheres to system-wide constraints, addressing the challenge of unpredictable performance in conventional methods.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- FUJITSU LTD
- Filing Date
- 2022-09-20
- Publication Date
- 2026-06-30
AI Technical Summary
Conventional multi-agent reinforcement learning methods fail to consider system-wide constraint conditions, leading to unpredictable performance during and after training, as they cannot account for the collective actions of multiple agents, resulting in an exponential increase in the action space dimensionality.
A method where each agent calculates the degree of influence on its own and system-wide constraints, updating its policy parameters only within defined ranges that satisfy both, and sharing this influence with subsequent agents in a predetermined order to ensure compliance with system-wide constraints.
This approach allows for learning that satisfies both agent-specific and system-wide constraints, ensuring stable performance by avoiding an exponential increase in action space dimensionality and maintaining constraint satisfaction during and after learning.
Smart Images
Figure 0007882069000002 
Figure 0007882069000003 
Figure 0007882069000004
Abstract
Description
[Technical Field]
[0001] This invention relates to a multi-agent reinforcement learning program, and the like. [Background technology]
[0002] In recent years, Multi-Agent Reinforcement Learning (MARL) has become known as a method for solving problems where multiple agents interact with a common environment and make sequential decisions while sharing the problem. In such multi-agent reinforcement learning, <1> There is a problem in that the number of dimensions of the learning action space increases exponentially as the number of agents increases. <2> There is a problem in that the performance during and after training is not guaranteed. In other words, the final performance is unknown until training is complete.
[0003] Here, a method is disclosed for learning a policy for each agent that maximizes the common reward for all agents in a constrained multi-agent reinforcement learning system, while satisfying agent-specific safe constraints (see, for example, Non-Patent Document 1). In this method, by learning a policy for each agent, the exponential increase in the dimension of the action space being learned can be avoided. That is, in this method, the number of dimensions of the action space being learned increases exponentially with increasing numbers of agents. <1> This can avoid the problem. [Prior art documents] [Patent Documents]
[0004] [Patent Document 1] Japanese Patent Publication No. 2009-014300 [Patent Document 2] Japanese Patent Publication No. 2020-080103 [Patent Document 3] U.S. Patent Application Publication No. 2021 / 0200163 [Non-patent literature]
[0005] [Non-Patent Document 1] Gu, Shangding, et al. “Multi-agent constrained policy optimisation.” arXiv preprint arXiv:2110.02793 (2021). [Summary of the Invention] [Problems to be Solved by the Invention]
[0006] However, conventional methods have a problem in that learning considering system-wide constraint conditions that depend on the results of actions of multiple agents cannot be performed. As a result, it can be said that there is a problem in <2> that the performance during and after learning is not guaranteed.
[0007] In one aspect, the present invention aims to perform learning considering system-wide constraint conditions in multi-agent reinforcement learning with constraints. [Means for Solving the Problems]
[0008] In one embodiment, in a multi-agent reinforcement learning program for a constrained control problem in which a plurality of agents exist, according to a predetermined update order of policy parameters of each agent, the degree of influence on the unique constraint conditions of its own agent and the degree of influence by the updated policy parameters of the agent in the previous update order are shared, and the degree of influence on the system-wide constraint conditions is calculated. When the update range of the policy parameters of its own agent exists within each range determined by the degree of influence on the unique constraint conditions of its own agent and the degree of influence on the system-wide constraint conditions, the policy parameters of its own agent are updated, and the updated degree of influence is shared with the agent in the next update order, causing a computer to execute the process. [Effects of the Invention]
[0009] According to one embodiment, learning can be performed while considering system-wide constraints that depend on the results of the actions of multiple agents. [Brief explanation of the drawing]
[0010] [Figure 1] Figure 1 is a block diagram showing an example of the functional configuration of an information processing device according to an embodiment. [Figure 2A] Figure 2A is a diagram (1) showing an image of the multi-agent reinforcement learning process according to the embodiment. [Figure 2B] Figure 2B is a diagram (2) showing an image of the multi-agent reinforcement learning process according to the embodiment. [Figure 3A] Figure 3A is a diagram (1) showing an example of the overall flowchart of the multi-agent reinforcement learning process according to the embodiment. [Figure 3B] Figure 3B is a diagram (2) showing an example of the overall flowchart of the multi-agent reinforcement learning process according to the embodiment. [Figure 3C] Figure 3C is a diagram (3) showing an example of the overall flowchart of the multi-agent reinforcement learning process according to the embodiment. [Figure 4A] Figure 4A is a diagram (1) showing an example of the application results of multi-agent reinforcement learning according to the embodiment. [Figure 4B] Figure 4B is Figure (2), which shows an example of the application results of multi-agent reinforcement learning according to the embodiment. [Figure 4C] Figure 4C is Figure (3), which shows an example of the application results of multi-agent reinforcement learning according to the embodiment. [Figure 5] Figure 5 shows an example of a computer running a multi-agent reinforcement learning program. [Figure 6] Figure 6 shows an example of a multi-agent reinforcement learning process. [Modes for carrying out the invention]
[0011] The following describes in detail, with reference to the drawings, embodiments of the multi-agent reinforcement learning program, information processing device, and multi-agent reinforcement learning method disclosed herein. However, the present invention is not limited to these embodiments.
[0012] First, we will explain one method of constrained multi-agent reinforcement learning, which involves multiple agents. Multi-agent reinforcement learning, as used here, solves problems where multiple agents interact with a common environment and make sequential decisions to share the problem. In other words, constrained multi-agent reinforcement learning involves each agent observing the current "state" and, using a series of "actions" as clues, learning a strategy to minimize (or maximize) the "reward" (objective function) while satisfying "constraints." Here, "constraints" refer to the conditions under which performance is guaranteed. An example of constrained multi-agent reinforcement learning is learning an algorithm for controlling the shutdown of each base station (BS) in a communication network in each area so as to minimize the sum of the power consumption of all BSs while maintaining the average satisfaction level of all users connected to each BS above a certain level. Here, BS is an example of an agent. Maintaining the average satisfaction level of all users above a certain level is an example of a "constraint." Time, grid demand, BS load at the previous time, and BS power consumption at the previous time are examples of "states." The sum of the power consumption of all BS (Broadcasting System) is an example of a "reward" (objective function). Whether or not to shut down BS (turning BS on / off) is an example of an "action".
[0013] One method of multi-agent reinforcement learning involves learning a policy for each agent that maximizes a common reward for all agents, while satisfying agent-specific safe constraints. This method allows for understanding the actions of each agent individually by learning a policy for each agent, thus avoiding an exponential increase in the dimensionality of the action space.
[0014] However, this method has a problem in that it cannot perform learning that takes into account system-wide constraints that depend on the results of the actions of multiple agents. This problem will be explained. Figure 6 is a reference example of multi-agent reinforcement learning processing. Note that in Figure 6, there are three agents.
[0015] As shown in Figure 6, in multi-agent reinforcement learning, each agent updates its policy to satisfy its own unique constraints. Here, in one learning step, agent a calculates the update range from its pre-update policy to satisfy its own unique constraints, and updates its policy by the calculated update range. In the same learning step, agents b and c similarly calculate the update range from their pre-update policies to satisfy their own unique constraints, and update their policies by the calculated update range.
[0016] However, in a method where each agent updates its policy to satisfy its own unique constraints, the constraints of the entire system are not necessarily satisfied. In other words, the updated policy of a particular agent may deviate from the range of policies that satisfy the constraints of the entire system. In short, such a method cannot perform learning that takes into account system-wide constraints that depend on the results of the actions of multiple agents.
[0017] Therefore, one possible approach is to calculate the impact of updating each agent's policy on each of the system-wide constraints, in addition to the constraints specific to each agent, and then restrict the range of policy updates at each learning step according to that impact.
[0018] However, while each agent updates its policy sequentially, even if an earlier agent updates its policy to satisfy the constraint, an update by a later agent may cause the constraint to no longer be satisfied. Therefore, it is necessary to appropriately update each agent's policy, but there is no specific and quantitative method for doing so.
[0019] Therefore, in the following embodiment, we will describe a constrained multi-agent reinforcement learning process that appropriately updates the policies of each agent and allows learning to be performed while considering the constraints of the entire system. [Examples]
[0020] Figure 1 is a block diagram showing an example of the functional configuration of an information processing device according to an embodiment. In the information processing device 1 shown in Figure 1, each agent sequentially calculates the degree of influence on its own unique constraints and the overall system constraints, and only if there is an update range within the range defined by the calculated influence, it updates the policy within this range. When the information processing device 1 updates the policy, it shares the degree of influence of the updated policy on the overall system constraints with the next agent in the update sequence. Here, "policy" refers to policy parameters.
[0021] The information processing device 1 comprises a control unit 10 and a storage unit 20. The control unit 10 includes a learning initialization unit 11, a decision unit 12, a calculation unit 13, a policy update unit 14, and a learning update unit 15. The storage unit 20 also contains combination history information 21. Note that the decision unit 12 and the calculation unit 13 are examples of calculation units. The policy update unit 14 is an example of an update unit.
[0022] The combination history information 21 is history information that stores the combination of state, action, and reward for each agent in a learning episode. A learning episode here refers to the interval from the start to the end of an action in response to the environment provided for reinforcement learning. For example, when learning the algorithm for controlling the shutdown of each BS (agent), the state, action, and reward are as follows: The state is information indicating the time, grid demand, BS load at the previous time, and BS power consumption at the previous time. The action is information indicating whether the BS is on or off (whether to shut down the signal or not). The reward is information indicating the sum of the power consumption of all BS.
[0023] The learning initialization unit 11 initializes the data used for learning. For example, the learning initialization unit 11 initializes the learning parameters used in reinforcement learning. The learning initialization unit 11 also initializes the learning episodes used in reinforcement learning. Furthermore, the learning initialization unit 11 acquires a history of state, action, and reward combinations for a certain period of time in the current learning episode for each agent and stores it in the combination history information 21.
[0024] The decision unit 12 randomly determines the order in which the policy parameters of each agent are updated. This allows the decision unit 12 to update the policy parameters of each agent evenly.
[0025] The calculation unit 13 calculates the degree of influence of its own agent on its own specific constraints and the degree of influence of the policy update of the agent in the previous update order on the overall system constraints, according to the determined update order. For example, the calculation unit 13 calculates the degree of influence of its own agent on its own specific constraints according to the update order determined by the determination unit 12. Then, the calculation unit 13 calculates the degree of influence of the overall system constraints, which is shared by the degree of influence of the updated policy parameters of the agent in the previous update order. The calculation unit 13 uses the combination history information 21 to calculate each degree of influence.
[0026] The policy update unit 14 updates its agent's policy parameters if the update range for its agent's policy parameters falls within a range determined by the degree of influence on its agent's specific constraints and the degree of influence on the system-wide constraints. The update range here refers to the range of change between the policy parameters before the update and the policy parameters after the update. In other words, if the policy update unit 14 updates its agent's policy parameters within a range of possible values for its agent's updated policy parameters, which falls within a range determined by the degree of influence on its agent's specific constraints and the degree of influence on the system-wide constraints, it updates its agent's policy parameters within this range. The possible range for the updated policy parameters is calculated, for example, based on equation (1). [Number]
[0027] The left side of Equation (1) represents the set of policy parameters π h that satisfy the agent-specific constraint conditions of agent i and the constraint conditions of the entire system. That is, the left side of Equation (1) indicates the range of possible values of the updated policy parameters. The right side of Equation (1) consists of (1.1), (1.2), and (1.3). ih
[0028] (1.1) is a condition for not significantly changing the policy parameter π h at the previous learning step k for agent i itself. The variable explanations in (1.1) are as follows. The variable π k ih represents the updated policy parameter of agent i. The variable π ih represents the policy parameter of agent i h before the update (at learning step k). The variable D k ih represents the maximum value of the Kullback-Leibler information measure between two policy parameters. The variable δ represents the upper limit regarding the change in the policy parameter before and after the update. h kL max
[0029] (1.2) is a condition for satisfying the specific constraint condition j of each agent i h . The variable explanations in (1.2) are as follows. The variable J j ih (·), the variable c j ih represents the cost function regarding the specific constraint condition j of agent i h and its upper limit value. The variable L ih j,πk (π ih ) represents the influence degree on the specific constraint condition j by the updated policy parameter of agent i h . The variable v j ih represents agent ih This shows the coefficients relating to the change in the policy parameters before and after updating, with respect to the intrinsic constraint j. Variable m ih This indicates the number of unique constraints for agent ih.
[0030] (1.3) is a condition for satisfying the system-wide constraint p, and includes the influence shared from the agent in the previous update order. The variables in (1.3) are described below. J p (·),d p This shows the cost function and its upper limit with respect to the system-wide constraint p. π k+1 i1:h-1 The update order is agent i h The agent before (i1,···,i h-1 The updated policy parameters are shown below. The variable L indicated by the symbol a1 is agent i h and update order is agent i h The agent before (i1,···,i h-1 This shows the impact of the updated policy parameters on the overall system constraint p. The variables L, l=1,···,h-1, denoted by a2, are those of agent i in the update order. h Shared by an earlier agent, in update order of agent i h This shows the impact of the updated policy parameters of the agent on the overall system constraint p. p This shows the coefficient relating to the change in the policy parameter before and after updating the system-wide constraint p. The variable M represents the number of system-wide constraints.
[0031] In other words, the policy update unit 14 determines, using equations (1)(1.1) and (1.2), whether there is an update range for its agent's policy parameters within the range determined by the degree of influence on its own agent's specific constraints. Then, using equation (1.3), the policy update unit 14 determines, whether there is an update range for its agent's policy parameters within the range determined by the degree of influence on the system-wide constraints. That is, the policy update unit 14 determines, using equation (1) which constitutes (1.1) to (1.3), whether there is an update range for its agent's policy parameters within the range determined by the degree of influence on its own agent's specific constraints and the degree of influence on the system-wide constraints.
[0032] Then, if the policy update unit 14 determines that there is an update range for its agent's policy parameters within a range determined by the degree of influence on its own agent's specific constraints and the degree of influence on the system-wide constraints, it performs the following process: The policy update unit 14 updates its agent's policy parameters with the available update range. This allows the policy update unit 14 to perform learning that takes into account system-wide constraints that depend on the results of the actions of multiple agents. In addition, by learning its own agent's policy in order, the policy update unit 14 understands only the actions of its own agent, thus avoiding an exponential increase in the dimensionality of the action space. Finally, the policy update unit 14 shares the updated influence with the next agent in the update sequence.
[0033] Furthermore, if the policy update unit 14 determines that there is no update range for its agent's policy parameters within the range determined by the degree of influence on its own agent's specific constraints and the range determined by the degree of influence on the system-wide constraints, it performs the following process: That is, the policy update unit 14 does not update the policy parameters of its own agent or agents in subsequent update order. This is because there is no update range for the policy parameters that can be updated. In other words, there is no common part of the update range between the range of policy parameters that satisfy the agent's specific constraints and the range of policy parameters that satisfy the system-wide constraints. As a result, the policy update unit 14 can guarantee that not only the constraints specific to each agent, but also the system-wide constraints that depend on the actions of multiple agents are satisfied during and after learning.
[0034] The learning update unit 15 updates the data used for learning. For example, the learning update unit 15 updates the learning parameters used in reinforcement learning. Here, learning parameters refer to learning parameters other than policy parameters. The learning update unit 15 also increments the number of learning episodes used in reinforcement learning. If the number of learning episodes is less than the maximum number of learning episodes, the learning update unit 15 performs learning in the next learning episode.
[0035] [Image of multi-agent reinforcement learning processing] Here, the image of the multi-agent reinforcement learning process according to the embodiment will be explained with reference to Figures 2A and 2B. Figures 2A and 2B are diagrams illustrating the image of the multi-agent reinforcement learning process according to the embodiment. It is assumed that there are three agents, a, b, and c. The order in which the policy parameters of each agent are updated is randomly determined as a, b, and c by the decision unit 12. For agent a, the policy update unit 14 shares the influence of the updated policy parameters with agent b, which is next in the update order.
[0036] The calculation unit 13 then calculates the impact on the system-wide constraints by combining the impact of agent b's unique constraints in the next update sequence with the impact of agent a's updated policy parameters in the previous update sequence. In other words, the calculation unit 13 calculates the impact of agent b's unique constraints and, by combining it with the impact of agent a's updated policy parameters, calculates the impact on the system-wide constraints. The policy update unit 14 then determines whether there is an update range for agent b's policy parameters that falls within the range determined by agent b's unique constraints and its impact on the system-wide constraints. In other words, the policy update unit 14 determines whether there is a range of possible updated policy parameters for agent b that falls within the range determined by agent b's unique constraints and its impact on the system-wide constraints.
[0037] Here, as shown in Figure 2A, the range determined by the degree of influence of agent b on its own specific constraints is the range indicated by the sign rb. This range is expressed by both sides of equation (1) (1.1) and (1.2). The range determined by the degree of influence of agent a on the system-wide constraints, sharing the influence of agent a's updated policy parameters, is the range indicated by the sign sb. This range is expressed by both sides of equation (1) (1.3). The range of possible updated policy parameters for agent b is the range indicated by the sign mb. This range is expressed by both sides of equation (1), and represents the range of possible updated policy parameters π for agent b. ihThis corresponds to the set of . That is, the range indicated by the symbol mb is the overlapping portion of the range indicated by the symbol rb and the range indicated by the symbol sb. If there is at least one updated policy parameter in the range indicated by the symbol mb, then there exists a range of possible updated policy parameters for agent b, and thus there exists an update range for the policy parameters. Note that the update range here refers to the line segment connecting a point in the range indicated by the symbol mb to the policy parameter before the update, when this point is taken as the updated policy parameter.
[0038] In Figure 2A, the policy update unit 14 determines that the update range for its agent b's policy parameters falls within a range (mb) determined by the degree of influence of its agent b's specific constraints and its influence of the system-wide constraints. Therefore, it performs the following process: The policy update unit 14 updates its agent's policy parameters within the available update range. Then, the policy update unit 14 shares the degree of influence of the updated policy parameters with the next agent in the update sequence.
[0039] The calculation unit 13 then calculates the impact on the system-wide constraints by combining the impact of agent c's unique constraints in the next update sequence with the impact of agent b's updated policy parameters in the previous update sequence. In other words, the calculation unit 13 calculates the impact of agent c's unique constraints and, by combining it with the impact of agent b's updated policy parameters, calculates the impact on the system-wide constraints. The policy update unit 14 then determines whether there is an update range for agent c's policy parameters that falls within the range determined by agent c's unique constraints and its impact on the system-wide constraints. In other words, the policy update unit 14 determines whether there is a range of possible updated policy parameters for agent c that falls within the range determined by agent c's unique constraints and its impact on the system-wide constraints.
[0040] Here, as shown in Figure 2A, the range determined by the degree of influence of agent c on its own specific constraints is the range indicated by the symbol rc. The range determined by the degree of influence of agent b on the overall system constraints, which shares the influence of agent b's updated policy parameters, is the range indicated by the symbol sc. The range of possible updated policy parameters for agent c is the overlapping portion of the range indicated by the symbols rc and sc. If even one updated policy parameter falls within this range, then there is a range of possible updated policy parameters for agent c, and therefore there is a range of policy parameter update options.
[0041] In Figure 2A, the policy update unit 14 determines that the update range for its agent c's policy parameters falls within a range determined by the degree of influence of its agent c's specific constraints and the degree of influence of its agent c's overall constraints. Therefore, it performs the following process: The policy update unit 14 updates its agent's policy parameters within the available update range.
[0042] This allows the policy update unit 14 to update the policy parameters of each agent within a range that satisfies the system-wide constraints. As a result, the policy update unit 14 can perform learning that takes into account system-wide constraints that depend on the results of the actions of multiple agents.
[0043] Figure 2B shows an example where the policy parameter update is terminated at an intermediate agent. As shown in Figure 2B, the policy update unit 14 finds that the update range wb of agent b's policy parameters lies within a range (rb) determined by the degree of influence of agent b's specific constraints, and within a range (sb) determined by the degree of influence of agent b's overall constraints. In other words, the policy update unit 14 finds that the update range wb of agent b's policy parameters lies within the policy range (sb) determined by the degree of influence of the policy update of agent a in the previous update sequence on the overall constraints. Therefore, the policy update unit 14 updates agent b's policy parameters and shares the degree of influence of the updated policy parameters with agent c in the next update sequence.
[0044] Next, the calculation unit 13 calculates the degree of influence of agent c on the specific constraints, and also calculates the degree of influence of agent b's updated policy parameters by sharing this degree of influence. Then, the policy update unit 14 determines whether there is an update range for agent c's policy parameters within the range determined by the degree of influence of agent c on the specific constraints and the degree of influence of agent c on the system-wide constraints. In other words, the policy update unit 14 determines whether there is a range of possible updated policy parameters for agent c within the range determined by the degree of influence of agent c on the specific constraints and the degree of influence of agent c on the system-wide constraints. Here, there is an update range wc for agent c's policy parameters within the range determined by the degree of influence of agent c on the specific constraints (rc). However, there is no update range wc for agent c's policy parameters within the range determined by the degree of influence of agent c on the system-wide constraints (sc). There is no common part of the update range in the overlapping range (rc,sc). For this reason, the policy update unit 14 does not update agent c's policy parameters. Furthermore, the policy update unit 14 does not update the policy parameters of agents in subsequent update order.
[0045] This is because there is no updatable update range for the policy parameters. In other words, there is no common update range in the overlapping portion between the range of policy parameters that satisfy agent-specific constraints and the range of policy parameters that satisfy system-wide constraints. As a result, the policy update unit 14 can avoid updating the policy parameters of agents c and beyond, thereby ensuring that not only the constraints specific to each agent, but also system-wide constraints that depend on the actions of multiple agents are satisfied during and after learning.
[0046] [Flowchart for Multi-Agent Reinforcement Learning Process] Next, we will describe an example of the overall flowchart of the multi-agent reinforcement learning process performed by the information processing device 1. Figures 3A to 3C are diagrams showing an example of the overall flowchart of the multi-agent reinforcement learning process according to the embodiment.
[0047] As shown in Figure 3A, the information processing device 1 initializes the learning parameters (step S11). The information processing device 1 initializes the number of learning episodes (step S12). Here, the information processing device 1 sets the number of learning episodes "1" to the variable e. The information processing device 1 performs control for a certain period of time as episode e and saves the history of the combination of state, action, and reward during that time in the combination history information 21 for each agent (step S13).
[0048] As shown in Figure 3B, the information processing device 1 randomly determines the update order of the agents (step S21). The information processing device 1 assigns each agent in update order, for example, i1, i2, ..., i n The numbers are assigned (step S22). Here, n is assumed to be the number of agents. The information processing device 1 initializes the agent update index h (step S23). Here, the information processing device 1 sets the variable h to "1".
[0049] The information processing device 1 uses the combination history information 21 to process agent i hThe degree of influence on the specific constraints (influence A) is calculated (step S24). The information processing device 1 uses the combination history information 21 to calculate the degree of influence on the constraints of the entire system (influence B) (step S25). In other words, the information processing device 1 calculates the degree of influence B on the constraints of the entire system based on the influence shared by the agents in the previous update order.
[0050] The information processing device 1 determines the range of influence A and influence B, and then processes agent i h It is determined whether there is an update range for the policy parameters (step S26). Within the range determined by influence A and influence B, agent i h If it is determined that there is no update range for the policy parameters (step S26; No), the information processing device 1 performs the following processing. That is, the information processing device 1 processes agent i h Without updating the policy parameters of the current agent or subsequent agents, proceed to step S41.
[0051] On the other hand, within the range determined by impact levels A and B, agent i h If it is determined that there is an update range for the policy parameters (step S26; Yes), the information processing device 1 performs the following processing. That is, within the range determined by influence A and influence B, agent i h Update the policy parameters (step S27).
[0052] Then, the information processing device 1 determines whether the agent update index h is less than the number of agents n (step S28). If it determines that h is greater than or equal to n (step S28; No), the information processing device 1 determines that processing for the number of agents is complete and proceeds to step S41.
[0053] If it is determined that h is less than n (step S28; Yes), the information processing device 1 will determine the next update order for agent i h+1The impact level B is shared (step S29). Then, the information processing device 1 updates the agent update index h by 1 (step S30). Then, the information processing device 1 updates agent i in the next update order. h To perform the update process, we proceed to step S24.
[0054] As shown in Figure 3C, in step S41, the information processing device 1 updates the learning parameters excluding the policy parameters (step S41). The information processing device 1 updates the number of learning episodes by 1 (step S42).
[0055] Then, the information processing device 1 determines whether the number of learning episodes e is greater than the maximum number of episodes (step S43). If it determines that the number of learning episodes e is not greater than the maximum number of episodes (step S43; No), the information processing device 1 proceeds to step S13 to process the next learning episode.
[0056] On the other hand, if it is determined that the number of learning episodes e is greater than the maximum number of episodes (step S43; Yes), the information processing device 1 terminates the multi-agent reinforcement learning process.
[0057] In Figures 3A to 3C, it was explained that when episode e is updated, the information processing device 1 performs control for a certain period of time and saves the combination history of state, action, and reward during that time for each agent in the combination history information 21. Then, the information processing device 1 uses this combination history information 21 to update the policy of each agent in the updated episode e. However, the information processing device 1 is not limited to this, and when episode e is updated, it may also use the combination history already saved in the combination history information 21 to update the policy of each agent in the updated episode e.
[0058] [An example of the results of applying multi-agent reinforcement learning] Here, the results of applying multi-agent reinforcement learning according to the embodiment will be explained with reference to Figures 4A to 4C. Figures 4A to 4C show an example of the results of applying multi-agent reinforcement learning according to the embodiment. Figures 4A to 4C explain the experimental results when an algorithm for controlling two cart poles (moving on two parallel rails) is learned.
[0059] As shown in Figure 4A, two cart poles are mounted on two parallel rails. Each cart pole is an example of an "agent". An example of a system-wide "constraint" is that the cost is set to "1" if the distance between the two carts is greater than 0.2, and to "0" otherwise, keeping the discounted cumulative cost (hereinafter referred to as "discounted cost") in each episode to 10 or less. The positions and velocities of the two carts and the angles and angular velocities of their poles are examples of "states". Moving each cart to the right or left (giving each cart an input of 0 or 1) is an example of an "action". The time it takes for the poles of both carts to not fall over from the initial state is an example of a "reward (objective function)". The maximum time is 200 seconds, and the initial state is set randomly within a certain range. In the experiment, to simplify the problem, only system-wide constraints were considered (without considering agent-specific constraints).
[0060] Under these experimental conditions, multi-agent reinforcement learning was attempted.
[0061] Figure 4B shows a graph illustrating the cumulative reward per episode for learning that considers system-wide constraints and learning that does not. The X-axis of the graph shows the number of episodes. The Y-axis shows the cumulative reward. The solid line represents the cumulative reward results for episodes of learning that considers system-wide constraints (Proposed). The dashed line represents the cumulative reward results for episodes of learning that does not consider system-wide constraints (MATPRO). Each line represents the average value over 5 trials.
[0062] As shown in Figure 4B, the cumulative reward increases to near its maximum value (200 seconds) through learning, both when learning takes into account the constraints of the entire system and when it does not.
[0063] Figure 4C also shows a graph illustrating the episode-by-episode discount cost against the system-wide constraint for learning that considers the system-wide constraint and learning that does not. The X-axis of the graph shows the number of episodes. The Y-axis of the graph shows the discount cost. The dashed line is the threshold, which is "10" as the discount cost against the system-wide constraint. The threshold is "10" because the system-wide constraint keeps the discount cost in each episode below 10. The solid line shows the episode-by-episode discount cost against the system-wide constraint for learning that considers the system-wide constraint (Proposed). The dashed line shows the episode-by-episode discount cost against the system-wide constraint for learning that does not consider the system-wide constraint (MATPRO). Each line is the average value over 5 trials.
[0064] As shown in Figure 4C, the discount cost for the overall system constraints is almost never above the threshold of "10" when learning takes the overall system constraints into account. In other words, when learning takes the overall system constraints into account, the overall system constraints are almost completely satisfied even in the episodes during the learning process. In contrast, when learning does not take the overall system constraints into account, the results are almost always above the threshold of "10".
[0065] Therefore, the multi-agent reinforcement learning according to the embodiment can achieve learning that updates the strategy for increasing rewards while satisfying the constraints of the entire system both during and after learning. In other words, the multi-agent reinforcement learning according to the embodiment can perform learning that takes into account the constraints of the entire system, such as those that depend on the results of the actions of multiple agents.
[0066] Figures 4A to 4C illustrate an example of multi-agent reinforcement learning, specifically the learning of an algorithm to control two cart poles. However, multi-agent reinforcement learning is not limited to this example.
[0067] For example, multi-agent reinforcement learning may be applied to learning an algorithm for controlling the deactivation of base stations (BS) in a communication network. In such a case, the algorithm learns a deactivation control method for each BS that minimizes the total power consumption of all BS while maintaining the average satisfaction level of all users connected to the BS in each area above a certain level. Here, each BS is an example of an "agent". Maintaining the average satisfaction level of all users above a certain level is an example of a "constraint" that includes the entire system. Time, grid demand, BS load at the previous time, and BS power consumption at the previous time are examples of "states". Whether or not to deactivate a BS (turning the BS on / off) is an example of an "action". The total power consumption of all BS is an example of a "reward (objective function)".
[0068] This enables multi-agent reinforcement learning to update the policy that minimizes rewards while satisfying the constraints of the entire system, both during and after learning. In addition, it enables cooperative BS signal shutdown control over a wide area spanning multiple areas, further improving power saving performance. Furthermore, multi-agent reinforcement learning guarantees user satisfaction across the entire area without tuning the weight coefficients or penalty terms of the objective function, reducing the effort required for trial and error during learning.
[0069] In another example, multi-agent reinforcement learning could be applied to learning an algorithm for controlling air conditioning in a data center. In this case, the algorithm learns how to control each air conditioner so as to minimize the total power consumption of all air conditioners installed in the data center while keeping the server temperature below a certain value. Here, keeping the server temperature below a certain value is an example of a "constraint" that includes the entire system. The time, server temperature, server power consumption at the previous time, and the air conditioner's set temperature at the previous time are examples of "states". The set temperature of each air conditioner (or commands to raise or lower the set temperature) and the air conditioning intensity are examples of "actions". The total power consumption of all air conditioners is an example of a "reward (objective function)".
[0070] This enables multi-agent reinforcement learning to update the policy that minimizes reward while satisfying the overall system constraints both during and after learning. In addition, it allows for cooperative air conditioning control of multiple air conditioners, further improving power saving performance. Furthermore, multi-agent reinforcement learning can guarantee that the temperature of servers in the data center remains below a certain value without tuning the weight coefficients or penalty terms of the objective function, reducing the effort required for trial and error during learning.
[0071] In another example, multi-agent reinforcement learning may be applied to learning an algorithm for transporting goods using multiple drones. In such a case, the system learns an algorithm for controlling each drone to minimize the time it takes to transport the goods (carrying items) while maintaining a certain distance between each drone. Here, maintaining a certain distance between each drone is an example of a "constraint" that includes the entire system. The position and speed of each drone are examples of "states". The rotation speed (or torque input of each propeller) of each drone is an example of an "action". The time it takes to transport the goods is an example of a "reward (objective function)".
[0072] This allows multi-agent reinforcement learning to update the policy that minimizes reward while satisfying the overall system constraints both during and after learning. In addition, it enables cooperative flight and transport by multiple drones, further reducing transport time. Furthermore, multi-agent reinforcement learning guarantees that the distance between each drone is above a certain level without tuning the weight coefficients or penalty terms of the objective function, thereby reducing the effort required for trial and error during learning.
[0073] [Effects of the Example] According to the above embodiment, in a constraint-based control problem with multiple agents, the information processing device 1 calculates the influence on the system-wide constraints, which is the sum of the influence of its own agent on the specific constraints and the influence of the updated policy parameters of the agent in the previous update sequence, according to a predetermined update sequence of each agent's policy parameters. If the update range of its own agent's policy parameters falls within the ranges determined by its own agent's influence on the specific constraints and its influence on the system-wide constraints, the information processing device 1 updates its own agent's policy parameters and shares the updated influence with the agent in the next update sequence. With this configuration, the information processing device 1 can perform learning that takes into account system-wide constraints that depend on the results of the actions of multiple agents.
[0074] Furthermore, according to the above embodiment, if the information processing device 1 does not update the policy parameters of its own agent or subsequent agents in the update order if there is no update range for its own agent's policy parameters within the ranges determined by the degree of influence on its own agent's specific constraints and the degree of influence on the system-wide constraints. With this configuration, the information processing device 1 can guarantee that not only the constraints specific to each agent, but also the system-wide constraints that depend on the actions of multiple agents are satisfied during and after learning.
[0075] Furthermore, according to the above embodiment, the information processing device 1 randomly determines the order in which each agent's policy parameters are updated with the aim of maximizing or minimizing a common reward. The information processing device 1 calculates the degree of influence on its own agent's specific constraints and the degree of influence on the constraints of the entire system according to the determined update order. With this configuration, the information processing device 1 can update each agent's policy parameters evenly by randomly determining the order in which each agent's policy parameters are updated.
[0076] It should be noted that the components of the illustrated information processing device 1 do not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of the information processing device 1 are not limited to those shown, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Furthermore, the storage unit 20 may be connected to the information processing device 1 as an external device via a network.
[0077] Furthermore, the various processes described in the above embodiment can be realized by executing a pre-prepared program on a computer such as a personal computer or workstation. Therefore, below, an example of a computer that executes a multi-agent reinforcement learning program that realizes the same functions as the information processing device 1 shown in Figure 1 will be described. Here, a multi-agent reinforcement learning program that realizes the same functions as the information processing device 1 will be described as an example. Figure 5 is a diagram showing an example of a computer that executes a multi-agent reinforcement learning program.
[0078] As shown in Figure 5, the computer 200 includes a CPU (Central Processing Unit) 203 that performs various calculations, an input device 215 that accepts data input from the user, and a display device 209. The computer 200 also includes a drive device 213 that reads programs and other data from a storage medium, and a communication interface 217 that exchanges data with other computers via a network. The computer 200 also includes a memory 201 for temporarily storing various information and an HDD (Hard Disk Drive) 205. The memory 201, CPU 203, HDD 205, display control unit 207, display device 209, drive device 213, input device 215, and communication interface 217 are connected by a bus 219.
[0079] The drive device 213 is, for example, a device for a removable disk 211. The HDD 205 stores the multi-agent reinforcement learning program 205a and multi-agent reinforcement learning processing-related information 205b. The communication interface 217 manages the interface between the network and the inside of the device and controls the input and output of data from other computers. For example, a modem or a LAN adapter can be used for the communication interface 217.
[0080] The display device 209 is a display device that displays data such as cursors, icons, or toolboxes, as well as documents, images, and functional information. The display device 209 can employ, for example, a liquid crystal display or an organic EL (electroluminescence) display.
[0081] The CPU 203 reads the multi-agent reinforcement learning program 205a, loads it into memory 201, and executes it as a process. Each of these processes corresponds to a functional unit of the information processing device 1. The multi-agent reinforcement learning processing-related information 205b includes, for example, combination history information 21. The removable disk 211 stores this information, such as the multi-agent reinforcement learning program 205a.
[0082] Furthermore, the multi-agent reinforcement learning program 205a does not necessarily have to be stored on the HDD 205 from the beginning. For example, the program can be stored on a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card inserted into the computer 200. The computer 200 can then read and execute the multi-agent reinforcement learning program 205a from these devices. [Explanation of symbols]
[0083] 1. Information Processing Device 10 Control Unit 11. Learning Initialization Section 12. Decision Section 13 Calculation Section 14 Policy Update Section 15 Learning Update Section 20 Memory section 21 Combination History Information
Claims
1. In a constrained control problem involving multiple agents, The system calculates the impact on the overall constraints, which is calculated by combining the impact of each agent's own unique constraints and the impact of the updated policy parameters of the agents in the previous update sequence, according to a predetermined update order for each agent's policy parameters. If the update range for the policy parameters of the agent falls within the ranges determined by the degree of influence on the agent's own specific constraints and the degree of influence on the system-wide constraints, the agent updates its policy parameters and shares the updated influence with the next agent in the update sequence. A multi-agent reinforcement learning program characterized by having a computer execute the processing.
2. Furthermore, if the update range for the policy parameters of the agent itself does not fall within the ranges determined by the degree of influence on the agent's own specific constraints and the degree of influence on the system-wide constraints, the policy parameters of the agent itself and subsequent agents in the update order will not be updated. The multi-agent reinforcement learning program according to feature 1.
3. Regarding the updating of each agent's policy parameters with the aim of maximizing or minimizing a common reward, the order in which each agent's policy parameters are updated is randomly determined. The calculation process calculates the degree of influence on the constraints specific to its own agent and the degree of influence on the constraints of the entire system, according to the determined update order. A multi-agent reinforcement learning program as described in claim 1 or 2.
4. In a constrained control problem involving multiple agents, A calculation unit calculates the impact on the system-wide constraints, which is calculated by combining the impact of each agent's own unique constraints and the impact of the updated policy parameters of the agents in the previous update order, according to a predetermined update order of each agent's policy parameters. If the update range for the policy parameters of the agent falls within the ranges determined by the degree of influence on the agent's own specific constraints and the degree of influence on the constraints of the entire system, the update unit updates the policy parameters of the agent and shares the updated influence with the next agent in the update sequence. An information processing device characterized by having the following features.
5. In a constrained control problem involving multiple agents, The system calculates the impact on the overall constraints, which is calculated by combining the impact of each agent's own unique constraints and the impact of the updated policy parameters of the agents in the previous update sequence, according to a predetermined update order for each agent's policy parameters. If the update range for the policy parameters of the agent falls within the ranges determined by the degree of influence on the agent's own specific constraints and the degree of influence on the system-wide constraints, the agent updates its policy parameters and shares the updated influence with the next agent in the update sequence. A multi-agent reinforcement learning method characterized by the fact that the processing is performed by a computer.