Learning program and information processor and learning method

Abstract

To perform learning taking into account the constraint condition of the entire system in constrained multi-agent reinforcement learning.SOLUTION: In a constrained control problem with multiple agents, an information processor 1 determines the priority of the state to be used when a first agent acts based on the relationship between the multiple agents, selects a value to be inputted to a first strategy parameter according to the priority of the state, and uses the first strategy parameter with the value inputted to it to determine, in accordance with the specified update order, a first influence degree on the constraint condition of the first agent and a third influence degree on the constraint condition of the entire system based on a second influence degree by the second strategy parameter which the second agent updated in the previous order, and determines a range of the strategy parameter that satisfies the constraint condition according to the first and third influence degrees. The processing of such an information processor 1 can be applied to the control of the end of transmission at a base station (BS), for example.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to learning programs and the like. [Background technology]

[0002] In recent years, multi-agent reinforcement learning (MARL) has become known, in which multiple agents interact with a common environment and solve a shared problem while making decisions sequentially. In such multi-agent reinforcement learning, <1> There is a problem that the number of dimensions of the learning action space increases exponentially as the number of agents increases. <2> The problem is that performance during and after training is not guaranteed, i.e. the final performance is not known until training is complete.

[0003] Here, a method has been disclosed for learning a policy for each agent that maximizes the common reward of all agents while satisfying safe constraints specific to each agent in a constrained multi-agent reinforcement learning system consisting of multiple agents (see, for example, Non-Patent Document 1). In this method, by learning a policy for each agent, it is possible to avoid an exponential increase in the dimensions of the action space to be learned. In other words, in this method, the number of dimensions of the action space to be learned increases exponentially as the number of agents increases. <1> This can avoid the problem. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-014300 A [Patent Document 2] JP 2020-080103 A [Patent Document 3] US Patent Application Publication No. 2021 / 0200163 [Non-patent literature]

[0005] [Non-Patent Document 1] Gu, Shangding, et al. “Multi-agent constrained policy optimization.” arXiv preprint arXiv:2110.02793(2021). Summary of the Invention [Problem to be solved by the invention]

[0006] However, conventional methods have the problem that they cannot perform learning that takes into account the constraints of the entire system, which depends on the results of the actions of multiple agents. As a result, performance during and after learning cannot be guaranteed. <2> It can be said that there is a problem.

[0007] In one aspect, the present invention aims to perform learning in consideration of the constraints of the entire system in constrained multi-agent reinforcement learning. [Means for solving the problem]

[0008] In one aspect, a learning program causes a computer to execute the following processes in a constraint-subject control problem in which multiple agents are present: for a first agent among the multiple agents, determine a state priority to be used when taking action based on the relationships between the multiple agents; for the first agent, select a true value or an alternative value as a value to be input into a first policy parameter according to the state priority; for the first agent, use the first policy parameter to which the value has been input to determine, in accordance with a predetermined update order, a first influence on the constraint conditions of the first agent and a third influence on the constraint conditions of the entire system based on a second influence of a second policy parameter updated by a second agent that comes earlier in the update order; and determine a range of policy parameters that satisfy the constraint conditions according to the first influence and the third influence. Effect of the Invention

[0009] According to one embodiment, learning can be performed taking into account constraints on the entire system that depend on the results of the actions of multiple agents. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating an example of a functional configuration of an information processing device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of priority information according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an image of the multi-agent reinforcement learning process according to the embodiment. [Figure 4A] FIG. 4A is a diagram (1) showing an example of an overall flowchart of the learning process according to the embodiment. [Figure 4B] FIG. 4B is a diagram (2) showing an example of an overall flowchart of the learning process according to the embodiment. [Figure 4C] FIG. 4C is a diagram (3) showing an example of an overall flowchart of the learning process according to the embodiment. [Figure 4D] FIG. 4D is a diagram (4) showing an example of an overall flowchart of the learning process according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of an overall flowchart of the prediction process according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example in which the multi-agent reinforcement learning according to the embodiment is applied. [Figure 7A] FIG. 7A is a diagram (1) illustrating an example of priority determination according to the embodiment. [Figure 7B] FIG. 7B is a diagram (2) illustrating an example of priority determination according to the embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a method for setting some states to alternative values ​​according to the priority of the states. [Figure 9A] FIG. 9A is a diagram (1) showing an example of a result of application of multi-agent reinforcement learning according to the embodiment. [Figure 9B] FIG. 9B is a diagram (2) showing an example of a result of application of the multi-agent reinforcement learning according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a computer that executes a learning program. [Figure 11] FIG. 11 is a diagram showing a reference example of the multi-agent reinforcement learning process. [Figure 12A] FIG. 12A is a diagram (1) showing an image of a multi-agent reinforcement learning process taking into account constraints on the entire system. [Figure 12B] FIG. 12B is a diagram (2) showing an image of the multi-agent reinforcement learning process taking into account the constraints of the entire system. [Figure 13A] FIG. 13A is a diagram (1) showing an example of an overall flowchart of a multi-agent reinforcement learning process taking into account constraints on the entire system. [Figure 13B] FIG. 13B is a diagram (2) showing an example of an overall flowchart of the multi-agent reinforcement learning process taking into account the constraint conditions of the entire system. [Figure 13C] FIG. 13C is a diagram (3) showing an example of an overall flowchart of the multi-agent reinforcement learning process taking into account the constraint conditions of the entire system. [Figure 14] FIG. 14 is a diagram for explaining a problem of multi-agent reinforcement learning processing taking into account constraints of the entire system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the learning program, information processing device, and learning method disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments.

[0012] First, we will explain a method of multi-agent reinforcement learning with constraints consisting of multiple agents. Multi-agent reinforcement learning here is a method in which multiple agents interact with a common environment and solve a shared problem while making decisions sequentially. In other words, in multi-agent reinforcement learning with constraints, each agent observes the current "state" and learns a policy to minimize (or maximize) the "reward" (objective function) while satisfying the "constraints" by using a series of "actions" as clues. The "constraints" here refer to conditions under which performance is guaranteed. One example of multi-agent reinforcement learning with constraints is learning an algorithm for controlling the outage of each BS (Base Station) in a communication network in each area, which minimizes the total power consumption of all BSs while maintaining the average satisfaction of all users connected to the BSs at a certain level or higher. The BSs here are an example of agents. Maintaining the average satisfaction of all users at a certain level or higher is an example of a "constraint". The time, the demand of the grid, the load of the BS at the previous time, and the power consumption of the BS at the previous time are examples of "states". The sum of the power consumption of all BSs is an example of a "reward" (objective function). Whether or not to shut down a BS (turning the BS on / off) is an example of an "action."

[0013] In one method of multi-agent reinforcement learning processing, a policy for each agent that maximizes the common reward of all agents is learned while satisfying the safe constraints specific to each agent. In this method, by learning a policy for each agent, the behavior of each agent is understood, and an exponential increase in the dimension of the action space can be avoided.

[0014] However, this method has a problem in that it is not possible to perform learning that takes into account the constraints of the entire system that depend on the results of the actions of multiple agents. This problem will be explained. Figure 11 is a diagram showing a reference example of multi-agent reinforcement learning processing. In Figure 11, there are three agents.

[0015] As shown in Fig. 11, in the multi-agent reinforcement learning process, each agent updates its own policy to satisfy its own unique constraint conditions. Here, in one learning step, agent a calculates an update range from the policy before the update to satisfy its own unique constraint conditions, and updates the policy by the calculated update range. In the same learning step, agents b and c similarly calculate an update range from the policy before the update to satisfy their own unique constraint conditions, and update the policy by the calculated update range.

[0016] However, a method in which each agent updates its own policy to satisfy its own unique constraints does not necessarily satisfy the constraints of the entire system. In other words, the updated policy of a certain agent may deviate from the range of policies that satisfy the constraints of the entire system. In other words, such a method cannot perform learning that takes into account the constraints of the entire system that depend on the results of the actions of multiple agents.

[0017] Therefore, one possible approach is to calculate the impact of updating each agent's policy for each of the constraints of the entire system, in addition to the constraints specific to each agent, and then limit the extent to which the policy can be updated at each learning step according to that impact.

[0018] However, when each agent updates its policy in turn, even if an earlier agent updates so as to satisfy the constraints, the constraints may not be satisfied due to the update of a later agent. For this reason, it is necessary to appropriately update the policy of each agent, but there is no specific and quantitative method for doing so.

[0019] Here, we will explain a constrained multi-agent reinforcement learning process that can appropriately update the policy of each agent and can perform learning taking into account the constraints of the entire system.

[0020] [Image of multi-agent reinforcement learning processing] An image of the multi-agent reinforcement learning process considering the constraint conditions of the entire system will be described with reference to Fig. 12A and Fig. 12B. Fig. 12A and Fig. 12B are diagrams showing an image of the multi-agent reinforcement learning process considering the constraint conditions of the entire system. It is assumed that there are three agents, a, b, and c. The update order of the policy parameters of each agent is randomly determined as a, b, and c by an information processing device. It is assumed that the influence of the updated policy parameters of agent a is shared with agent b, which is next in the update order.

[0021] Then, the information processing device calculates the influence of the next agent b on the inherent constraint condition and the influence of the updated policy parameter of the previous agent a on the constraint condition of the entire system, which are calculated by sharing the influence of the updated policy parameter of the agent a. That is, the information processing device calculates the influence of the agent b on the inherent constraint condition, and calculates the influence of the updated policy parameter of the agent a on the constraint condition of the entire system by sharing the influence of the updated policy parameter of the agent a. Then, the information processing device determines whether or not the update range of the policy parameter of the agent b exists within the range determined by the influence of the agent b on the inherent constraint condition and the influence of the policy parameter of the entire system. In other words, the information processing device determines whether or not the possible range of the updated policy parameter of the agent b exists within the range determined by the influence of the agent b on the inherent constraint condition and the influence of the policy parameter of the entire system.

[0022] Here, the possible range of the updated policy parameters of each agent is calculated based on, for example, equation (1).

number

[0023] The left side of equation (1) is h The policy parameter π that satisfies the agent-specific constraints and the system-wide constraints ofih In other words, the left side of equation (1) represents the possible range of the policy parameters after the update. The right side of equation (1) is composed of (1.1), (1.2), and (1.3).

[0024] (1.1) is the same as for the own agent i h The policy parameter π for the previous learning step k in k ih This is the condition to prevent a large change from . The explanation of the variables in (1.1) is as follows. The variable π ih Agent i h The policy parameters after the update are shown in the table. k ih Agent i h The policy parameters before the update (learning step k) are shown as variables D. kL max denotes the maximum value of the Kullback-Leibler divergence between two policy parameters. The variable δ denotes an upper bound on the change of the policy parameters before and after the update.

[0025] (1.2) is the h The condition for satisfying the unique constraint condition j of (1.2) is as follows. The explanation of the variables in (1.2) is as follows. Variable J j ih (·), variable c j ih Agent i h We denote the cost function and its upper bound for the unique constraint j of the variable L. ih j,πk (π ih ) is the agent i h The variable v indicates the degree of influence of the updated policy parameters on the unique constraint j. j ih Agent i h The coefficients for the change in the policy parameters before and after the update for the unique constraint condition j of the variable m ih Agent i h Denote the number of unique constraints.

[0026] (1.3) is the condition for satisfying the constraint p of the whole system, including the influence shared by the agents in the previous update order. The explanation of the variables in (1.3) is as follows. J p (·),d p denotes the cost function and its upper bound with respect to the constraint p of the entire system. k+1 i1:h-1 The update order is agent i h Agents before (i1, ,i h-1 ) after the policy parameters are updated. The variable L denoted by symbol a1 is h and the update order is agent i h Agents before (i1, ,i h-1 ) on the constraint condition p of the entire system. The variable L, l = 1, , h-1 indicated by the symbol a2 indicates the degree of influence of the updated policy parameters on the constraint condition p of the entire system. h Shared by an agent earlier than agent i h This indicates the degree of influence of the updated policy parameters of the agent before v on the constraint condition p of the entire system. p indicates the coefficient of change before and after updating the policy parameters for the constraint p of the entire system. The variable M indicates the number of constraints of the entire system.

[0027] In other words, the information processing device determines whether or not the update range of the policy parameter of its own agent is within the range determined by the influence of its own agent on the constraint condition specific to the agent in (1.1) and (1.2) of formula (1).Then, the information processing device determines whether or not the update range of the policy parameter of its own agent is within the range determined by the influence of its own agent on the constraint condition of the entire system in formula (1.3).That is, the information processing device determines whether or not the update range of the policy parameter of its own agent is within the range determined by the influence of its own agent on the constraint condition specific to the agent and the influence of its own agent on the constraint condition of the entire system in formula (1) constituting (1.1) to (1.3).

[0028] Here, as shown in FIG. 12A, the range determined by the degree of influence on the unique constraint condition of agent b is the range indicated by the symbol rb. This range is expressed by both the left and right sides of (1.1) and (1.2) in equation (1). Moreover, the range determined by the degree of influence on the constraint condition of the entire system, which shares the degree of influence of the updated policy parameters of agent a, is the range indicated by the symbol sb. This range is expressed by both the left and right sides of (1.3) in equation (1). And the possible range of the updated policy parameters of agent b is the range indicated by the symbol mb. This range is expressed by both the left and right sides of equation (1), and the possible updated policy parameters π of agent b ih In other words, the range indicated by the symbol mb is the overlapping portion of the range indicated by the symbol rb and the range indicated by the symbol sb. If there is at least one updated policy parameter in the range indicated by the symbol mb, then there exists a possible range of the updated policy parameters of agent b, and there exists an update range for the policy parameters. Note that the update range here refers to the line segment that connects a certain point within the range indicated by the symbol mb to the policy parameter before the update, when this point is set as the updated policy parameter.

[0029] In Fig. 12A, the information processing device performs the following process since the update range of the policy parameters of its own agent b is within a range (mb) determined by the influence of its own agent b on its own constraint condition and the influence of its own agent b on the constraint condition of the entire system. The information processing device updates the policy parameters of its own agent with the existing update range. Then, the information processing device shares the influence of the updated policy parameters with the next agent in the update order.

[0030] Then, the information processing device calculates the influence of the next agent c in the update order on the inherent constraint conditions and the influence of the updated policy parameters of the previous agent b in the update order on the constraint conditions of the entire system, which are shared. That is, the information processing device calculates the influence of the agent c on the inherent constraint conditions, and calculates the influence of the updated policy parameters of the agent b on the constraint conditions of the entire system by sharing the influence of the updated policy parameters of the agent b. Then, the information processing device determines whether or not the update range of the policy parameters of the agent c exists within a range determined by the influence of the agent c on the inherent constraint conditions and the influence of the policy parameters of the entire system. In other words, the information processing device determines whether or not the possible range of the updated policy parameters of the agent c exists within a range determined by the influence of the agent c on the inherent constraint conditions and the influence of the policy parameters of the entire system.

[0031] Here, as shown in Fig. 12A, the range determined by the influence of agent c on its own specific constraint is the range indicated by the symbol rc. The range determined by the influence of agent b on the constraint of the entire system, sharing the influence of the updated policy parameters of agent b, is the range indicated by the symbol sc. The possible range of the updated policy parameters of agent c on its own is the overlapping part of the range indicated by the symbol rc and the range indicated by the symbol sc. If there is at least one updated policy parameter within this range, then there exists a possible range of the updated policy parameters of agent c, and there exists an update range for the policy parameters.

[0032] In Fig. 12A, the information processing device performs the following process since the update range of the policy parameters of its own agent c is within the range determined by the influence of its own agent c on the specific constraint condition and the influence of its own agent c on the constraint condition of the entire system. That is, the information processing device updates the policy parameters of its own agent with the existing update range.

[0033] This allows the information processing device to update the policy parameters of each agent within a range that satisfies the constraints of the entire system, thereby enabling the information processing device to perform learning that takes into account the constraints of the entire system that depend on the results of the actions of multiple agents.

[0034] FIG. 12B shows an example in which the update of the policy parameters is terminated for an agent in the middle of the order. As shown in FIG. 12B, the information processing device has an update range wb of the policy parameters of agent b within a range (rb) determined by the influence of agent b on its own constraint condition, and an update range wb of the policy parameters of agent b within a range (sb) determined by the influence on the constraint condition of the entire system. That is, the information processing device has an update range wb of the policy parameters of agent b within a policy range (sb) determined by the influence on the constraint condition of the entire system caused by the update of the policy of agent a in the previous update order. For this reason, the information processing device updates the policy parameters of agent b, and shares the influence of the updated policy parameters with agent c in the next update order.

[0035] Next, the information processing device calculates the influence of agent c on the unique constraint conditions, and calculates the influence of the updated policy parameters of agent b on the constraint conditions of the entire system by sharing the influence of the updated policy parameters of agent b. Then, the information processing device judges whether or not the update range of the policy parameters of agent c exists within a range determined by the influence of agent c on the unique constraint conditions and the influence of the updated policy parameters of the entire system. In other words, the information processing device judges whether or not the possible range of the updated policy parameters of agent c exists within a range determined by the influence of agent c on the unique constraint conditions and the influence of the updated policy parameters of the entire system. Here, the update range wc of the policy parameters of agent c exists within a range (rc) determined by the influence of agent c on the unique constraint conditions. However, the update range wc of the policy parameters of agent c does not exist within a range (sc) determined by the influence of agent c on the constraint conditions of the entire system. There is no common part of the update ranges in the overlapping ranges (rc,sc). For this reason, the information processing device does not update the policy parameters of agent c. In addition, the information processing device does not update the policy parameters of the subsequent agents in the update order.

[0036] This is because there is no update range in which the policy parameters can be updated. In other words, there is no common update range in the overlapping part between the policy parameter range satisfying the agent-specific constraints and the policy parameter range satisfying the system-wide constraints. As a result, the information processing device can ensure that not only the constraints specific to each agent but also the system-wide constraints that depend on the action results of multiple agents are satisfied during and after learning by avoiding updating the policy parameters of agent c and onwards.

[0037] [Flowchart of multi-agent reinforcement learning process] Next, an example of an overall flowchart of the multi-agent reinforcement learning process taking into account the constraint conditions of the entire system will be described. Figures 13A to 13C are diagrams showing an example of an overall flowchart of the multi-agent reinforcement learning process taking into account the constraint conditions of the entire system.

[0038] As shown in FIG. 13A, the information processing device initializes learning parameters (step S11). The information processing device initializes the number of learning episodes (step S12). Here, the information processing device sets the number of learning episodes to "1" as a variable e. The information processing device executes control for a certain period of time as episode e, and saves the combination history of state, action, and reward during that time in combination history information for each agent (step S13). Note that the combination history information here refers to history information that stores the combination of state, action, and reward for each agent in the learning episode.

[0039] As shown in FIG. 13B, the information processing device randomly determines the update order of the agents (step S21). The information processing device randomly determines the update order of the agents, for example, i1, i2, . . . , i n (step S22). Here, n is the number of agents. The information processing device initializes an agent update index h (step S23). Here, the information processing device sets the variable h to "1".

[0040] The information processing device uses the combination history information to h The information processing device calculates the degree of influence (impact A) on a specific constraint condition (step S24). The information processing device calculates the degree of influence (impact B) on the constraint condition of the entire system using the combination history information (step S25). That is, the information processing device calculates the degree of influence B on the constraint condition of the entire system based on the degree of influence shared by the agent in the previous update order.

[0041] The information processing device detects the agent i within a range determined by the influence degree A and the influence degree B. hIt is determined whether there is an update range for the policy parameter of agent i within the range determined by the influence degree A and the influence degree B (step S26). h If it is determined that there is no update range for the policy parameter of agent i (step S26; No), the information processing device performs the following process. h The policy parameters of the first agent and the subsequent agents are not updated, and the process proceeds to step S41.

[0042] On the other hand, within the range determined by the influence degree A and the influence degree B, agent i h If it is determined that there is an update range for the policy parameter (step S26; Yes), the information processing device performs the following process. That is, the information processing device performs the update range for the policy parameter for the agent i within the range determined by the influence degree A and the influence degree B. h The policy parameters are updated (step S27).

[0043] Then, the information processing device judges whether the agent update index h is smaller than the number of agents n (step S28). If it is judged that h is equal to or larger than n (step S28; No), the information processing device proceeds to step S41 since the processing for the number of agents has been completed.

[0044] If it is determined that h is smaller than n (step S28; Yes), the information processing device selects the next agent i h+1 (Step S29). Then, the information processing device updates the agent update index h by 1 (Step S30). Then, the information processing device updates the agent update index h by 1 (Step S31). h Then, the process proceeds to step S24 to perform the update process.

[0045] 13C, in step S41, the information processing device updates the learning parameters excluding the policy parameters (step S41). The information processing device updates the number of learning episodes by 1 (step S42).

[0046] Then, the information processing device determines whether the number of learning episodes e is greater than the maximum number of episodes (step S43). If it is determined that the number of learning episodes e is not greater than the maximum number of episodes (step S43; No), the information processing device proceeds to step S13 to process the next learning episode.

[0047] On the other hand, if it is determined that the number of learning episodes e is greater than the maximum number of episodes (step S43; Yes), the information processing device ends the multi-agent reinforcement learning process.

[0048] 13A to 13C, when an episode e is updated, the information processing device executes control for a certain period of time, and saves the combination history of states, actions, and rewards during that time in combination history information for each agent. Then, it has been explained that the information processing device uses this combination history information to update the policy of each agent in the updated episode e. However, the information processing device 1 is not limited to this, and when an episode e is updated, the information processing device 1 may update the policy of each agent in the updated episode e using the combination history already saved in the combination history information.

[0049] [Learning process issue] In the multi-agent reinforcement learning process considering the constraints of the entire system, each agent inputs the true values ​​of all states obtained by executing control for a certain period of time to learn a policy. Then, even when each agent uses each policy after learning to decide (predict) an action, the true values ​​of all states are input to the policy after learning of each agent to decide (predict) an action. Therefore, in the multi-agent reinforcement learning process considering the constraints of the entire system, it is necessary to aggregate the true values ​​of all states to each agent even in prediction after learning, and there is a problem that the cost of aggregating the state information of all agents to each agent is high. In other words, it is necessary to exchange the true values ​​of the states between all agents, and there is a problem that the cost of exchanging state information between all agents is high. Such a problem will be described with reference to FIG. 14.

[0050] Fig. 14 is a diagram explaining the problem of multi-agent reinforcement learning processing considering the constraints of the entire system. The left diagram of Fig. 14 shows the processing flow during learning, and the right diagram of Fig. 14 shows the processing flow during prediction after learning.

[0051] As shown in the left diagram of FIG. 14, in the learning process, the information processing device acquires N states s and actions a as the environment, inputs the acquired environment and reward, and learns the policy (machine learning model) of each agent to determine the behavior of each agent. The N states s and actions a acquired as the environment here are, for example, (s1, , s N ), (a1,...a N ), which means the state and action to which each agent applies an action. As an example, the state and action to which agent 1 applies an action are s1 and a1. The state and action to which agent 2 applies an action are s2 and a2. Furthermore, reward indicates the reward value function and cost value function, and indicates the objective function using the state of each agent.

[0052] The N states s input to the learning process are the true values ​​of the states acquired by each agent. That is, in the learning process, the information processing device inputs the true values ​​of the states acquired by each agent, and learns the policy (machine learning model) of each agent in order to determine the behavior of each agent. As an example, in order to learn the policy (machine learning model) of agent 1, the information processing device needs to input the true values ​​of the states acquired by each agent.

[0053] Also, as shown in the right diagram of FIG. 14, in the process of prediction after learning, the information processing device acquires N states s as the environment, inputs the acquired environment, and determines (predicts) the behavior of each agent using the policy after learning of each agent. The N states s input in the process of prediction after learning are the true values ​​of the states acquired from each agent. That is, in the process of prediction after learning, the true values ​​of all the states acquired by each agent are given to determine the behavior of each agent. Therefore, in the multi-agent reinforcement learning process considering the constraint conditions of the entire system, it is necessary to aggregate the true values ​​of all the states to each agent even in the process of prediction after learning, and there is a problem that the cost of aggregating the state information of all the agents to each agent is high. In other words, it is necessary to exchange the true values ​​of the states between all the agents, and there is a problem that the cost of exchanging the state information between all the agents is high.

[0054] In the following embodiment, a multi-agent reinforcement learning process with constraints will be described, which can perform learning taking into account the constraints of the entire system while reducing the number of states that give true values. EXAMPLES

[0055] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device according to an embodiment. The information processing device 1 shown in FIG. 1 performs a multi-agent reinforcement learning process with constraint conditions that can perform learning taking into account the constraint conditions of the entire system. In addition, the information processing device 1 determines the priority of the state used for action decision for each agent, selects a true value or an alternative value as a value of the state to be input to the policy of each agent according to the priority, and updates the policy of each agent. The method of updating the policy is the same as that of the multi-agent reinforcement learning process taking into account the constraint conditions of the entire system. That is, the information processing device 1 calculates the influence of each agent on its own unique constraint conditions and the constraint conditions of the entire system in turn, and updates the policy within this range only if the update width is within the range determined by the calculated influence degree. Then, when the information processing device 1 updates the policy, it causes the influence degree of the updated policy on the constraint conditions of the entire system to be shared with the next agent in the update order. The policy here includes a policy parameter.

[0056] The information processing device 1 has a control unit 10 and a storage unit 40. The control unit 10 has a learning unit 20 and a prediction unit 30. The storage unit 40 has priority information 41 and combination history information 42. The priority determination unit 21 is an example of a priority determination unit. The selection unit 24 is an example of a selection unit. The policy update unit 26 is an example of a determination unit.

[0057] The combination history information 42 is history information that stores the combination of state, action, and reward for each agent in a learning episode. A learning episode here means a section from the start to the end of an action on the environment given to reinforcement learning. For example, when learning an algorithm for shutting down control of each BS (agent), the state, action, and reward are as follows. The state is information indicating the time, the demand of the grid, the load of the BS at the previous time, the power consumption of the BS at the previous time, etc. The action is information indicating whether the BS is on / off (whether to shut down or not). The reward is information indicating the sum of the power consumption of all BSs.

[0058] The priority information 41 is information that stores the priority of a state used when determining an action for each agent. The priority of a state for one agent represents the priority (importance) of the state obtained from the relationship with other agents. Here, an example of the priority information 41 will be described with reference to FIG. 2.

[0059] 2 is a diagram showing an example of priority information according to an embodiment. As shown in FIG. 2, the priority information 41 stores the priority of a state used for determining the behavior of an agent. That is, the priority p of a state for an agent i is i is the priority p of N states. isj (j=1,2,...N). The N states represent the states of each target to which each agent applies an action.

[0060] As an example, in the first line, the state s that agent 1 uses to decide its action is j Priority p of (j=1, ,N) 1sj In other words, p isj is the state s used to determine the action of agent i j In other words, the state s for agent i j Priority of p isj is the state s obtained from the relationship with other agents. j Represents the priority (importance) of.

[0061] Returning to FIG. 1, the learning unit 20 includes a priority determination unit 21, a learning initialization unit 22, a determination unit 23, a selection unit 24, a calculation unit 25, a policy update unit 26, and a learning update unit 27.

[0062] The priority determination unit 21 determines the priority of a state used when determining an action for each agent. For example, the priority determination unit 21 identifies the relationship between agents using information collected in advance or through a preliminary experiment. Then, the priority determination unit 21 uses the relationship between the agents to determine the priority of each state used when determining the action of the agent. The relationship between agents here means, for example, an index showing how much the agents are related to each other. Examples of the index include the importance according to a decision tree algorithm and the correlation coefficient between agents. Then, the priority determination unit 21 stores the priority of each state in the priority information 41 in association with the agent.

[0063] As an example, the priority determination unit 21 learns according to a decision tree algorithm using input information (state, action) of each agent at time t and output information (state) at time t+1 collected in advance. Then, the priority determination unit 21 identifies relationships between agents using the learned decision tree, and calculates the importance (priority) of the state of each target to which each agent applies an action using the identified relationships between agents.

[0064] Examples of decision tree algorithms that are methods for determining the importance (priority) of a state include Random Forest and XGBoost. The method for determining the importance (priority) of a state is not limited to decision tree algorithms, and a correlation function between agents may be used. Values ​​used as the importance (priority) of a state include permutation importance and feature importance when the method is a decision tree algorithm. Values ​​used as the importance (priority) of a state may be the absolute value of a correlation coefficient when the method is a correlation function between agents. Values ​​used as the importance (priority) of a state may be the absolute value of a correlation coefficient and the p-value of a non-correlation test when the method is a correlation function between agents.

[0065] In addition, in one example, the priority determination unit 21 calculates the relationship between agents and the priority of the state as the same thing, but they may be separated. For example, the priority determination unit 21 calculates a correlation coefficient between agents as the relationship between the agents using data collected in advance. In addition, the priority determination unit 21 learns a random forest using input information (state, action) at time t and output information (state) at time t+1 of each agent collected in advance. Then, the priority determination unit 21 may use the importance of the learned random forest to calculate the priority of a state to which an agent with a high calculated correlation coefficient applies an action.

[0066] The learning initialization unit 22 initializes data used in learning. For example, the learning initialization unit 22 initializes learning parameters used in reinforcement learning. The learning initialization unit 22 also initializes learning episodes used in reinforcement learning. The learning initialization unit 22 also acquires a combination history of states, actions, and rewards for a certain period of time in the current learning episode for each agent, and stores the combination history information 42.

[0067] The decision unit 23 randomly decides the update order of the policy parameters of each agent, thereby enabling the decision unit 23 to update the policy parameters of each agent evenly.

[0068] The selection unit 24 selects, for its own agent, a true value or an alternative value for the state value to be input to its own policy parameter according to the priority information 41 in accordance with the determined update order. For example, the selection unit 24 selects, from the priority information 41, a state s j Priority of p isj (j=1, . . . , N). Then, the selection unit 24 obtains the priority p isj is compared with a predetermined threshold, and the priority p isj If is equal to or larger than the threshold, the state s j Get the truth value of p isj If is less than the threshold, state s jThe selection unit 24 changes the state s corresponding to its own agent i to an alternative value. j (j = 1, ,N).

[0069] The threshold value may be determined as follows. As an example, when the value used as the priority of a state is the importance according to a decision tree algorithm, the threshold value may be 1 / 10 of the minimum value of the importance of multiple elements included in the state to which each agent applies its action, and if the threshold value is equal to or greater than the threshold value, a true value may be given. As another example, when the value used as the priority of a state is the absolute value of a correlation coefficient, the true value may be given if the absolute value of the correlation coefficient with the state to which each agent applies its action is 0.2 (or 0.4) or greater. That is, 0.2 (or 0.4) may be set as the threshold value. As another example, when the value used as the priority of a state is the absolute value of a correlation coefficient and the p-value of a no-correlation test, the true value may be given if the absolute value of the correlation coefficient with the state to which each agent applies its action is 0.2 (or 0.4) or greater and the p-value of the no-correlation test is less than the significance level of 0.05. In such a case, 0.2 and 0.05 may be set as the threshold values.

[0070] As an example, the replacement value may be the average value of the state. As another example, the replacement value may be the median value of the state, a predicted value, an initial state, or an equilibrium point. As another example, the replacement value may be "0".

[0071] The calculation unit 25 calculates the influence of its own agent on the inherent constraint conditions and the influence of the policy update of the agent in the previous update order on the constraint conditions of the entire system according to the determined update order. For example, the calculation unit 25 calculates the influence of its own agent i on the inherent constraint conditions according to the update order determined by the determination unit 23. Then, the calculation unit 25 calculates the influence of the updated policy parameters of the agent i-1 in the previous update order on the constraint conditions of the entire system, sharing the influence of the updated policy parameters of the agent i-1 in the previous update order. The calculation unit 25 calculates the influence of the state s corresponding to its own agent i on the inherent constraint conditions. j(j=1, . . . , N) and the combination history information 42 are used to calculate the respective influences.

[0072] The policy update unit 26 updates the policy parameters of its own agent if the update range of the policy parameters of its own agent is within a range determined by the degree of influence of its own agent on the inherent constraint conditions and the degree of influence on the constraint conditions of the entire system. The update range here refers to the update range between the policy parameters before the update and the policy parameters after the update. In other words, if the possible range of the updated policy parameters of its own agent is within a range determined by the degree of influence of its own agent on the inherent constraint conditions and the degree of influence on the constraint conditions of the entire system, the policy update unit 26 updates the policy parameters of its own agent within this range. The possible range of the updated policy parameters is calculated, for example, based on formula (1).

[0073] In other words, the policy update unit 26 determines whether the update range of the policy parameters of its own agent is within the range determined by the degree of influence of its own agent on the constraint conditions specific to the agent in (1.1) and (1.2) of formula (1). Then, the policy update unit 26 determines whether the update range of the policy parameters of its own agent is within the range determined by the degree of influence of its own agent on the constraint conditions of the entire system in formula (1.3). That is, the policy update unit 26 determines whether the update range of the policy parameters of its own agent is within the range determined by the degree of influence of its own agent on the constraint conditions specific to the agent and the degree of influence of its own agent on the constraint conditions of the entire system in formula (1) constituting (1.1) to (1.3).

[0074] Then, when the policy update unit 26 determines that the update range of the policy parameter of its own agent exists within the range determined by the influence of its own agent on the specific constraint condition and the range determined by the influence of the entire system on the constraint condition, the policy update unit 26 performs the following process. That is, the policy update unit 26 updates the policy parameter of its own agent with the existing update range. This allows the policy update unit 26 to perform learning taking into account the constraint condition of the entire system that depends on the results of the actions of multiple agents. Furthermore, by learning the policies of its own agents in order, the policy update unit 26 grasps the actions of only its own agent, and can avoid an exponential increase in the dimensions of the action space. Then, the policy update unit 26 shares the updated influence with the next agent in the update order.

[0075] Furthermore, when the policy update unit 26 determines that the update range of the policy parameters of its own agent does not exist within the range determined by the influence of its own agent on the constraint conditions specific to the agent and the range determined by the influence of its own agent on the constraint conditions of the entire system, the policy update unit 26 performs the following process. That is, the policy update unit 26 does not update the policy parameters of its own agent and the agents following it in the update order. This is because there is no update range in which the policy parameters can be updated. In other words, there is no common part of the update range between the range of policy parameters that satisfies the constraint conditions specific to the agent and the range of policy parameters that satisfies the constraint conditions of the entire system. This allows the policy update unit 26 to guarantee the satisfaction of the constraint conditions of the entire system that depend on the action results of multiple agents as well as the constraint conditions specific to each agent during and after learning.

[0076] The learning update unit 27 updates data used in learning. For example, the learning update unit 27 updates learning parameters used in reinforcement learning. The learning parameters here refer to learning parameters other than policy parameters. The learning update unit 27 also increments the number of learning episodes used in reinforcement learning. If the number of learning episodes is smaller than the maximum number of learning episodes, the learning update unit 27 performs learning in the next learning episode. If the number of learning episodes reaches the maximum number of learning episodes, the learning update unit 27 ends learning. That is, a learned policy for each agent is generated.

[0077] The prediction unit 30 includes a selection unit 31 and a behavior prediction unit 32 .

[0078] The selection unit 31 selects, for the agent to be predicted, the acquired true value or alternative value as the value of the state to be input to its own policy, according to the priority information 41. For example, the selection unit 31 selects, from the priority information 41, the state s j Priority p of (j=1, ,N) isj Then, the selection unit 31 obtains the priority p isj is compared with a predetermined threshold, and the priority p isj If is greater than or equal to the threshold, agent j sends state s j The selection unit 31 also obtains the true value of the priority p isj If is less than the threshold, state s j The selection unit 31 changes the state s corresponding to its own agent i to an alternative value. j (j = 1, ,N).

[0079] The behavior prediction unit 32 predicts the behavior of the agent to be predicted using the learned policy. For example, the behavior prediction unit 32 predicts the behavior of the agent i to be predicted in the state s j (j = 1, , N) is input to the trained policy, and the trained policy parameters are used to predict the behavior of agent i to be predicted.

[0080] [Image of multi-agent reinforcement learning processing] Here, an image of the multi-agent reinforcement learning process according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an image of the multi-agent reinforcement learning process according to the embodiment.

[0081] As shown in FIG. 3, the priority determination unit 21 determines each state (s1, . . . , s N ) priority p isj Determine the priority p isj is the state s of agent i in the decision-making process. j (j=1,...,N). For example, the priority determination unit 21 uses prior data to identify relationships between agents. Then, the priority determination unit 21 uses the relationships between the agents to determine the priority of each state used when determining the behavior of the agent. That is, the priority determination unit 21 determines the priority of each state (s1,...,s N Then, the priority determination unit 21 determines the priority of each state (s1, . . . , s N ) is stored in the priority information 41.

[0082] It is also assumed that combination history information 42 of combinations of states, actions, and rewards resulting from execution of control for a certain period of time is stored. The policy update unit 26 inputs the combinations of states, actions, and rewards for a certain period of time stored in the combination history information 42 to update the policies of each agent. Here, it is assumed that an environment e0 at a certain time point is input. The environment e0 at a certain time point refers to N states s at a certain time point. The states s are expressed as (s1,...,s N ) at the same time. Action a is input. Action a is input by dividing each action (a1,...,a n ) and at the same time the reward is entered.

[0083] In this case, the selection unit 24 performs the following in accordance with the update order of the agents. The selection unit 24 selects the state s corresponding to its own agent i from the priority information 41. j Priority of p isj (j=1, . . . , N). Then, the selection unit 24 obtains the priority p isj is set to a predetermined threshold r i Compared with, priority p isj is the threshold r i If so, the combination history information 42 indicates the state s j Get the truth value of p isj is the threshold r i If it is less than s, j The selection unit 24 changes the state s corresponding to its own agent i to an alternative value. j (j = 1, , N). Note that r i Let be the threshold value corresponding to agent i.

[0084] Symbol b1 denotes the state s corresponding to agent 1. j (j=1, . . . , N) is selected. The selection unit 24 selects the state s j Priority of p 1sj Then, the selection unit 24 obtains the priority p 1sj is compared with a predetermined threshold r1, and the priority p 1sj If is equal to or greater than the threshold r1, the state s j Get the truth value of p 1sj If is less than the threshold r1, state s j The selection unit 24 changes the state s j (j = 1, , N). In the generated state s, the states that have been changed to alternative values ​​are changed to black.

[0085] Then, the calculation unit 25 calculates the influence of the agent 1 on the unique constraint condition and the influence of the policy update of the previous agent on the constraint condition of the entire system. Then, the policy update unit 26 updates the policy parameters of the agent 1 if the update range of the policy parameters of the agent 1 is within the range determined by the influence of the agent 1 on the unique constraint condition and the influence of the policy update of the previous agent on the constraint condition of the entire system. On the other hand, if the update range of the policy parameters of the agent 1 is not within the range determined by the influence of the agent 1 on the unique constraint condition and the influence of the policy update of the previous agent on the constraint condition of the entire system, the policy update unit 26 does not update the policy parameters of the agent 1 and the subsequent agents on the update order. Then, when the policy update unit 26 updates the policy parameters of the agent 1, it makes the next agent on the update order share the influence after the update.

[0086] Also, the symbol bn represents the state s corresponding to agent n according to the update order. j This shows the case where (j = 1, , N) is selected. For agent n, the selection is the same as for agent 1. In the generated state s, the states that have been changed to alternative values ​​are changed to black.

[0087] Then, in the case of agent n, the calculation unit 25 calculates the influence of agent n on the specific constraint conditions and the influence of the previous agent in the update order on the constraint conditions of the entire system, in the same manner as in the case of agent 1. Then, the policy update unit 26 updates the policy parameters of agent n when the update range of the policy parameters of agent n is within the range determined by the influence of agent n on the specific constraint conditions and the influence of agent n on the constraint conditions of the entire system. On the other hand, when the update range of the policy parameters of agent n is not within the range determined by the influence of agent n on the specific constraint conditions and the influence of agent n on the constraint conditions of the entire system, the policy update unit 26 does not update the policy parameters of agent n and the subsequent agents in the update order. In this way, the multi-agent reinforcement learning process learns the policies of each agent.

[0088] As a result, in the multi-agent reinforcement learning process according to the embodiment, even if the number of states that give true values ​​for each agent is reduced during learning, the policy of each agent can be appropriately learned. Also, even if the number of states that give true values ​​for the agent to be predicted is reduced during prediction after learning, it is possible to appropriately determine (predict) an action using the learned policy. In addition, during prediction after learning, the cost of exchanging information about true values ​​of states between the agent to be predicted and other agents can be reduced.

[0089] [Flowchart of multi-agent reinforcement learning process] Next, an example of an overall flowchart of a multi-agent reinforcement learning process according to an embodiment will be described. Figures 4A to 4D are diagrams showing an example of an overall flowchart of a learning process according to an embodiment. Figure 5 is a diagram showing an example of an overall flowchart of a prediction process according to an embodiment.

[0090] [Learning process flowchart] As shown in FIG. 4A, the information processing device 1 identifies the relationships between the agents from the prior data (step S11). Then, the information processing device 1 determines the state priority p used by each agent when deciding on an action based on the identified relationships between the agents. isj (Step S12). Here, i is an index for identifying an agent, and indicates 1, 2, . . . , n. Here, j is an index for identifying a state, and indicates 1, 2, . . . , N.

[0091] Then, the information processing device 1 initializes the learning parameters (step S13). The information processing device 1 initializes the number of learning episodes e to "1" (step S14). The information processing device 1 executes control for a certain period of time by setting some states to substitute values ​​according to the state priorities determined for each agent as episode e, and saves the combination history of [state, action, reward] during that time in the combination history information 42 (step S15). Note that a flowchart of the process of setting some states to substitute values ​​according to the state priorities will be described later. Then, the information processing device 1 proceeds to step S21.

[0092] As shown in FIG. 4B, the information processing device 1 randomly determines the update order of the agents (step S21). The information processing device 1 randomly determines the update order of the agents, for example, i1, i2, . . . , i n (Step S22). Here, n is the number of agents. The information processing device 1 initializes an agent update index h (Step S23). Here, the information processing device 1 sets the variable h to "1".

[0093] The information processing device 1 uses the priority information 41 and the combination history information 42 to substitute some states according to the priority of the states, and executes the agent i h The degree of influence (degree of influence A) for the unique constraint condition is calculated (step S24). Note that a flowchart of the process of substituting some states for alternative values ​​according to the priority of the states will be described later.

[0094] Furthermore, the information processing device 1 calculates the impact (impact B) on the constraint conditions of the entire system by substituting some states according to the priority of the states using the priority information 41 and the combination history information 42 (step S25). Note that a flowchart of the process of substituting some states according to the priority of the states will be described later.

[0095] The information processing device 1 detects an agent i within a range determined by the influence degree A and the influence degree B. hIt is determined whether there is an update range for the policy parameter of agent i within the range determined by the influence degree A and the influence degree B (step S26). h If it is determined that there is no update range for the policy parameter of the agent i (step S26; No), the information processing device 1 performs the following process. h The policy parameters of the first agent and the subsequent agents are not updated, and the process proceeds to step S41.

[0096] On the other hand, within the range determined by the influence degree A and the influence degree B, agent i h If it is determined that there is an update range for the policy parameter (step S26; Yes), the information processing device 1 performs the following process. That is, the information processing device 1 performs the update range for the policy parameter for the agent i within the range determined by the influence degree A and the influence degree B. h The policy parameters are updated (step S27).

[0097] Then, the information processing device 1 judges whether the agent update index h is smaller than the number of agents n (step S28). If it is judged that h is equal to or larger than n (step S28; No), the information processing device 1 proceeds to step S41 since the processing for the number of agents has been completed.

[0098] If it is determined that h is smaller than n (step S28; Yes), the information processing device 1 selects the next agent i h+1 (Step S29). Then, the information processing device 1 updates the agent update index h by 1 (Step S30). Then, the information processing device 1 makes the next agent i h Then, the process proceeds to step S24 to perform the update process.

[0099] 4C, in step S41, the information processing device 1 updates the learning parameters excluding the policy parameters (step S41). The information processing device 1 updates the number of learning episodes by 1 (step S42).

[0100] Then, the information processing device 1 determines whether the number of learning episodes e is greater than the maximum number of episodes (step S43). If it is determined that the number of learning episodes e is not greater than the maximum number of episodes (step S43; No), the information processing device 1 proceeds to step S15 to process the next learning episode.

[0101] On the other hand, if it is determined that the number of learning episodes e is greater than the maximum number of episodes (Step S43; Yes), the information processing device 1 ends the multi-agent reinforcement learning process.

[0102] Here, an example of a flowchart of a process of substituting a part of the states according to the priority of the states will be described with reference to FIG. 4D. As shown in FIG. 4D, the information processing device 1 acquires the state s of the agent i from the combination history information 42 (step S51). The state s is (s1, . . . , s N ). The information processing device 1 initializes a state index j to "1" (step S52).

[0103] Then, the information processing device 1 calculates the state s of the agent i. j Priority of p iSj from the priority information 41 (step S53). Then, the information processing device 1 obtains the priority p iSj is the threshold r i It is determined whether or not the value is equal to or greater than the threshold (step S54). i is the threshold value corresponding to agent i. Priority p iSj is the threshold r i If it is determined that the state s of the agent i is less than s (step S54; No), the information processing device 1 j to the alternative value (step S55). Then, the information processing device 1 proceeds to step S56.

[0104] On the other hand, priority p iSj is the threshold r i If it is determined that the number of states is equal to or greater than the number of states (step S54; Yes), the information processing device 1j The true value of state s j Then, the information processing device 1 proceeds to step S56.

[0105] In step S56, the information processing device 1 updates the state index by 1 (step S56). Then, the information processing device 1 determines whether the state index j is greater than N, which is the maximum value of the index for identifying a state (step S57). If it is determined that the state index j is equal to or less than N (step S57; No), the information processing device 1 proceeds to step S53 to process the value of the next state.

[0106] On the other hand, if it is determined that the index j of the state is larger than N (step S57; Yes), the information processing device 1 ends the process of setting some states to substitute values ​​according to the priority of the states for the agent i.

[0107] [Prediction process flowchart] 5, the information processing device 1 initializes all elements of a state s (step S61). The state s here is defined as (s1, . . . , s N ). The information processing device 1 initializes each element of the state s to "0". Then, the information processing device 1 initializes the state index j to "1" (step S62).

[0108] Then, the information processing device 1 calculates the state s of the agent i. j Priority of p iSj from the priority information 41 (step S63). Then, the information processing device 1 obtains the priority p iSj is the threshold r i It is determined whether or not the value is equal to or greater than the threshold (step S64). i is the threshold value corresponding to agent i. Priority p iSj is the threshold r i If it is determined that the number of times is equal to or greater than the state number (step S64; Yes), the information processing device 1 j For example, the information processing device 1 acquires the true value of the state s jFrom the agent corresponding to state s j Then, the information processing device 1 proceeds to step S67.

[0109] On the other hand, priority p iSj is the threshold r i If it is determined that the state s of the agent i is less than s (step S64; No), the information processing device 1 j (step S66). Then, the information processing device 1 proceeds to step S67.

[0110] In step S67, the information processing device 1 updates the state index j by 1 (step S67). Then, the information processing device 1 determines whether the state index j is greater than N, which is the maximum value of the index for identifying a state (step S68). If it is determined that the state index j is equal to or less than N (step S68; No), the information processing device 1 proceeds to step S63 to process the value of the next state.

[0111] On the other hand, when it is determined that the index j of the state is larger than N (step S68; Yes), the information processing device 1 calculates the learned policy π i Using the action a i (step S69). Then, the information processing device 1 ends the prediction process for the agent i.

[0112] 4A to 4D, when an episode e is updated, the information processing device 1 executes control for a certain period of time and stores the combination history of states, actions, and rewards during that time for each agent in the combination history information 42. Then, it has been explained that the information processing device 1 uses this combination history information 42 to update the policy of each agent in the updated episode e. However, the information processing device 1 is not limited to this, and when an episode e is updated, the information processing device 1 may also use the combination history already stored in the combination history information 42 to update the policy of each agent in the updated episode e.

[0113] [Application of multi-agent reinforcement learning] Here, an application example of the multi-agent reinforcement learning according to the embodiment will be described with reference to Fig. 6, Fig. 7A, Fig. 7B, Fig. 8, Fig. 9A, and Fig. 9B. First, an application example of the multi-agent reinforcement learning according to the embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing an application example of the multi-agent reinforcement learning according to the embodiment. In Fig. 6, an experimental result in the case where an algorithm for controlling three cart poles (moving on three parallel rails) is learned will be described.

[0114] As shown in Figure 6, three cart poles are mounted on three parallel rails. Cart 1 and cart 2 are connected by springs, and there is interference between their state transitions. Each cart pole is an example of an "agent." If the distance between all carts is greater than 0.2, the cost is set to "1," otherwise the cost is set to "0," and the discounted cumulative cost in each episode (hereafter referred to as "discounted cost") is kept below 25. This is an example of a "constraint" for the entire system. The positions and velocities of the three carts and the angles and angular velocities of their poles are an example of a "state." The state in this case has a total of 12 (=3 cars x 4 types) dimensions. Moving each cart to the right or left (giving each cart an input of 0 or 1) is an example of an "action." The time from the initial state until the poles of all three carts do not fall is an example of a "reward (objective function)." This maximum time is 200 steps (seconds), and the initial state is set randomly within a certain range. In the experiments, in order to simplify the problem, we only considered the constraints of the entire system (without considering the constraints specific to each agent).

[0115] Under these experimental conditions, multi-agent reinforcement learning was attempted.

[0116] 7A and 7B are diagrams illustrating an example of priority determination according to the embodiment.

[0117] As shown in FIG 7A, the information processing device 1 learns a random forest (RF1 to RF3), which is an example of a decision tree algorithm, by using input / output information collected in advance. The input information in this case is the state s of all cart poles at time t. i and Action a i (i=1,2,3), where i is an index that identifies an agent. i (t) is the cart position x of agent i at time t i (t) Speed ​​v i (t) and its pole angle θ i (t) · Angular velocity ω i (t) consists of action a i (t) is the behavior of agent i, and indicates ±1 or 0 (no input). The output information of the input / output information is the state s of each cart pole at time t+1. i (i=1,2,3).

[0118] As shown in FIG. 7B, the information processing device 1 uses the trained random forest to determine the permutation importance of states used to determine the behavior of each agent. Then, the information processing device 1 stores the priority of each state in the priority information 41 in association with the agent. Here, agent 1 corresponds to cart pole 1. agent 2 corresponds to cart pole 2. agent 3 corresponds to cart pole 3. s1 corresponds to the state of cart pole 1 to which agent 1 applies a behavior, and [x1, v1, θ1, ω1] T s2 corresponds to the state of Cartpole 2 to which Agent 2 adds an action, [x2, v2, θ2, ω2] T s3 corresponds to the state of Cartpole 3 to which Agent 3 adds an action, [x3, v3, θ3, ω3] T It is.

[0119] For example, state s1(=[x1, v1, θ1, ω1]) of Cartpole 1 corresponding to Agent 1 TThe priority of agent 2 is determined as [0.15, 0.79, 0.083, 1.6]. T The priority of agent 3 is determined as [0.011, 0.0014, 0.0013, 0.0013]. T ) is determined to be [0.0014, 0.0008, 0.0011, 0.0008].

[0120] And the priority threshold r i For example, the threshold r1 of the priority of the state of the cart pole to which each agent adds an action is set to 1 / 10. The threshold r1 of the priority of the state of the cart pole to which the agent adds an action is set to "0.0083", which is 1 / 10 of the minimum value "0.083" of the priority of the state of the cart pole to which the agent adds an action, [0.15, 0.79, 0.083, 1.6] (symbol c1). The threshold r2 of the priority of the state of the cart pole to which the agent adds an action is set to "0.0054", which is 1 / 10 of the minimum value "0.054" of the priority of the state of the cart pole to which the agent adds an action, [0.14, 0.9, 0.054, 1.6] (symbol c2). The threshold r3 of the priority of the state of the cart pole to which the agent adds an action is set to "0.011", which is 1 / 10 of the minimum value "0.11" of the priority of the state of the cart pole to which the agent adds an action, [0.23, 1.1, 0.11, 1.1] (symbol c3).

[0121] The bold characters in the priority information 41 indicate priorities that are equal to or greater than the threshold. T is the state of Cart Pole 1 to which Agent 1 applies an action (symbol c1), so the priority is greater than or equal to the threshold r1. Also, the state of Cart Pole 1 corresponding to Agent 2 [x1, v1, θ1, ω1] T Among them, state x1 is connected to cart pole 2 by a spring and is subject to interference, so it has a priority level (symbol c4) that is equal to or higher than threshold r1. Tis the state of Cart Pole 2 to which Agent 2 adds an action (symbol c2), so the priority is greater than or equal to the threshold r2. Also, the state of Cart Pole 2 corresponding to Agent 1 [x2, v2, θ2, ω2] T Among them, state x2 is connected to cart pole 1 by a spring and is subject to interference, so it has a priority (symbol c5) that is equal to or higher than threshold r2. In addition, the state [x3, v3, θ3, ω3] of cart pole 3 corresponding to agent 3 T is the state of Cart Pole 3 to which Agent 3 applies an action (symbol c3), and therefore has a priority equal to or greater than the threshold r3.

[0122] Fig. 8 is a diagram showing an example of a method of setting some states to alternative values ​​according to the state priorities. In Fig. 8, control of three cart poles is learned by assigning true values ​​or alternative values ​​(e.g., "0") to each state under the conditions of Case 1 and Case 2 according to the state priorities of the priority information 41 shown in Fig. 7B. In Case 1, true values ​​are assigned to all elements of the state of each cart pole if the priority of even one element is equal to or higher than a threshold. In Case 2, true values ​​are assigned to only elements of the state of each cart pole whose priority is equal to or higher than a threshold.

[0123] The information processing device 1 determines the state s corresponding to its own agent i from the priority information 41. j Priority of p isj Then, the information processing device 1 acquires the priority p isj is set to a predetermined threshold r i Compared with, priority p isj is the threshold r i If it is, state s j Get the truth value of p isj is the threshold r i If it is less than s, j to an alternative value.

[0124] Here, the information processing device 1 detects the state s j Priority of p 1sjis obtained from the first row of the priority information 41 shown in FIG. 7B. Then, in the state s1 of Cart Pole 1, all elements are equal to or greater than the threshold r1, and in the state s2 of Cart Pole 2, the element x2 is equal to or greater than the threshold r1. In the case of Case 1, all elements of the state s1 of Cart Pole 1 are given true values ​​(x1, v1, θ1, ω1), all elements of the state s2 of Cart Pole 2 are given true values ​​(x2, v2, θ2, ω2), and all elements of the state s3 of Cart Pole 3 are given substitute values ​​(0, 0, 0, 0) (symbol d1). On the other hand, in the case of Case 2, all elements of the state s1 of Cart Pole 1 are given true values ​​(x1, v1, θ1, ω1), but in the state s2 of Cart Pole 2, only x2 is given true values, and the other elements are given substitute values ​​of "0". Then, all elements of the state s3 of Cart Pole 3 are given substitute values ​​(0, 0, 0, 0) (symbol d2).

[0125] Next, the information processing device 1 calculates the state s j Priority of p 2sj 7B from the second row of the priority information 41, and substitutes some states according to the state priorities in the same manner as in the case of agent 1. In addition, the information processing device 1 obtains the state s corresponding to agent 3. j Priority of p 3sj is obtained from the third row of the priority information 41 shown in FIG. 7B, and some states are set to substitute values ​​according to the state priorities, as in the case of agents 1 and 2.

[0126] As a result, when a true value or an alternative value (e.g., "0") is assigned to each state under the conditions of Case 1, the information processing device 1 can reduce the number of states to which true values ​​are assigned by 44%, compared to when true values ​​are assigned to all states. Also, when a true value or an alternative value (e.g., "0") is assigned to each state under the conditions of Case 2, the information processing device 1 can reduce the number of states to which true values ​​are assigned by 61%, compared to when true values ​​are assigned to all states. In other words, the information processing device 1 can reduce the amount of data exchanged between agents when making predictions after learning.

[0127] 9A and 9B are diagrams showing an example of the application result of the multi-agent reinforcement learning according to the embodiment. FIG. 9A shows a graph showing the accumulated reward (accumulated reward) for each episode when a true value or an alternative value is given to each state under the conditions of Case 1 and Case 2 and when a true value is given to all states. The X-axis of the graph shows the number of episodes. The Y-axis of the graph shows the accumulated reward. The graph shown in gray is the result of learning (Proposed Case 1) in which a true value or an alternative value is given to each state under the condition of Case 1. The graph shown in black is the result of learning (Proposed Case 2) in which a true value or an alternative value is given to each state under the condition of Case 2. And the graph shown by dots is the result of learning in which a true value is given to all states. Each broken line is the average value of five trials.

[0128] As shown in Figure 9A, the cumulative reward increases with learning in both the case of learning in which true or alternative values ​​are given to each state under the conditions of Cases 1 and 2, and in the case of learning in which true values ​​are given to all states. In other words, we can see that the reward increases with learning even if the number of states to which true values ​​are given is reduced.

[0129] FIG. 9B shows a graph showing the discounted cost for the constraint of the entire system for each episode when a true value or an alternative value is assigned to each state under the conditions of Cases 1 and 2 and when a true value is assigned to all states. The X-axis of the graph shows the number of episodes. The Y-axis of the graph shows the discounted cost. The dashed line is a threshold, which shows "25" as the discounted cost for the constraint of the entire system. The threshold is "25" because the constraint of the entire system is to keep the discounted cost in each episode below 25. The graph shown in gray is the result of learning in which a true value or an alternative value is assigned to each state under the conditions of Case 1 (Proposed Case 1). The graph shown in black is the result of learning in which a true value or an alternative value is assigned to each state under the conditions of Case 2 (Proposed Case 2). And the graph shown by dots is the result of learning in which a true value is assigned to all states. Each broken line is the average value of five trials.

[0130] As shown in Figure 9B, the discounted cost for the constraints of the entire system almost never exceeds the threshold value of 25, whether it is learning in which true or alternative values ​​for each state are given under the conditions of Cases 1 and 2, or learning in which true values ​​are given to all states. In other words, even if the number of states to which true values ​​are given is reduced, the constraints of the entire system are almost satisfied, even in episodes during learning.

[0131] Therefore, in the multi-agent reinforcement learning according to the embodiment, even if the number of states for which true values ​​are given is reduced during learning, it is possible to learn an appropriate policy for increasing rewards while satisfying the constraints of the entire system. In addition, even during prediction after learning, by reducing the number of states for which true values ​​are given, it is possible to reduce the cost of exchanging information on true values ​​of states between the agent to be predicted and other agents.

[0132] 6, 7A, 7B, 8, 9A, and 9B, a case where an algorithm for controlling three cart poles is learned has been described as an application example of the multi-agent reinforcement learning according to the embodiment. However, the multi-agent reinforcement learning according to the embodiment is not limited to this.

[0133] For example, the multi-agent reinforcement learning according to the embodiment may be applied to the case of learning an algorithm for shutting down control in a base station (BS) of a communication network. In such a case, an algorithm for shutting down control of each BS is learned so as to minimize the total power consumption of all BSs while maintaining the average satisfaction of all users connected to the BS in each area at a certain level or higher. Here, each BS is an example of an "agent". Maintaining the average satisfaction of all users at a certain level or higher is an example of a "constraint condition" including the entire system. Time, grid demand, BS load at the previous time, power consumption of BS at the previous time, etc. are examples of "state". Whether or not to shut down a BS (BS on / off) is an example of "action". The total power consumption of all BSs is an example of "reward (objective function)".

[0134] As a result, in multi-agent reinforcement learning, by learning and predicting low-priority states by giving alternative values, the amount of data collected at the time of prediction after learning can be reduced. A low-priority state is, for example, the state of an area far away from the area controlled by each agent. In addition, cooperative BS shutdown control over a wide area spanning multiple areas becomes possible, further improving power saving performance. In addition, multi-agent reinforcement learning can guarantee user satisfaction throughout the entire area without tuning the weight coefficients or penalty terms of the objective function, and can reduce the amount of work required for trial and error in learning.

[0135] In another example, multi-agent reinforcement learning may be applied to learning an algorithm for controlling air conditioning in a data center. In such a case, an algorithm for controlling each air conditioner is learned that minimizes the total power consumption of all the air conditioners installed in the data center while keeping the temperature of the servers in the data center below a certain value. In this case, keeping the temperature of the servers in the data center below a certain value is an example of a "constraint condition" including the entire system. The time, the temperature of the server, the power consumption of the server at the previous time, the set temperature of the air conditioner at the previous time, etc. are examples of "states". The set temperature of each air conditioner (or a command to raise or lower the set temperature), the strength of the air conditioning, etc. are examples of "actions". The total power consumption of all the air conditioners is an example of a "reward (objective function)".

[0136] As a result, in multi-agent reinforcement learning, by learning and predicting by giving alternative values ​​for low-priority states, it is possible to reduce the amount of data collected when making predictions after learning. A low-priority state is, for example, the state of a server that is far from the air conditioners controlled by each agent. It also enables cooperative air conditioning control of multiple air conditioners, further improving power saving performance. Furthermore, multi-agent reinforcement learning can guarantee that the temperature of servers in a data center will be below a certain value without tuning the weight coefficient or penalty term of the objective function, and it can reduce the amount of work required for trial and error in learning.

[0137] [Effects of the embodiment] According to the above embodiment, the information processing device 1 determines the priority of a state to be used when acting on a first agent among the agents in a control problem with constraint conditions in which a plurality of agents exist, based on the relationship between the plurality of agents. Then, for the first agent, the information processing device 1 selects a true value or an alternative value as a value to be input to a first policy parameter according to the priority of the state. Then, for the first agent, the information processing device 1 determines, according to a predetermined update order, a first influence degree of the first agent on the constraint conditions and a third influence degree of the constraint conditions of the entire system based on a second influence degree by a second policy parameter updated by a second agent that is an earlier order in the update order, using the first policy parameter to which a value has been input. Then, the information processing device 1 determines a range of policy parameters that satisfy the constraint conditions according to the first influence degree and the third influence degree. According to this configuration, the information processing device 1 can appropriately learn the policy of each agent even if the number of states that give a true value is reduced. For example, the information processing device 1 can appropriately learn the policy of each agent that increases the reward while satisfying the constraint, even if the number of states that give a true value is reduced.

[0138] According to the above embodiment, the information processing device 1 uses a predetermined decision tree algorithm to determine the priority of the state to be used when acting for the first agent. With this configuration, the information processing device 1 can specify the relationship between the first agent and other agents and calculate the priority of the state for the first agent by using the predetermined decision tree algorithm.

[0139] Furthermore, according to the above embodiment, the information processing device 1 uses the correlation between agents to determine the priority of the state to be used when acting for the first agent. With this configuration, the information processing device 1 can specify the relationship between the first agent and other agents by using the correlation between agents, and calculate the priority of the state for the first agent.

[0140] Also, according to the above embodiment, the information processing device 1 compares the priority of the determined state for the first agent with a threshold indicating the condition for the state to give a true value, and selects the true value or an alternative value as the value to be input. With this configuration, the information processing device 1 can select the true value or an alternative value as the value to be input by using the threshold indicating the condition for the state to give a true value.

[0141] Furthermore, according to the above embodiment, the information processing device 1 selects a true value or an alternative value acquired from another agent as a value to be input to the state of the agent to be predicted, according to the priority of the state. Then, the information processing device 1 inputs the state into which the value has been input to the learned policy, and predicts the behavior of the agent to be predicted using the learned policy parameters. According to this configuration, the information processing device 1 can reduce the cost of exchanging information on the true value of the state of the agent to be predicted with other agents at the time of prediction after learning.

[0142] In the illustrated information processing device 1, a case has been described in which the device has a learning unit 20 and a prediction unit 30. However, there may be a plurality of information processing devices 1, and a first information processing device among the plurality of information processing devices 1 may have the learning unit 20, and a second information processing device among the plurality of information processing devices 1 may have the prediction unit 30. In such a case, the second information processing device may store in the storage unit 40 the learned policy learned by the first information processing device.

[0143] Moreover, each component of the illustrated information processing device 1 does not necessarily have to be physically configured as illustrated. In other words, the specific form of distribution / integration of the information processing device 1 is not limited to that shown in the figure, and all or part of it can be functionally or physically distributed / integrated in any unit according to various loads, usage conditions, etc. Also, the storage unit 40 may be connected as an external device of the information processing device 1 via a network.

[0144] Moreover, the various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. Hereinafter, an example of a computer that executes a multi-agent reinforcement learning program that realizes the same functions as the information processing device 1 shown in Fig. 1 will be described. Here, a multi-agent reinforcement learning program that realizes the same functions as the information processing device 1 will be described as an example. Fig. 10 is a diagram showing an example of a computer that executes a learning program.

[0145] 10, the computer 200 includes a CPU (Central Processing Unit) 203 that executes various arithmetic processes, an input device 215 that accepts data input from a user, and a display device 209. The computer 200 also includes a drive device 213 that reads programs and the like from a storage medium, and a communication I / F (Interface) 217 ​​that transmits and receives data to and from other computers via a network. The computer 200 also includes a memory 201 that temporarily stores various information, and a HDD (Hard Disk Drive) 205. The memory 201, the CPU 203, the HDD 205, the display control unit 207, the display device 209, the drive device 213, the input device 215, and the communication I / F 217 are connected by a bus 219.

[0146] The drive device 213 is, for example, a device for the removable disk 211. The HDD 205 stores the learning program 205a and the learning process related information 205b. The communication I / F 217 manages the interface between the network and the inside of the device, and controls the input and output of data from other computers. For example, a modem or a LAN adapter can be used as the communication I / F 217.

[0147] The display device 209 is a display device that displays a cursor, an icon, a tool box, and data such as documents, images, function information, etc. The display device 209 can be, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0148] The CPU 203 reads out the learning program 205a, expands it in the memory 201, and executes it as a process. Such a process corresponds to each functional unit of the information processing device 1. The learning process related information 205b includes, for example, priority information 41 and combination history information 42. For example, the removable disk 211 stores each piece of information such as the learning program 205a.

[0149] It should be noted that the learning program 205a does not necessarily have to be stored in the HDD 205 from the beginning. For example, the program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD disk, magneto-optical disk, or IC card that is inserted into the computer 200. The computer 200 may then read out the learning program 205a from these and execute it. [Explanation of symbols]

[0150] 1. Information processing device 10 Control section 20 Learning Department 21 Priority determination section 22 Learning Initialization Unit 23 Decision Section 24 Selection section 25 Calculation section 26 Policy Update Division 27 Learning and Updating Department 30 Prediction Department 31 Selection section 32 Behavioral Prediction Department 40 Storage section 41 Priority information 42 Combination History Information

Claims

1. In a constrained control problem with multiple agents, determining a priority of a state to be used when acting on a first agent among the plurality of agents based on a relationship between the plurality of agents; selecting, for the first agent, a true value or an alternative value as a value to be input to a first policy parameter in response to the priority of the state; determining, for the first agent, a first influence on the constraint of the first agent in accordance with a predetermined update order using the first policy parameter into which the value has been input, and a third influence on the constraint of the entire system based on a second influence of a second policy parameter updated by a second agent preceding the first agent in the update order; determining a range of a policy parameter that satisfies a constraint condition according to the first degree of influence and the third degree of influence; A learning program that causes a computer to execute a process.

2. The process of determining the priority of the state uses a predetermined decision tree algorithm to determine the priority of the state to be used when acting on the first agent.

2. The learning program according to claim 1 .

3. The process of determining the priority of the state uses the correlation between agents to determine the priority of the state to be used when acting on the first agent.

2. The learning program according to claim 1 .

4. The process of selecting the true value or the alternative value includes comparing the determined state priority with a threshold value indicating a condition for the state that gives a true value for the first agent, and selecting the true value or the alternative value as the value to be input.

2. The learning program according to claim 1 .

5. selecting, for an agent to be predicted, a true value or an alternative value obtained from another agent as a value to be input to the state in accordance with the priority of the state; The state in which the values ​​are input is input to a trained policy, and the trained policy parameters are used to predict the behavior of the agent to be predicted.

2. The learning program according to claim 1 .

6. In a constrained control problem with multiple agents, a priority determination unit that determines a priority of a state to be used when acting on a first agent among the plurality of agents based on a relationship between the plurality of agents; a selection unit that selects, for the first agent, a true value or an alternative value as a value to be input to a first policy parameter according to the priority of the state; a first determination unit that determines, for the first agent, a first influence of the first agent on a constraint condition in accordance with a predetermined update order using the first policy parameter whose value has been input by the selection unit, and a third influence of the first agent on a constraint condition of the entire system based on a second influence of a second policy parameter updated by a second agent preceding the first agent in the update order; a second determination unit that determines a range of a policy parameter that satisfies a constraint condition according to the first influence degree and the third influence degree; 13. An information processing device comprising:

7. In a constrained control problem with multiple agents, determining a priority of a state to be used when acting on a first agent among the plurality of agents based on a relationship between the plurality of agents; selecting, for the first agent, a true value or an alternative value as a value to be input to a first policy parameter in response to the priority of the state; determining, for the first agent, a first influence on the constraint of the first agent in accordance with a predetermined update order using the first policy parameter into which the value has been input, and a third influence on the constraint of the entire system based on a second influence of a second policy parameter updated by a second agent preceding the first agent in the update order; determining a range of a policy parameter that satisfies a constraint condition according to the first degree of influence and the third degree of influence; A learning method characterized in that the processing is executed by a computer.

Images