Strategy learning device, strategy learning method, and communication system

Abstract

To provide a strategy learning device capable of efficiently learning an agent in reinforcement learning with constraints, a strategy learning method, and a communication system.SOLUTION: A strategy learning device that learns, by reinforcement learning, an agent that outputs action of a control target in accordance with an input of a state of the control target, includes: a parameter changing unit that changes a parameter related to constraint conditions in the reinforcement learning for each of a predetermined number of learning steps in the reinforcement learning; and an agent learning unit for learning the agent by using a parameter as at least part of the state.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a policy learning device, a policy learning method, and a communication system. [Background technology]

[0002] For example, there is a technology of reinforcement learning that learns an optimal policy for a controlled object (hereinafter also referred to as the environment) by referring to a reward from the controlled object according to an action toward the controlled object. The policy is, for example, a function that determines the next action of the controlled object according to the state of the controlled object. In other words, reinforcement learning is a technology that makes an agent learn a policy that can obtain a higher reward by repeating trial and error based on experience (for example, see Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2022 / 044191 [Patent Document 2] Patent Publication No. 2021-064222 Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned reinforcement learning includes, for example, constrained reinforcement learning (hereinafter also simply referred to as constrained reinforcement learning), which learns an optimal policy while observing a constraint condition (hereinafter also simply referred to as a constraint condition) that the cost from the control object is kept below a predetermined threshold.

[0005] In the above-mentioned constrained reinforcement learning, for example, when learning a policy corresponding to various constraint conditions, it is necessary to learn a different agent for each constraint condition. Therefore, in the above-mentioned constrained reinforcement learning, for example, there are cases where the agent learning cannot be performed efficiently.

[0006] Therefore, in one aspect, an object of the present invention is to provide a policy learning device, a policy learning method, and a communication system that enable efficient learning of an agent in constrained reinforcement learning. [Means for solving the problem]

[0007] A policy learning device in one aspect of an embodiment is a policy learning device that learns an agent that outputs a behavior of a control object in response to an input of a state of the control object by reinforcement learning, and includes a parameter changing unit that changes parameters related to constraint conditions in the reinforcement learning for each of a predetermined number of learning steps in the reinforcement learning, and an agent learning unit that learns the agent so as to satisfy the constraint conditions and by using the parameters as at least a part of the state. Effect of the Invention

[0008] According to one aspect, it becomes possible to efficiently learn an agent in constrained reinforcement learning. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating the configuration of an information processing system 10. As shown in FIG. [Diagram 2] FIG. 2 is a diagram illustrating constrained reinforcement learning. [Diagram 3] FIG. 3 is a graph showing the relationship between reward, cost, and cost threshold. [Figure 4] FIG. 4 is a diagram illustrating the hardware configuration of the information processing device 1. As shown in FIG. [Diagram 5] FIG. 5 is a diagram illustrating functions in the information processing device 1. As shown in FIG. [Figure 6] FIG. 6 is a diagram illustrating information stored in the information storage area 130. As shown in FIG. [Figure 7] FIG. 7 is a diagram illustrating the agent AG stored in the storage device 2. As shown in FIG. [Figure 8]FIG. 8 is a flowchart illustrating an outline of the policy learning process in the first embodiment. [Figure 9] FIG. 9 is a diagram for explaining a specific example of the policy learning process in the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining a specific example of the policy learning process in the first embodiment. [Figure 11] FIG. 11 is a diagram for explaining a specific example of the policy learning process in the first embodiment. [Figure 12] FIG. 12 is a diagram for explaining a specific example of the policy learning process in the first embodiment. [Figure 13] FIG. 13 is a diagram for explaining a specific example of the policy learning process in the first embodiment. [Figure 14] FIG. 14 is a diagram for explaining a specific example of the policy learning process in the first embodiment. [Figure 15] FIG. 15 is a flowchart illustrating details of the policy learning process in the first embodiment. [Figure 16] FIG. 16 is a flowchart illustrating details of the policy learning process in the first embodiment. [Figure 17] FIG. 17 is a flowchart illustrating details of the policy learning process in the first embodiment. [Figure 18] FIG. 18 is a flowchart illustrating details of the policy learning process in the first embodiment. [Figure 19] FIG. 19 is a diagram for explaining another specific example of the policy learning process in the first embodiment. [Figure 20] FIG. 20 is a diagram for explaining another specific example of the policy learning process in the first embodiment. [Figure 21] FIG. 21 is a diagram for explaining details of the policy learning process in the first embodiment. [Figure 22] FIG. 22 is a flowchart illustrating the measure estimation process in the first embodiment. [Diagram 23]FIG. 23 is a diagram illustrating a specific example of the measure estimation process in the first embodiment. [Figure 24] FIG. 24 is a diagram illustrating a specific example of the measure estimation process in the first embodiment. [Diagram 25] FIG. 25 is a diagram for explaining a specific example in which the control target OB is the start-up control of the base station device 11. In FIG. [Figure 26] FIG. 26 is a diagram for explaining a specific example in which the control target OB is the start-up control of the base station device 11. In FIG. [Figure 27] FIG. 27 is a flow chart illustrating the range determination process performed in the example shown in FIG. 25 and the like. [Figure 28] FIG. 28 is a flow chart illustrating the range determination process performed in the example shown in FIG. 25 and the like. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, such descriptions should not be interpreted in a limiting sense and do not limit the subject matter described in the claims. In addition, various changes, substitutions, and modifications can be made without departing from the spirit and scope of the present disclosure. In addition, different embodiments can be appropriately combined.

[0011] [Configuration of wireless communication system in the first embodiment] First, a description will be given of the configuration of the information processing system 10. Fig. 1 is a diagram for explaining the configuration of the information processing system 10. Fig. 2 is a diagram for explaining constrained reinforcement learning.

[0012] As shown in FIG. 1, the information processing system 10 includes, for example, an information processing device 1, a storage device 2, and an operation terminal 5.

[0013] The storage device 2 is, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0014] The operation terminal 5 is, for example, a PC (Personal Computer) through which an operator inputs necessary information to the information processing device 1.

[0015] The information processing device 1 is, for example, one or more physical machines or virtual machines. The information processing device 1 performs a process (hereinafter, also referred to as a policy learning process) of learning an agent AG (hereinafter, also referred to as an agent AG1) that corresponds to various constraint conditions by constrained reinforcement learning. After that, the information processing device 1 stores the agent AG1 after learning in the storage device 2, for example.

[0016] Specifically, as shown in Fig. 2, the information processing device 1 makes the agent AG1 learn a policy for maximizing the reward while observing the constraints by repeatedly learning a policy (hereinafter, also simply referred to as a policy) for determining the action (hereinafter, also simply referred to as an action) of the controlled object OB according to a combination of a state (hereinafter, also simply referred to as a state) from the controlled object OB, a reward (hereinafter, also simply referred to as a reward) from the controlled object OB, and a cost (hereinafter, also simply referred to as a cost) from the controlled object OB. In other words, the information processing device 1 generates the agent AG1 capable of outputting the next action of the controlled object OB by repeatedly inputting and learning, for example, a combination of the state, reward, and cost from the controlled object OB. Hereinafter, the learning of the agent AG1 performed with one input of a combination of the state, reward, and cost from the controlled object OB is also simply referred to as a learning step.

[0017] More specifically, the information processing device 1 changes parameters related to constraint conditions (hereinafter also simply referred to as parameters) for each of a predetermined number of learning steps in the policy learning process, for example. The predetermined number of times may be, for example, once. The parameters related to constraint conditions may be, for example, a cost threshold value included in the constraint conditions (hereinafter also simply referred to as cost threshold value). Then, the information processing device 1 learns the agent AG1 so as to satisfy the constraint conditions and by using the parameters related to the constraint conditions as at least a part of the state in the policy learning process, for example.

[0018] In addition, the information processing device 1 performs a process (hereinafter also referred to as a policy estimation process) of determining (estimating) a new behavior (hereinafter also referred to as simply a new behavior) of the controlled object OB, for example, by using an agent AG1 (agent AG1 generated in the policy learning process) stored in the storage device 2.

[0019] Specifically, the information processing device 1 acquires a new action output from the agent AG1 in response to an input of a new state (hereinafter also simply referred to as a new state) from the control target OB. Then, the information processing device 1 outputs the acquired new action to the control target OB, for example.

[0020] That is, the information processing device 1 in the present embodiment generates an agent AG1 that corresponds to various constraint conditions by, for example, having the agent AG1 learn while changing the constraint conditions. Therefore, the information processing device 1 does not need to generate an agent AG1 for each constraint condition, for example.

[0021] This enables the information processing device 1 in this embodiment to, for example, shorten the time required for learning of agent AG1. Also, the information processing device 1 can, for example, reduce the amount of storage space (for example, storage space in the storage device 2) used by agent AG1 and reduce the cost required for managing agent AG1. Furthermore, even if a measure to deal with new constraints becomes necessary after the generation of agent AG1, for example, the information processing device 1 does not need to generate a new agent AG1.

[0022] In addition, when the control target OB is a so-called cart-pole problem, the state may be, for example, the position of the cart, the acceleration of the cart, the angle of the pole, and the angular acceleration of the pole, the reward may be, for example, a value that changes depending on whether the pole is standing (whether the pole is fallen or not), the cost may be, for example, a value that increases as the cart moves in a predetermined direction, and the action may be, for example, the direction in which the cart is moved (the predetermined direction or the opposite direction to the predetermined direction). Specifically, in the above example, the reward may be "1" when the pole is standing, and "0" when the pole is fallen.

[0023] Here, in the above example, the possibility of the pole falling over decreases with an increase in the movable range of the cart. Therefore, the reward from the controlled object OB is, for example, a value that increases with an increase in the cost from the controlled object OB. The relationship between the reward, the cost, and the cost threshold will be described below.

[0024] [Relationship between reward, cost, and cost threshold] Fig. 3 is a graph showing the relationship between reward, cost, and cost threshold. The vertical axis in the graph shown in Fig. 3 corresponds to the cost and reward, respectively, and the horizontal axis in the graph shown in Fig. 3 corresponds to the cost threshold. Moreover, the solid line in Fig. 3 corresponds to a function indicating the cost (hereinafter also referred to as a cost function), and the dashed line in Fig. 3 corresponds to a function indicating the reward (hereinafter also referred to as a reward function). Note that Fig. 3 is a graph showing the relationship between reward, cost, and cost threshold when the control target OB is a cart pole problem.

[0025] Specifically, in the graph shown in FIG. 3, for example, when the cost threshold is equal to or less than 0, the cost and the reward are both 0.

[0026] 3, for example, the cost when the cost threshold is 10 is 10, and exceeds the cost when the cost threshold is 0. Also, in the graph shown in FIG. 3, for example, the reward when the cost threshold is 10 is more than 10, and exceeds the reward when the cost threshold is 0.

[0027] 3, for example, the cost when the cost threshold is 20 is 20, and exceeds the cost when the cost threshold is 10. Also, in the graph shown in FIG. 3, for example, the reward when the cost threshold is 20 is more than 20, and exceeds the reward when the cost threshold is 10.

[0028] 3, for example, the cost is 20 when the cost threshold is 30, and is the same as the cost when the cost threshold is 20. Also, in the graph shown in FIG. 3, for example, the reward when the cost threshold is 30 is greater than 20, and is the same as the reward when the cost threshold is 20.

[0029] That is, the graph shown in FIG. 3 indicates that, for example, when the cost threshold is strict (when the cost threshold is equal to or less than 0), it becomes difficult to learn a policy that satisfies the constraint conditions.

[0030] The graph in FIG. 3 also shows that for cost thresholds between 0 and 20, for example, the reward increases as the cost increases.

[0031] In addition, the graph shown in Fig. 3 indicates that, for example, when the cost threshold is 20 or more, a policy that obtains the maximum reward at the smallest possible cost is learned, so that the cost and reward do not become large. Below, we will explain using the relationship between reward, cost, and cost threshold shown in Fig. 3.

[0032] [Hardware configuration of information processing system] Next, a description will be given of the hardware configuration of the information processing system 10. FIG.

[0033] 4, the information processing device 1 includes, for example, a CPU (Central Processing Unit) 101, which is a processor, a memory 102, a communication device (I / O interface) 103, and a storage device 104. Each unit is connected to each other via a bus 105.

[0034] The storage device 104 has, for example, a program storage area (not shown) that stores a program 110 for performing a policy learning process and a policy estimation process (hereinafter, these are also collectively referred to as a policy learning process, etc.). The storage device 104 also has, for example, a storage unit 130 (hereinafter, also referred to as an information storage area 130) that stores information used when performing the policy learning process, etc. The storage device 104 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0035] The CPU 101 executes, for example, a program 110 loaded from the storage device 104 to the memory 102 to perform policy learning processing.

[0036] Furthermore, the communication device 103 communicates with the operation terminal 5 via a network (not shown) such as the Internet.

[0037] [Functions in information processing devices] Next, functions of the information processing device 1 will be described. Fig. 5 is a diagram for explaining the functions of the information processing device 1. Fig. 6 is a diagram for explaining information stored in the information storage area 130. Fig. 7 is a diagram for explaining an agent AG stored in the storage device 2.

[0038] As shown in FIG. 5, the information processing device 1 realizes various functions including a first learning unit 111 (hereinafter also referred to as an agent learning unit 111), a parameter changing unit 112, a cost calculation unit 113, a second learning unit 114, and a range determination unit 115 as the function of policy learning processing, by organic cooperation between hardware such as a CPU 101 and a memory 102 and a program 110.

[0039] As shown in FIG. 5 , the information processing device 1 realizes various functions including a measure estimation unit 116 and a measure output unit 117 as functions of the measure estimation process by organic cooperation between hardware such as a CPU 101 and a memory 102 and a program 110.

[0040] 6, for example, a cost calculation formula 131 and a constraint condition 132 are stored in the information storage area 130. Note that, although a case where the cost calculation formula 131 and the constraint condition 132 are stored in the information storage area 130 will be described below, the cost calculation formula 131 and the constraint condition 132 may be stored in the storage device 2, for example.

[0041] First, the functions in the policy learning process will be explained.

[0042] The first learning unit 111 generates an agent AG1 capable of outputting an action (next action) of the controlled object OB in response to inputs of a state, reward, and cost from the controlled object OB. Then, the first learning unit 111 stores the generated agent AG1 in the storage device 2, for example, as shown in Fig. 7. Note that the first learning unit 111 may store the agent AG1 in the information storage area 130, for example.

[0043] Specifically, the first learning unit 111, for example, inputs a combination of a state, a reward, and a cost acquired from the control target OB to the agent AG1, thereby making the agent AG1 execute a learning step. In this case, the agent AG1 executes the learning step so as to satisfy the constraint condition 132 stored in the information storage area 130, for example, by using the combination of the state, the reward, and the cost input by the first learning unit 111. The constraint condition 132 may be, for example, a predetermined condition that is stored in the information storage area 130 in advance by an operator. Furthermore, for example, when making the agent AG1 execute the learning step, the first learning unit 111 inputs a parameter related to the constraint condition 132 as at least a part of the state. Thereafter, the first learning unit 111 generates the agent AG1 by repeatedly executing the learning step until, for example, the operator inputs information to the information processing device 1 indicating that the learning is to be terminated.

[0044] The parameter modification unit 112 modifies a parameter related to the constraint condition 132 stored in the information storage area 130, for example, every time the number of times the learning step is executed by the first learning unit 111 reaches a predetermined number of times. The parameter related to the constraint condition 132 may be, for example, a cost threshold value included in the constraint condition 132.

[0045] Specifically, in this case, the parameter change unit 112 randomly changes, for example, a parameter related to the constraint condition 132. Note that, although a case where the cost threshold included in the constraint condition 132 is an upper limit threshold for the cost will be described below, the cost threshold included in the constraint condition 132 may be, for example, a lower limit threshold for the cost.

[0046] The cost calculation unit 113 calculates the cost by using, for example, a cost calculation formula 131 stored in the information storage area 130. The cost calculation formula 131 may be, for example, a predetermined formula that is stored in the information storage area 130 in advance by an operator.

[0047] Specifically, the cost calculation unit 113 calculates the cost by, for example, substituting data from the control target OB (data used to calculate the cost) into the cost calculation formula 131. That is, the first learning unit 111 uses the cost calculated by the cost calculation unit 113 as one of the inputs to the agent AG1 when causing the agent AG1 to execute a learning step, for example.

[0048] The parameters related to the constraints 132 (the parameters changed by the parameter change unit 112) may be parameters such as coefficients included in the cost calculation formula 131 stored in the information storage area 130, for example.

[0049] The second learning unit 114 generates an agent AG (hereinafter also referred to as agent AG2) capable of outputting an action (next action) of the controlled object OB in response to an input of a combination of a state, a reward, and a cost from the controlled object OB. Then, the second learning unit 114 stores the generated agent AG2 in the storage device 2, for example, as shown in Fig. 7. Note that the second learning unit 114 may store the agent AG2 in the information storage area 130, for example.

[0050] Specifically, the second learning unit 114 may generate an agent AG2 by using, for example, the cost calculation formula 131 and the constraint condition 132 stored in the information storage area 130, similar to the generation of the agent AG1. Meanwhile, the parameter change unit 112 may generate an agent AG2 without changing, for example, the parameters related to the constraint condition 132, unlike the generation of the agent AG1.

[0051] The range determination unit 115 determines the range of parameter change (hereinafter also referred to as the specified change range) by the parameter change unit 112, for example, depending on other costs (hereinafter also referred to as other costs) incurred when the controlled object OB performs other actions (hereinafter also referred to as other actions) output from agent AG2 in response to input of other states (hereinafter also referred to as other states) from the controlled object OB.

[0052] That is, the parameter modification unit 112 modifies the parameters related to the constraint condition 132 within the modification range determined by the range determination unit 115, for example.

[0053] Next, the functions in the policy estimation process will be described.

[0054] The measure estimation unit 116, for example, inputs a new state from the control object OB to the agent AG1. Then, the measure estimation unit 116 acquires, for example, a new action (estimated result) of the control object OB output from the agent AG1.

[0055] The measure output unit 117, for example, outputs the new action acquired by the measure estimation unit 116 to the control target OB, thereby causing the control target OB to perform the new action.

[0056] [Outline of policy learning process in the first embodiment] Next, an outline of the policy learning process in the first embodiment will be described below. Fig. 8 is a flow chart for explaining an outline of the policy learning process in the first embodiment.

[0057] 8, the first learning unit 111 waits until the first learning start timing comes (NO in S1). The first learning start timing may be, for example, the timing when the operator inputs information to the information storage area 130 to start learning of the agent AG1.

[0058] Then, when the first learning start timing arrives (YES in S1), the first learning unit 111 causes the agent AG1 to execute a learning step, for example, with the parameters related to the constraint conditions 132 stored in the information storage area 130 as at least one of the states (S2).

[0059] Specifically, the first learning unit 111 causes the agent AG1 to execute a learning step, for example, with the cost threshold included in the constraint condition 132 as at least one of the states.

[0060] That is, in this case, the first learning unit 111 inputs, for example, the state, reward, and cost from the control object OB, as well as the cost threshold included in the constraint condition 132, to the agent AG1 as one of the states.

[0061] Next, the first learning unit 111 determines whether or not a first learning end timing has been reached (S3). The first learning end timing may be, for example, the timing when the operator inputs information to end the learning of the agent AG1 to the information processing device 1. The first learning end timing may be, for example, the timing when the number of times the learning step has been executed (the number of times the process of S2 has been executed) has reached a predetermined number of times.

[0062] As a result, when it is determined that the first learning end timing has been reached (YES in S3), the first learning unit 111 ends the policy learning process, for example.

[0063] On the other hand, if it is determined that the first learning end timing has not been reached (NO in S3), the parameter change unit 112 determines, for example, whether the number of times the learning step has been executed has reached a predetermined number (S4).

[0064] Specifically, the parameter change unit 112 determines, for example, whether the number of times the learning step has been executed since the execution of the policy learning process has started, or the number of times the learning step has been executed since the process of S5 described below was last executed, has reached a predetermined number.

[0065] As a result, when it is determined that the number of times the learning step has been executed reaches a predetermined number of times (YES in S4), the parameter change unit 112 changes, for example, parameters related to the constraint conditions 132 stored in the information storage area 130 (S5).

[0066] Specifically, in this case, the parameter modification unit 112 randomly modifies the cost threshold included in the constraint condition 132, for example.

[0067] Then, for example, after the process of S5 is performed, the first learning unit 111 performs the processes from S2 onward again. Similarly, the first learning unit 111 performs the processes from S2 onward again even when it is determined that the number of times the learning step has been executed has not reached a predetermined number of times (NO in S4).

[0068] That is, the information processing device 1 in the present embodiment generates an agent AG1 corresponding to various constraint conditions 132 by, for example, having the agent AG1 learn while changing the constraint conditions 132. Therefore, the information processing device 1 does not need to generate an agent AG1 for each constraint condition 132, for example.

[0069] This enables the information processing device 1 in this embodiment to efficiently perform learning of the agent AG1 corresponding to various constraint conditions 132, for example.

[0070] [Specific example (1) of policy learning process in the first embodiment] Next, a specific example of the policy learning process in the first embodiment will be described. Figures 9 to 14 are diagrams for explaining a specific example of the policy learning process in the first embodiment. Below, a case where the parameter related to the constraint condition 132 is a cost threshold value included in the constraint condition 132 will be described.

[0071] As shown in FIG. 9, the cost calculation unit 113 calculates the cost by, for example, substituting data from the control target OB (data used for calculating the cost) into a cost calculation formula 131 stored in the information storage area .

[0072] Specifically, the cost calculation formula 131 shown in Fig. 10 is, for example, a formula indicating that data (x) from the control target OB is set as the cost. In other words, the cost calculation formula 131 shown in Fig. 10 is, for example, a formula indicating that a coefficient to be multiplied by the data (x) from the control target OB is 1. Therefore, for example, when the cost calculation formula 131 shown in Fig. 10 is used, the cost calculation unit 113 calculates the data from the control target OB as it is as the cost.

[0073] Then, the first learning unit 111 adds, for example, the cost threshold value included in the constraint condition 132 stored in the information storage area 130 to the state from the control object OB.

[0074] Furthermore, the first learning unit 111 inputs, for example, a state in which the cost threshold value is added, a reward from the control target OB, and a cost calculated by the cost calculation unit 113 to the agent AG1. In this case, the agent AG1 executes a learning step so as to satisfy the constraint condition 132 stored in the information storage area 130, for example, by using the state input by the first learning unit 111, etc.

[0075] Specifically, the constraint condition 132 shown in FIG. 11 is, for example, a condition indicating that the data (x) from the control target OB is 10 or less. In other words, the constraint condition 132 shown in FIG. 11 is, for example, a condition indicating that the cost threshold is 10. Therefore, for example, when the constraint condition 132 shown in FIG. 11 is used, the agent AG1 learns, for example, in the learning step, a measure such that the data from the control target OB does not exceed 10. Note that the constraint condition 132 in this case may be, for example, a condition indicating that the cost calculated by the cost calculation unit 113 is 10 (cost threshold) or less.

[0076] After that, the first learning unit 111 outputs, for example, to the control object OB, the action (next action) of the control object OB determined by the agent AG1 after the learning step has been performed.

[0077] Furthermore, the parameter change unit 112 randomly changes the cost threshold included in the constraint condition 132 stored in the information storage area 130, for example, every time the number of times the learning step is executed in the agent AG1 reaches a predetermined number of times.

[0078] 12, the parameter modification unit 112 modifies the constraint condition 132 stored in the information storage area 130 so as to indicate that the data (x) from the control target OB is equal to or less than 5. In other words, in this case, the parameter modification unit 112 modifies the cost threshold to 5, for example.

[0079] 13, the first learning unit 111 may add, for example, a coefficient or the like included in the cost calculation formula 131 stored in the information storage area 130 to the state from the control target OB. That is, the first learning unit 111 may use, for example, a coefficient or the like included in the cost calculation formula 131 stored in the information storage area 130 as a parameter related to the constraint condition 132 stored in the information storage area 130.

[0080] In this case, the parameter modification unit 112 may, for example, randomly modify the coefficients and the like included in the cost calculation formula 131 stored in the information storage area .

[0081] 14, the parameter modification unit 112 may modify the cost calculation formula 131 stored in the information storage area 130 so as to indicate that the cost is a value obtained by multiplying the data (x) from the control target OB by 2. In other words, in this case, the parameter modification unit 112 may modify the coefficient by which the data (x) from the control target OB is multiplied to 2.

[0082] [Details of policy learning process in the first embodiment] Next, details of the policy learning process in the first embodiment will be described. Figures 15 to 18 are flow charts for explaining the details of the policy learning process in the first embodiment. Figures 19 to 21 are diagrams for explaining the details of the policy learning process in the first embodiment.

[0083] [Second learning process] First, a process (hereinafter also referred to as a second learning process) of generating an agent AG2 used to determine a change range of a parameter related to the constraint condition 132 will be described in the policy learning process. Fig. 15 is a flowchart for explaining the second learning process.

[0084] 15, the second learning unit 114 waits until the second learning start timing comes (NO in S11). The second learning start timing may be, for example, the timing when the operator inputs information to the information storage area 130 to start learning of the agent AG2.

[0085] Then, when the second learning start timing arrives (YES in S11), the second learning unit 114 executes, for example, a learning step (S12).

[0086] Specifically, the second learning unit 114 performs learning of the agent AG2 by inputting, for example, a combination of a state, a reward, and a cost from the control object OB.

[0087] Next, the second learning unit 114 determines whether or not a second learning end timing has been reached (S13). The second learning end timing may be, for example, the timing when the operator inputs information to end the learning of the agent AG2 to the information processing device 1. In addition, the second learning end timing may be, for example, the timing when the number of times the learning step has been executed (the number of times the process of S12 has been executed) has reached a predetermined number of times.

[0088] As a result, when it is determined that the second learning end timing has not been reached (NO in S13), the second learning unit 114 performs, for example, the processes from S12 onwards again.

[0089] On the other hand, if it is determined that the second learning end timing has been reached (YES in S13), the second learning unit 114 ends the second learning process, for example.

[0090] [First range determination process] Next, a process of determining a parameter change range using the agent AG2 (hereinafter, also referred to as a range determination process or a first range determination process) in the policy learning process will be described. Fig. 16 is a flow chart for explaining the first range determination process.

[0091] 16, the range determination unit 115 waits until the range determination timing arrives (NO in S21). The range determination timing may be, for example, the timing when the operator inputs information to the information storage area 130 to determine the parameter change range.

[0092] Then, when the time for determining the range arrives (YES in S21), the range determining unit 115 determines the change range of the parameters related to the constraint condition 132 from, for example, the learning results of the agent AG2 stored in the storage device 2 (S22).

[0093] Specifically, in this case, the range determination unit 115 acquires, for example, a new action of the control target OB output from the agent AG2 in association with the input of a new state from the control target OB. In other words, the range determination unit 115 acquires the new action of the control target OB, for example, by performing a policy estimation process using the agent AG2. Then, the range determination unit 115 calculates a new cost from the control target OB (hereinafter also simply referred to as a new cost) by, for example, having the control target OB perform the acquired new action. After that, the range determination unit 115 determines a parameter change range according to, for example, the calculated new cost.

[0094] That is, for example, if the upper limit of the cost threshold included in the constraint conditions 132 stored in the information storage area 130 (the upper limit of the cost threshold included in the constraint conditions 132 at the time of generating agent AG2) is 30, and the new maximum cost is smaller than 30, the range determination unit 115 determines that it is possible to reduce the upper limit of the change range of the cost threshold when generating agent AG1. Specifically, for example, if the calculated new maximum cost is 20, the range determination unit 115 sets the upper limit of the change range of the cost threshold when generating agent AG1 to 20.

[0095] Furthermore, for example, if the lower limit of the cost threshold included in the constraint conditions 132 stored in the information storage area 130 (the lower limit of the cost threshold included in the constraint conditions 132 at the time of generating agent AG2) is 10, and the new minimum cost is greater than 10, the range determination unit 115 determines that it is possible to increase the lower limit of the change range of the cost threshold when generating agent AG1. Specifically, for example, if the calculated new minimum cost is 20, the range determination unit 115 sets the lower limit of the change range of the cost threshold when generating agent AG1 to 20.

[0096] This makes it possible for the information processing device 1 in this embodiment to further reduce the time required to generate the agent AG1, for example.

[0097] Note that the range determination unit 115 may perform another range determination process (hereinafter also referred to as range determination process without using the agent AG2) that determines the change range of the cost threshold without using the agent AG2, for example.

[0098] Specifically, for example, if the cost threshold included in the constraint condition 132 stored in the information storage area 130 is 30, and the maximum cost when all or part of the actions that can be performed by the controlled object OB are repeated is 20, the range determination unit 115 may set the upper limit of the change range of the cost threshold when generating agent AG1 to 20.

[0099] Also, for example, if the cost threshold included in the constraint condition 132 stored in the information storage area 130 is 30, and the maximum cost when an action that may not satisfy the constraint condition 132 is repeated is 20, the range determination unit 115 may set the upper limit of the change range of the cost threshold when generating agent AG1 to 20.

[0100] Also, for example, if the cost threshold included in the constraint condition 132 stored in the information storage area 130 is 30, and the maximum cost when an action that is likely to satisfy the constraint condition 132 is repeated is 20, the range determination unit 115 may set the upper limit of the change range of the cost threshold when generating agent AG1 to 20.

[0101] Furthermore, for example, when there is a constraint on the operation of the control target OB, the range determination unit 115 may determine the change range of the cost threshold so as to satisfy the constraint on the operation.

[0102] [First learning process] Next, the process of generating the agent AG1 (hereinafter, also referred to as the first learning process) in the policy learning process will be described. Fig. 17 is a flow chart for explaining the first learning process.

[0103] As shown in FIG. 17, the first learning unit 111 waits until the first learning start timing, for example (NO in S31).

[0104] Then, when the learning start timing arrives (YES in S31), the first learning unit 111 executes a learning step, for example, with a parameter related to the constraint condition 132 stored in the information storage area 130 as at least one of the states (S32).

[0105] Specifically, in this case, the first learning unit 111 executes the learning step by inputting, for example, the cost threshold included in the constraint condition 132 to the agent AG1 as at least one of the states.

[0106] Next, the first learning unit 111 determines, for example, whether or not the first learning end timing has been reached (S33).

[0107] As a result, when it is determined that the first learning end timing has been reached (YES in S33), the first learning unit 111 ends the first learning process, for example.

[0108] On the other hand, if it is determined that the learning end timing has not been reached (NO in S33), the parameter change unit 112 determines, for example, whether the number of times the learning step has been executed has reached a predetermined number (S34).

[0109] Specifically, the parameter change unit 112 determines, for example, whether the number of times the learning step has been executed since the execution of the first learning process has started, or the number of times the learning step has been executed since the processing of S35 described below was last performed, has reached a predetermined number.

[0110] As a result, if it is determined that the number of times the learning step has been executed has reached a predetermined number (YES in S34), the parameter modification unit 112 modifies, for example, parameters related to the constraint conditions 132 stored in the information storage area 130 within the parameter modification range determined in the processing of S24 (S35).

[0111] Specifically, in this case, the parameter modification unit 112 randomly modifies, for example, the parameters related to the constraint condition 132 within the parameter modification range determined in the process of S24.

[0112] Then, for example, after the process of S35 is performed, the first learning unit 111 performs the processes from S32 onward again. Similarly, the first learning unit 111 also performs the processes from S32 onward again when, for example, it is determined that the number of times the learning step has been executed has not reached a predetermined number of times (NO in S34).

[0113] That is, the information processing device 1 in this embodiment changes, for example, the parameters related to the constraint condition 132 stored in the information storage area 130 within the parameter change range determined in the first range determination process.

[0114] This enables the information processing device 1 in this embodiment to perform learning of the agent AG1 corresponding to various constraint conditions 132 more efficiently, for example.

[0115] [Specific example (2) of policy learning process in the first embodiment] Next, another specific example of the policy learning process in the first embodiment will be described. Figures 19 and 20 are diagrams for explaining another specific example of the policy learning process in the first embodiment. Below, differences from Figures 9 and 13 will be described.

[0116] The range determination unit 115 determines the change range of the cost threshold included in the constraint condition 132 stored in the information storage area 130, for example, by using the agent AG2 generated by the second learning unit 114, as shown in Figure 19.

[0117] Then, the parameter modification unit 112 modifies the cost threshold included in the constraint condition 132 stored in the information storage area 130 within the cost threshold modification range determined by the range determination unit 115, for example, each time the number of times the learning step is executed reaches a predetermined number.

[0118] In addition, the range determination unit 115 may determine the change range of coefficients, etc. included in the cost calculation formula 131 stored in the information storage area 130, for example, by using an agent AG2 generated by the second learning unit 114, as shown in Figure 20.

[0119] The parameter modification unit 112 may modify the coefficients, etc. included in the cost calculation formula 131 stored in the information storage area 130 within the modification range of the coefficients, etc. determined by the range determination unit 115, for example, each time the number of times the learning step is executed reaches a predetermined number of times.

[0120] [Second range determination process] Next, another range determination process (hereinafter also referred to as second range determination process) will be described. Fig. 18 is a flow chart diagram for explaining the second range determination process. Fig. 21 is a diagram for explaining the second range determination process. Note that the second range determination process is, for example, a process performed in place of the first range determination process. Also, the second range determination process includes, for example, a second learning process, as described later.

[0121] First, the process of S41, the process of S42, and the process from S43 to S45 that is performed for the first time will be described.

[0122] As shown in FIG. 18, range determination section 115 waits until the range determination timing, for example (NO in S41).

[0123] Then, when it is time to determine the range (YES in S41), the range determination section 115 determines, for example, parameters (initial values) related to the constraint conditions 132 stored in the information storage area 130 (S42).

[0124] 21, the range determination unit 115 determines, for example, "0", which is the minimum value in the range of values ​​selectable as the cost threshold included in the constraint condition 132, as the parameter (initial value) related to the constraint condition 132. Note that the range of values ​​selectable as the cost threshold included in the constraint condition 132 may be determined in advance by, for example, an operator.

[0125] Next, the second learning unit 114 executes, for example, a second learning process to generate an agent AG2 using the constraint condition 132 corresponding to the parameters determined in the process of S42 (S43).

[0126] After that, the range determination unit 115 determines parameters (next parameters) related to the constraint conditions 132 from the learning results of the agent AG2 generated in the immediately preceding process of S43 (S44).

[0127] Specifically, as shown in FIG. 21, for example, in the first processing of S44, the range determination unit 115 determines “50”, which is the maximum value in the range of values ​​that can be selected as the cost threshold included in the constraint condition 132, as the parameter for the constraint condition 132.

[0128] Thereafter, the range determination unit 115 determines whether or not a termination condition of the second range determination process (hereinafter also simply referred to as the termination condition) has been met (S45). The termination condition may be, for example, that the difference between the value indicated by the parameter determined in the current process of S44 and the value indicated by the parameter determined in the previous process of S44 is equal to or smaller than a predetermined value.

[0129] As a result, when it is determined that the end condition has not been met (NO in S45), the range determination unit 115 performs, for example, the processes from S43 onwards again.

[0130] Next, the process from S43 to S45 that is performed for the second time will be described.

[0131] The second learning unit 114 regenerates an agent AG2 using, for example, the constraint condition 132 corresponding to the parameters determined in the immediately preceding process of S44 (S43).

[0132] That is, in the second range determination process, unlike the first range determination process, for example, the parameter change range is specified by using each of the multiple agents AG2.

[0133] Then, the range determination unit 115 determines parameters (next parameters) related to the constraint conditions 132 from the learning results of the agent AG2 generated in the immediately preceding process of S43 (S44).

[0134] Specifically, as shown in FIG. 21, the range determination unit 115 determines the parameter related to the constraint condition 132 to be "25", which is the average value of "0", the value indicated by the parameter determined in the processing of S42, and "50", the value indicated by the parameter determined in the first processing of S44.

[0135] Thereafter, the range determination unit 115 determines, for example, whether or not a termination condition has been achieved (S45).

[0136] As a result, when it is determined that the end condition has not been met (NO in S45), the range determination unit 115 performs, for example, the processes from S43 onwards again.

[0137] Next, the process from S43 to S45 that is performed for the third time will be described.

[0138] The second learning unit 114 regenerates an agent AG2 using, for example, the constraint condition 132 corresponding to the parameters determined in the immediately preceding process of S44 (S43).

[0139] Then, the range determination unit 115 determines parameters (next parameters) related to the constraint conditions 132 from the learning results of the agent AG2 generated in the immediately preceding process of S43 (S44).

[0140] Specifically, the range determination unit 115 acquires a new action of the control target OB output from the agent AG2 (the agent AG2 generated in the immediately preceding process of S43) in response to the input of a new state from the control target OB, for example. In other words, the range determination unit 115 acquires a new action of the control target OB, for example, by performing a policy estimation process using the agent AG2 generated in the immediately preceding process of S43. Then, the range determination unit 115 acquires a new reward (hereinafter also simply referred to as a new reward) from the control target OB, for example, by having the control target OB perform the acquired new action. After that, the range determination unit 115 determines a parameter (next parameter) from each of the new rewards acquired in the process of S44 performed up to this time, for example.

[0141] More specifically, as shown in FIG. 21, for example, when the reward obtained in the second processing of S44 and the reward obtained in the third processing of S44 are the same, the range determination unit 115 determines that there is a value that is greater than the value indicated by the parameter determined in the processing of S42 and smaller than the value indicated by the parameter determined in the second processing of S44, and that can be used to obtain the same reward as the reward obtained in the third processing of S44.

[0142] Therefore, in this case, the range determination unit 115 determines, for example, a value greater than "0", which is the value indicated by the parameter determined in the process of S42, and less than "25", which is the value indicated by the parameter determined in the process of S44 for the second time, as the parameter (next parameter) related to the constraint condition 132. Specifically, in this case, the range determination unit 115 determines, for example, "12.5", which is the average value of "0", which is the value indicated by the parameter determined in the process of S42, and "25", which is the value indicated by the parameter determined in the process of S44 for the second time, as the parameter related to the constraint condition 132.

[0143] As a result, when it is determined that the end condition has not been met (NO in S45), the range determination unit 115 performs, for example, the processes from S43 onwards again.

[0144] Next, the processes from S43 to S45 performed for the fourth time will be described.

[0145] The second learning unit 114 regenerates an agent AG2 using, for example, the constraint condition 132 corresponding to the parameters determined in the immediately preceding process of S44 (S43).

[0146] Then, the range determination unit 115 determines parameters (next parameters) related to the constraint conditions 132 from the learning results of the agent AG2 generated in the immediately preceding process of S43 (S44).

[0147] Specifically, as shown in FIG. 21, for example, when the reward obtained in the fourth processing of S44 is smaller than the reward obtained in the third processing of S44, the range determination unit 115 determines that there is a value that is greater than the value indicated by the parameter determined in the third processing of S44 and smaller than the value indicated by the parameter determined in the second processing of S44, and that can obtain the same reward as the reward obtained in the third processing of S44.

[0148] Therefore, in this case, the range determination unit 115 determines, for example, a value larger than "12.5" which is the value indicated by the parameter determined in the third process of S44 and smaller than "25" which is the value indicated by the parameter determined in the second process of S44, as the parameter (next parameter) related to the constraint condition 132. Specifically, in this case, the range determination unit 115 determines, for example, "18.75" which is the average value of "25" which is the value indicated by the parameter determined in the second process of S44 and "12.5" which is the value indicated by the parameter determined in the third process of S44, as the parameter related to the constraint condition 132.

[0149] Thereafter, the range determination unit 115 determines, for example, whether or not a termination condition has been achieved (S45).

[0150] As a result, when it is determined that the end condition has not been met (NO in S45), the range determination unit 115 performs, for example, the processes from S43 onwards again.

[0151] In addition, for example, in the fifth or subsequent processing of S44, if the reward obtained in the previous processing of S44 is the same as the reward obtained in the current processing of S44, the range determination unit 115 may perform processing similar to the third processing of S44, and if the reward obtained in the current processing of S44 is smaller than the reward obtained in the previous processing of S44, the range determination unit 115 may perform processing similar to the fourth processing of S44.

[0152] On the other hand, if it is determined that the end condition is met (YES in S45), the range determination unit 115 determines, for example, the change range of the parameter related to the constraint condition 132 (S46).

[0153] Specifically, the range determination unit 115, for example, identifies the value of the parameter when the reward acquired in the process of S44 is the maximum value. Then, the range determination unit 115 determines, for example, the minimum value of the identified values ​​(a value close to "20" in the example shown in FIG. 28) as the upper limit of the change range of the parameter related to the constraint condition 132.

[0154] That is, the range determination unit 115, for example, by repeatedly performing the process of S44 until the end condition is achieved, gradually narrows the range including the parameter with the smallest new cost from the control target OB among the parameters that maximize the new reward from the control target OB. Then, the range determination unit 115 determines the change range of the parameter related to the constraint condition 132 by using the range narrowed until the end condition is achieved, for example.

[0155] This enables the information processing device 1 to specify the change range of the parameters related to the constraint condition 132, regardless of the complexity of the cost function, for example.

[0156] In the above example, the case where the parameter related to the constraint condition 132 is determined by comparing the reward acquired in the current process of S44 with the reward acquired in the previous process of S44 has been described, but the present invention is not limited to this. Specifically, the range determination unit 115 may acquire a new cost from the control target OB by, for example, making the control target OB perform a new action in the process of S44. Then, for example, when the new cost satisfies the constraint condition 132, the range determination unit 115 may determine a value smaller than the value indicated by the parameter determined in the previous process of S44 as the parameter related to the constraint condition 132 (next parameter). Also, for example, when the new cost does not satisfy the constraint condition 132, the range determination unit 115 may determine a value larger than the value indicated by the parameter determined in the previous process of S44 as the parameter related to the constraint condition 132 (next parameter).

[0157] In the above example, the upper limit of the change range of the parameter related to the constraint condition 132 is determined, but the present invention is not limited to this. Specifically, the range determination unit 115 may determine the lower limit of the change range of the parameter related to the constraint condition 132, for example.

[0158] [Measure Estimation Processing in the First Embodiment] Next, a description will be given of the measure estimation process in the first embodiment. Fig. 22 is a flow chart illustrating the measure estimation process in the first embodiment.

[0159] 22, the measure estimation unit 116 waits until the estimation start timing comes (NO in S101). The estimation start timing may be, for example, the timing when the worker inputs information to the effect that the estimation of measures is to start, into the information storage area 130.

[0160] Then, when the estimation start timing arrives (YES in S101), the measure estimation unit 116 acquires, for example, a new action of the control target OB output from the agent AG1 in conjunction with the input of a new state from the control target OB (S102).

[0161] After that, the measure output unit 117 outputs, for example, the new action acquired in the process of S102 to the control target OB (S103).

[0162] [Specific example of the strategy estimation process in the first embodiment] Next, a specific example of the measure estimation process in the first embodiment will be described. Figures 23 and 24 are diagrams for explaining a specific example of the measure estimation process in the first embodiment.

[0163] As shown in FIG. 23, for example, the measure estimation unit 116 adds a cost threshold value included in the constraint condition 132 stored in the information storage area 130 to a new state from the control object OB.

[0164] Then, the measure estimation unit 116 outputs, to the control object OB, a new action of the control object OB output from the agent AG1 in response to the input of a new state to which a cost threshold has been added, for example.

[0165] As shown in FIG. 24, the measure estimation unit 116 may add coefficients and the like included in the cost calculation formula 131 stored in the information storage area 130 to a new state from the control target OB, for example.

[0166] The measure estimation unit 116 may output, to the control target OB, a new action of the control target OB output from the agent AG1 in response to the input of a new state to which a coefficient or the like has been added.

[0167] In this way, the information processing device 1 in this embodiment changes the parameters related to the constraint conditions 132 for each predetermined number of learning steps in the policy learning process. Then, the information processing device 1 learns the agent AG1, for example, so as to satisfy the constraint conditions 132 and by using the parameters related to the constraint conditions 132 as at least a part of the state.

[0168] Furthermore, the information processing device 1 in this embodiment acquires, for example, a new behavior of the control target OB output from the agent AG1 in response to an input of a new state from the control target OB. Then, the information processing device 1 outputs, for example, the acquired new behavior to the control target OB.

[0169] That is, the information processing device 1 in the present embodiment can generate an agent AG1 corresponding to various constraint conditions 132 by, for example, having the agent AG1 learn while changing the constraint conditions 132. Therefore, the information processing device 1 does not need to generate an agent AG1 for each constraint condition 132, for example.

[0170] This enables the information processing device 1 in this embodiment to, for example, shorten the time required for learning of agent AG1. Also, the information processing device 1 can, for example, reduce the usage of memory area (for example, memory area in the storage device 2) of agent AG1 and reduce the cost required for managing agent AG1. Furthermore, even if a measure corresponding to a new constraint condition 132 becomes necessary after the generation of agent AG1, the information processing device 1 does not need to generate a new agent AG1.

[0171] In addition, the policy learning process and the like in the present embodiment may be, for example, a case where the control target OB is used for the startup control of the base station device. The following describes the case where the control target OB is used for the startup control of the base station device.

[0172] [Specific example when the control target is the startup control of base station equipment] 25 and 26 are diagrams for explaining a specific example in which the control target OB is the start-up control of the base station device 11. FIG.

[0173] In the example shown in FIG. 25 etc., the control target OB includes, for example, a base station device 11a which is an MBS (Macro Base Station), a base station device 11b which is an SBS (Small Base Station), and a base station device 11c which is an SBS. In the example shown in FIG. 25 etc., at least one of the base station device 11a, the base station device 11b, and the base station device 11c performs wireless communication with, for example, a terminal device 12a, a terminal device 12b, a terminal device 12c, a terminal device 12d, a terminal device 12e, and a terminal device 12f. Hereinafter, the base station device 11a, the base station device 11b, and the base station device 11c are collectively referred to simply as the base station device 11. Hereinafter, the terminal device 12a, the terminal device 12b, the terminal device 12c, the terminal device 12d, the terminal device 12e, and the terminal device 12f are collectively referred to simply as the terminal device 12.

[0174] Specifically, in the example shown in Fig. 25, for example, the base station device 11b and the base station device 11c are stopped. Therefore, in the example shown in Fig. 25, each of the terminal device 12a, the terminal device 12b, the terminal device 12c, the terminal device 12d, the terminal device 12e, and the terminal device 12f is performing wireless communication with the base station device 11a.

[0175] On the other hand, in the example shown in Fig. 26, for example, the base station device 11b and the base station device 11c are activated. Therefore, in the example shown in Fig. 26, unlike the example shown in Fig. 25, the terminal device 12c is performing wireless communication with the base station device 11b, and each of the terminal devices 12d and 12e is performing wireless communication with the base station device 11c.

[0176] Therefore, in the example shown in Fig. 25, the load on base station device 11a is larger while the overall power consumption of base station device 11 is suppressed compared to, for example, the example shown in Fig. 26. In other words, in the example shown in Fig. 26, the load on base station device 11a is smaller while the overall power consumption of base station device 11 is increased compared to, for example, the example shown in Fig. 25.

[0177] Here, the maximum loads (hereinafter also referred to as maximum loads) in each of the base station devices 11a, 11b, and 11c vary depending on, for example, the traffic volume between each of the terminal devices 12. The load thresholds (hereinafter also referred to as load thresholds) in each of the base station devices 11a, 11b, and 11c are parameters related to constraint conditions.

[0178] Therefore, the information processing device 1 in this embodiment determines a policy for each load threshold by using, for example, an agent AG1 generated in the policy learning process.

[0179] Specifically, in this case, the information processing device 1 in this embodiment learns the agent AG1 by, for example, setting the state to the predicted traffic volume for the next 30 minutes and the load of each base station device 11, the reward to exp (-the total power consumption in all base station devices 11 for the last 30 minutes), the cost to a value calculated by cost calculation formula 131 shown in the following equation (1), the action to the startup control of each base station device 11 for the next 30 minutes, and the constraint condition 132 to cost <= 1 (cost threshold = 1).

[0180]

number

[0181] In the above formula (1), the maximum load value is, for example, the maximum load value in each of base station device 11a, base station device 11b, and base station device 11c. That is, the above formula 1 indicates that, for example, when the maximum load value exceeds the load threshold, the cost increases.

[0182] [Range determination process performed in the example shown in Figure 25, etc.] Next, the range determination process performed in the example shown in FIG. 25 and FIG. 26 will be described. FIG. 27 and FIG. 28 are flow charts for explaining the range determination process performed in the example shown in FIG. 25 and the like. Hereinafter, the description will be made assuming that the parameter related to the constraint condition 132 is the load threshold included in the cost calculation formula 131. Hereinafter, the upper limit of the change range of the load threshold is also simply referred to as the change range upper limit, and the lower limit of the change range of the parameter is also simply referred to as the change range lower limit. In addition, in the example shown in FIG. 27 and the like, the process of S202 corresponds to the range determination process without using the agent AG2, and each process from S203 onwards corresponds to the second range determination process. In addition, each of the upper limit of the change range, the lower limit of the change range, the upper candidate of the change range lower limit (hereinafter also simply referred to as the upper side of the change range lower limit), the lower candidate of the change range lower limit (hereinafter also simply referred to as the lower side of the change range lower limit), and the new candidate of the change range lower limit (hereinafter also referred to as the new candidate of the change range lower limit) may be stored in the information storage area 130, for example.

[0183] As shown in FIG. 27, range determining section 115 waits until the range determining timing, for example (NO in S201).

[0184] Then, when the range determination timing arrives (YES in S201), the range determination unit 115 sets, for example, a maximum load value for the upper limit of the changed range (S202). Note that the maximum load value here may be, for example, a maximum load value in the case where the control target OB repeats an action that may not satisfy the constraint condition 132.

[0185] Next, range determination unit 115 sets, for example, 0 below the lower limit of the change range (S203).

[0186] Furthermore, range determination section 115 sets, for example, the value indicated by the upper limit of the change range set in the process of S202 above the lower limit of the change range (S204).

[0187] Next, the range determination unit 115 sets, for example, the average value of the lower value of the change range lower limit set in the processing of S203 and the upper value of the change range lower limit set in the processing of S204 as a new candidate for the lower limit of the change range (S205).

[0188] Furthermore, the second learning unit 114 generates an agent AG2 by using the constraint condition 132 in the case where the new candidate for the lower limit of the change range set in the process of S205 is set as the load threshold (S206).

[0189] Then, as shown in FIG. 28, the range determining unit 115 determines whether or not the agent AG2 generated in the process of S206 satisfies the first condition (S211).

[0190] Specifically, the range determination unit 115 acquires, for example, an action of the controlled object OB output from the agent AG2 (the agent AG2 generated in the immediately preceding process of S206) in response to an input of a state from the controlled object OB. Then, the range determination unit 115 acquires a maximum load value from the controlled object OB, for example, by having the controlled object OB perform the acquired action. After that, for example, if the acquired maximum load value is smaller than the sum of a load threshold value and a predetermined margin, the range determination unit 115 determines that the agent AG2 generated in the process of S206 satisfies the first condition.

[0191] As a result, if it is determined that the agent AG2 generated in the processing of S206 satisfies the first condition (YES in S211), the range determination unit 115 sets, for example, the value indicated by the new candidate for the lower limit of the change range set in the processing of S205 below the lower limit of the change range (S212).

[0192] On the other hand, if it is determined that the agent AG2 generated in the processing of S206 does not satisfy the first condition (NO in S211), the range determination unit 115 sets, for example, the value indicated by the new candidate for the lower limit of the change range set in the processing of S205 to the upper side of the lower limit of the change range (S213).

[0193] Then, the range determination unit 115 determines, for example, whether the relationship between the value indicated by the lower side of the change range lower limit set in the processing of S212 and the value indicated by the upper side of the change range lower limit set in the processing of S213 satisfies a second condition (S214).

[0194] Specifically, for example, if the difference between the value indicated by the lower side of the change range lower limit set in processing S212 and the value indicated by the upper side of the change range lower limit set in processing S213 is smaller than a predetermined threshold, range determination unit 115 determines that the relationship between the value indicated by the lower side of the change range lower limit set in processing S212 and the value indicated by the upper side of the change range lower limit set in processing S213 satisfies the second condition.

[0195] As a result, if it is determined that the relationship between the value indicated by the lower side of the change range lower limit set in the processing of S212 and the value indicated by the upper side of the change range lower limit set in the processing of S213 does not satisfy the second condition (NO in S214), the second learning unit 114 and the range determination unit 115 perform, for example, the processing from S206 onwards again.

[0196] On the other hand, if it is determined that the relationship between the value indicated by the lower side of the change range lower limit set in the processing of S212 and the value indicated by the upper side of the change range lower limit set in the processing of S213 satisfies the second condition (YES in S214), the range determination unit 115 sets, for example, the value indicated by the upper side of the change range lower limit set in the processing of S212 to the change range lower limit (S215).

[0197] As a result, the information processing device 1 in this embodiment becomes able to identify, for example, the range from the lower limit of the change range to the upper limit of the change range as the change range of a parameter related to the constraint condition 132 (the load threshold included in the cost calculation formula 131).

[0198] The above embodiment can be summarized as follows:

[0199] (Appendix 1) A policy learning device that learns an agent that outputs an action of a control object in response to an input of a state of the control object by reinforcement learning, a parameter changing unit that changes a parameter related to a constraint condition in the reinforcement learning for each of a predetermined number of learning steps in the reinforcement learning; and an agent learning unit that learns the agent so as to satisfy the constraint condition and by using the parameter as at least a part of the state. A policy learning device comprising:

[0200] (Appendix 2) In Appendix 1: The parameter change unit randomly changes the parameters within a predetermined change range. A policy learning device comprising:

[0201] (Appendix 3) Appendix 1 further states: a cost calculation unit that calculates a cost from the state of the control object; the constraint is a constraint on the cost, The parameter is a cost threshold. A policy learning device comprising:

[0202] (Appendix 4) In Appendix 1: a cost calculation unit that calculates a cost from the state of the control object; the constraint is a constraint on the cost, The parameters are parameters used in calculating the cost. A policy learning device comprising:

[0203] (Appendix 5) Appendix 1 further states: a range determination unit that determines a change range of the parameter in accordance with another cost when the control target performs another action output from another agent learned by another reinforcement learning; A policy learning device comprising:

[0204] (Appendix 6) In Appendix 1: a policy estimation unit that acquires a new action of the control object output from the agent in response to an input of a new state of the control object; a measure output unit that outputs the acquired new action, the policy estimation unit inputs new parameters related to the constraint conditions to the agent as at least a part of the new state; A policy learning device comprising:

[0205] (Appendix 7) A policy learning method for learning an agent that outputs an action of a control object in response to an input of a state of the control object by reinforcement learning, comprising the steps of: changing a parameter related to a constraint condition in the reinforcement learning for each of a predetermined number of learning steps in the reinforcement learning; training said agent to satisfy said constraints and by using said parameters as at least a part of said states; 2. A policy learning method comprising the steps of:

[0206] (Appendix 8) Appendix 7 further states: determining a change range of the parameter according to another cost when the control target performs another action output from another agent learned by another reinforcement learning; A policy learning method, characterized in that the processing is executed by a computer.

[0207] (Appendix 9) A communication system having a base station device and a policy learning device that learns an agent that outputs an action of the base station device in response to an input of a state of the base station device by reinforcement learning, The policy learning device includes: changing a parameter related to a constraint condition in the reinforcement learning for each of a predetermined number of learning steps in the reinforcement learning; training said agent to satisfy said constraints and by using said parameters as at least a part of said states; A communication system comprising:

[0208] (Appendix 10) Appendix 9 further states: the policy learning device determines a change range of the parameter in accordance with another cost when the control target performs another action output from another agent learned by another reinforcement learning. A communication system comprising: [Explanation of symbols]

[0209] 1: Information processing device 2: Storage device 5: Operation terminal 10: Information processing system 11: Base station device 12: Terminal device 101: CPU 102: Memory 103: Communication circuit 104: Storage device 105: Bus 110: Program 130: Information storage area AG1: Agent AG2: Agent OB: Control object

Claims

1. A policy learning device that learns an agent that outputs an action of a control object in response to an input of a state of the control object by reinforcement learning, a parameter changing unit that changes a parameter related to a constraint condition in the reinforcement learning for each of a predetermined number of learning steps in the reinforcement learning; and an agent learning unit that learns the agent so as to satisfy the constraint condition and by using the parameter as at least a part of the state. A policy learning device comprising:

2. In claim 1, The parameter change unit randomly changes the parameters within a predetermined change range. A policy learning device comprising:

3. In claim 1, further comprising: a cost calculation unit that calculates a cost from the state of the control object; the constraint is a constraint on the cost, The parameter is a cost threshold. A policy learning device comprising:

4. In claim 1, a cost calculation unit that calculates a cost from the state of the control object; the constraint is a constraint on the cost, The parameters are parameters used in calculating the cost. A policy learning device comprising:

5. In claim 1, further comprising: a range determination unit that determines a change range of the parameter in accordance with another cost when the control target performs another action output from another agent learned by another reinforcement learning; A policy learning device comprising:

6. In claim 1, a policy estimation unit that acquires a new action of the control object output from the agent in response to an input of a new state of the control object; a measure output unit that outputs the acquired new action, the policy estimation unit inputs new parameters related to the constraint conditions to the agent as at least a part of the new state; A policy learning device comprising:

7. A policy learning method for learning an agent that outputs an action of a control object in response to an input of a state of the control object by reinforcement learning, comprising the steps of: changing a parameter related to a constraint condition in the reinforcement learning for each of a predetermined number of learning steps in the reinforcement learning; training said agent to satisfy said constraints and by using said parameters as at least a part of said states; 2. A policy learning method comprising the steps of:

8. A communication system having a base station device and a policy learning device that learns an agent that outputs an action of the base station device in response to an input of a state of the base station device by reinforcement learning, The policy learning device includes: changing a parameter related to a constraint condition in the reinforcement learning for each of a predetermined number of learning steps in the reinforcement learning; training said agent to satisfy said constraints and by using said parameters as at least a part of said states; A communication system comprising:

Images