Control system, control method, and control program

Abstract

This document provides a control system, method, and program that utilize machine learning techniques. [Solution] The control system obtains state variables from the environment, including the operating status of the control device corresponding to each agent, assigns an order to multiple agents, predicts the action of the nth agent based on the corresponding state variable and the reward-related cumulative value provided by the (n-1)th agent, predicts the corresponding reward-related value based on the predicted action, generates a reward-related cumulative value provided to the (n+1)th agent by subtracting the reward-related value corresponding to the (n+1)th agent from the sum of the reward-related value corresponding to the nth agent and the reward-related cumulative value provided by the (n-1)th agent, and sequentially performs action prediction, reward-related value prediction, and cumulative value generation for all target agents, controlling each control device based on the predicted action of each agent.

Description

[Technical Field]

[0001] This invention relates to control systems, particularly control systems that utilize machine learning technology. [Background technology]

[0002] In recent years, research has been progressing on techniques for controlling targets using multi-agent reinforcement learning. For example, Non-Patent Document 1 describes a method in multi-agent machine learning in which a series of agents sequentially select actions and share those selected actions with the next agent to control the target. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "Jose R. Vazquez-Canteli et al.,"MARLISA: Multi-Agent Reinforcement Learning with Iterative Sequential Action Selection for Load Shaping of Grid-Interactive Connected Buildings", 18 November 2020, BuildSys '20: Proceedings of the 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation,Association for Computing Machinery, Pages 170-179" [Overview of the project] [Problems that the invention aims to solve]

[0004] However, this type of multi-agent machine learning technology as described in Non-Patent Document 1 has still been difficult to apply to actual control situations.

[0005] For example, in a real environment, some of the plurality of control devices may stop operating, and the operating status may be dynamically changed. Conventional systems cannot cope with such a situation where the number of agents changes.

[0006] The present invention has been made under the above-mentioned technical background, and an object of the present invention is to solve problems in applying multi-agent reinforcement learning technology to actual control systems. [Means for Solving the Problems]

[0007] The above-mentioned technical problem can be solved by a control system, method, program or the like having the following configuration.

[0008] In other words, the control device according to the present invention is a control system comprising a plurality of control devices for controlling an object in an environment, wherein each control device is associated with a virtual agent that predicts the actions related to the control of each control device, the control system comprising: a state variable acquisition unit that acquires state variables corresponding to each agent, including the operating state of the control device corresponding to each agent, from the environment; a sequence assignment unit that assigns an order to the plurality of agents; an action prediction unit that predicts the action of the nth agent based on the state variables corresponding to the nth agent and the reward-related cumulative value provided by the (n-1)th agent; and the nth agent based on the predicted action. The system includes: a reward-related value prediction unit that predicts reward-related values ​​corresponding to agents; a cumulative value calculation unit that generates a cumulative reward-related value provided to the (n+1)th agent by subtracting the reward-related value corresponding to the (n+1)th agent from the sum of the reward-related value corresponding to the nth agent and the cumulative reward-related value provided by the (n-1)th agent; a repeating operation unit that sequentially operates the action prediction unit, the reward-related value prediction unit, and the cumulative value calculation unit for each of the target agents, thereby predicting the action of all of the agents; and a control unit that controls each of the control devices based on the action predicted by each agent.

[0009] With this configuration, each agent makes action predictions while considering the prediction results of other agents, thus enabling overall control optimization. Furthermore, since the state variables include the operating status of the control devices corresponding to other agents, flexible control can be performed even if the operating status of the control devices changes. In other words, it can resolve the challenges of applying multi-agent reinforcement learning technology to real-world control systems, such as responding to changes in the operating status of control devices.

[0010] Each agent may be assigned a normalized coefficient, and the behavior prediction unit may predict the behavior of the nth agent based on the state variable corresponding to the nth agent, the reward-related cumulative value provided by the (n-1)th agent, and the sum of the coefficients for each of the agents up to the (n-1)th agent.

[0011] With this configuration, the influence of the agent referencing order can be reduced using coefficients, thereby improving the accuracy of behavior prediction.

[0012] The reward-related value prediction unit may also predict the reward-related value based on predetermined input values ​​assigned to each agent, in addition to the predicted behavior.

[0013] With this configuration, by adding predetermined inputs such as state variables, reward-related values ​​can be predicted with greater accuracy.

[0014] The system may further include a second repeating operation unit that causes the aforementioned repeating operation unit to operate a predetermined number of times.

[0015] This configuration allows for further improvement in the accuracy of behavioral predictions.

[0016] The behavior prediction unit may use a recurrent neural network to predict the behavior.

[0017] This configuration allows for the prediction of behavior that takes context into account.

[0018] The aforementioned recurrent neural network may also be an LSTM.

[0019] This configuration resolves the vanishing gradient problem, integrates short-term and long-term memory, and enables the learning of long-term dependencies. This is suitable for dealing with real-world environments that do not necessarily possess Markov properties (or are partial Markov processes).

[0020] The reward-related value prediction unit may use a decision tree to predict the reward-related value.

[0021] With this configuration, reward-related values ​​can be predicted in a manner that is relatively easy to interpret and highly versatile.

[0022] The aforementioned decision tree may also be a gradient boosting decision tree.

[0023] With this configuration, it is possible to predict reward-related values ​​using a model that strikes a good balance between the amount of memory used and the learning and prediction accuracy.

[0024] The agent may function as an actor in actor-critic reinforcement learning.

[0025] With this configuration, the results of behavior generation from the agent related to the actor can be used to train both the actor and the critic to maximize future rewards.

[0026] In the actor-critic type reinforcement learning described above, the critic may learn an action-value function.

[0027] With this configuration, it is possible to induce more rewarding and desirable behaviors through learning from the critique.

[0028] Viewed from a different perspective, the present invention is a control method for a control system. That is, the control method according to the present invention includes a plurality of control devices for controlling an object in an environment, each of which control device is associated with a virtual agent that predicts the actions related to the control of each control device, and the method for controlling a control system includes: a state variable acquisition step of acquiring state variables corresponding to each agent, including the operating state of the control device corresponding to each agent, from the environment; an ordering step of assigning an order to the plurality of agents; an action prediction step of predicting the action of the nth agent based on the state variables corresponding to the nth agent and the reward-related cumulative value provided by the (n-1)th agent; and based on the predicted action, the nth agent The system includes: a reward-related value prediction step for predicting a reward-related value corresponding to an agent; a cumulative value calculation step for generating a cumulative reward-related value provided to the (n+1)th agent by subtracting the reward-related value corresponding to the (n+1)th agent from the sum of the reward-related value corresponding to the nth agent and the cumulative reward-related value provided by the (n-1)th agent; a repeating operation step for sequentially applying the action prediction step, the reward-related value prediction step, and the cumulative value calculation step to each of the target agents, thereby causing all of the agents to predict their actions; and a control step for controlling each of the control devices based on the actions predicted by each agent.

[0029] Viewed from a different perspective, the present invention is a control program for a control system. That is, the control program according to the present invention includes a plurality of control devices that control an object in an environment, and each of the control devices is associated with a virtual agent that predicts the actions related to the control of each of the control devices, and the control program for a control system includes a state variable acquisition step of acquiring state variables corresponding to each of the agents, including the operating state of the control device corresponding to each of the agents from the environment; an ordering step of assigning an order to the plurality of agents; an action prediction step of predicting the action of the nth agent based on the state variables corresponding to the nth agent and the reward-related cumulative value provided by the (n-1)th agent; and based on the predicted action, the nth The system includes: a reward-related value prediction step for predicting a reward-related value corresponding to an agent; a cumulative value calculation step for generating a cumulative reward-related value provided to the (n+1)th agent by subtracting the reward-related value corresponding to the (n+1)th agent from the sum of the reward-related value corresponding to the nth agent and the cumulative reward-related value provided by the (n-1)th agent; a repeating operation step for sequentially applying the action prediction step, the reward-related value prediction step, and the cumulative value calculation step to each of the target agents, thereby causing all of the agents to predict their actions; and a control step for controlling each of the control devices based on the actions predicted by each agent. [Effects of the Invention]

[0030] According to the present invention, the challenges in applying multi-agent reinforcement learning technology to real-world control systems can be resolved. [Brief explanation of the drawing]

[0031] [Figure 1] Figure 1 is a detailed configuration diagram of the information processing device. [Figure 2] Figure 2 is a functional block diagram of the information processing device. [Figure 3] Figure 3 is an overall diagram of the control system for a chemical plant (reactor). [Figure 4] Figure 4 is a conceptual diagram of the reinforcement learning method employed by the control system. [Figure 5] Figure 5 is a general flowchart relating to the predictive operation of the control system. [Figure 6] Figure 6 is a detailed flowchart of the prediction process. [Figure 7] Figure 7 is a conceptual diagram of the prediction process. [Figure 8] Figure 8 is a detailed flowchart of the reinforcement learning process. [Figure 9] Figure 9 is an explanatory diagram showing an example of a modified application to an energy management system. [Modes for carrying out the invention]

[0032] Hereinafter, one embodiment of the present invention will be described in detail with reference to the attached drawings.

[0033] (1. First Embodiment) The first embodiment will be described with reference to the drawings. In this embodiment, an example will be described in which the control system according to the present invention is applied to a chemical plant, in particular to the control of the input of raw materials to a reactor and stirring.

[0034] (1.1 Composition) Figure 1 is a detailed configuration diagram of the information processing device 1 included in the control system 40 according to this embodiment. As is clear from the figure, the information processing device 1 comprises a processor 10, a storage unit 11, a communication unit 12, a display control unit 15, an I / O processing unit 16, and an audio control unit 17.

[0035] The processor 10 is an arithmetic unit such as a CPU, which executes programs to realize the various operations described later. The storage unit 11 is a storage medium such as ROM / RAM, flash memory, or hard disk (including non-temporary computer-readable storage media), which stores programs and data to realize the various operations described later. The communication unit 12 is a communication unit that enables the exchange of information with external devices. The display control unit 15 performs processing to control image information and the like that is displayed on the display device 19. The I / O processing unit 16 processes input / output signals with external devices. The audio control unit 17 performs processing to control audio information and the like that is output to the speaker.

[0036] The processor 10, together with the memory unit 11 and other components, provides various functions.

[0037] Figure 2 is a functional block diagram of the information processing device 1. As is clear from the figure, the processor 10 includes a data acquisition unit 101, a preprocessing unit 102, a prediction processing unit 103, a storage processing unit 104, and a learning processing unit 105.

[0038] The data acquisition unit 101 acquires data from the storage unit 11 and provides it to the preprocessing unit 102, the prediction processing unit 103, and the learning processing unit 105. The preprocessing unit 102 performs preprocessing on the data acquired from the data acquisition unit 101 and provides the processed data to the prediction processing unit 103.

[0039] In this embodiment, the term "prediction" refers to the output of the trained model and can be rephrased using other terms. For example, terms such as "inference" may be used.

[0040] The prediction processing unit 103 performs prediction processing based on the data provided by the data acquisition unit 101 or the preprocessing unit 102. The prediction processing results are provided to the storage processing unit 104. The storage processing unit 104 provides the provided data to the storage unit 11 for storage.

[0041] The learning processing unit 105 performs machine learning processing based on the data provided by the data acquisition unit 101. The learning processing results are provided to the memory processing unit 104. The memory processing unit 104 provides the provided data to the memory unit 11 for storage.

[0042] Note that the hardware configuration is not limited to the configuration according to this embodiment. Therefore, other configurations may be adopted, for example, it may be configured as a server-client system, or a separate server may be provided for storing and providing data.

[0043] Furthermore, in this embodiment, the computer program may be provided as a computer program product or a recording medium for storing the computer program. The functions corresponding to the processor 10 may also be implemented circuit-wise using an IC such as an FPGA.

[0044] Figure 3 is an overall configuration diagram of the control system 40 of the chemical plant according to this embodiment. In the example shown, the control system 40 controls the amount of various raw materials supplied from the tank 30 to the reactor 45, and the amount of stirring of the supplied raw materials within the reactor 45.

[0045] In the example shown in the figure, (N-3) tanks 31 are provided for raw material A, and the amount of raw material A supplied from each tank 31 is controlled by controlling the output (in this embodiment, the rotation frequency) of the corresponding pump 310. For raw material B, one tank 32 is provided, and the amount of raw material B supplied from the tank 32 is controlled by controlling the output (in this embodiment, the rotation frequency) of the corresponding pump 320. For raw material C, one tank 33 is provided, and the amount of raw material C supplied from the tank 33 is controlled by controlling the output (in this embodiment, the rotation frequency) of the corresponding pump 330. In addition, a stirrer is placed inside the reactor 45, and the amount of stirring by the stirrer is controlled by controlling the output (in this embodiment, the rotation speed) of the stirrer motor 43 connected to the stirrer.

[0046] Furthermore, each of the N control devices, consisting of the pumps (310, 320, 330) and the agitator motor 43, is connected to an information processing device 1 (not shown) and controlled by N virtual agents.

[0047] Figure 4 is a conceptual diagram of the reinforcement learning employed by the control system 40 according to this embodiment. As is clear from the figure, this embodiment employs a so-called soft actor-critic type of reinforcement learning technique that includes an actor 50 and a critic 60, and is suitable for real-world learning by maximizing the policy entropy.

[0048] Actor 50 predicts action a based on information such as state s obtained from the environment 70. Actor 50 includes a policy network 51 containing multiple trained models corresponding to virtual agents that predict action a based on a given state s, etc. The action a predicted by each agent becomes the output to each control device, i.e., the rotation frequency of each pump (310, 320, 330) and the rotation speed of the agitator motor 43.

[0049] In this embodiment, a recurrent neural network (RNN), particularly LSTM (Long Short-Term Memory), is used as the trained model.

[0050] This configuration allows for context-aware behavioral prediction. In particular, employing an LSTM eliminates the vanishing gradient problem and enables the integration of short-term and long-term memory, allowing for the learning of long-term dependencies. This is well-suited for dealing with real-world environments that do not necessarily possess Markov properties (or are partial Markov processes).

[0051] In this embodiment, actor 50 further includes a reward-related value prediction model 52 which includes multiple trained models that correspond to each trained model (LSTM) and predict reward-related values ​​based on the predicted action a, etc. The reward-related value is a value of the same dimension as the reward, which facilitates reward calculation. In this embodiment, the reward is the sum of the energy consumed by all agents or an equivalent amount, and the reward-related value is the energy consumed e for each agent. The trained model related to the reward-related value prediction model 52 is a decision tree, in particular a gradient-boosting decision tree (GBDT).

[0052] With this configuration, reward-related values ​​can be predicted using decision trees in a manner that is relatively easy to interpret and highly generalizable. In particular, by using GBDT, reward-related values ​​can be predicted with a model that has a good balance between the amount of memory used and the learning and prediction accuracy.

[0053] In this embodiment, the reward is set to a value that increases as the total energy consumption of the entire system decreases, but the present invention is not limited to such a configuration. Therefore, it may be set from the viewpoint of, for example, production speed, the intended characteristics of the product, etc.

[0054] The policy network 51 is updated based on the loss function (P-Loss) 53 related to the actor 50. The update method is not particularly limited, but in this embodiment, it is performed by mini-batch stochastic gradient descent.

[0055] All information regarding the environment 70 is aggregated and buffered in the memory unit 11 and stored. The information regarding the environment 70 includes all the variables described below, namely, state s, action a, predetermined coefficient c, reward-related value e, reward r, etc. In this embodiment, the reward r is set as a variable that increases as the total energy consumption of the system over time decreases.

[0056] Critic 60 includes a Q-network 61 containing an action-value function (Q-function) and a target Q-network 62 containing a target Q-function. Both the Q-function and the target Q-function are trained models such as neural networks. The Q-network 61 and the target Q-network 62 output Q-values ​​and target Q-values ​​based on values ​​read from the memory unit 11.

[0057] The loss function (P-Loss) 53 for actor 50 and the loss function (Q-Loss) 63 for the critic are calculated based on these Q values ​​and target Q values, etc.

[0058] The Q network 61 is updated based on the loss function (Q-Loss) 63 of the critique 60. The update method is not particularly limited, but in this embodiment, it is performed by minibatch stochastic gradient descent.

[0059] In this embodiment, the target Q network 62 is updated by a soft update method. Specifically, in this embodiment, the target Q network 62 is updated by taking a weighted average of the model parameters related to the Q network 61 and the target Q network 62.

[0060] (1.2.1 Predictive behavior) Next, we will describe the predicted operation using the control system 40, that is, the actual control operation.

[0061] Figure 5 is a general flowchart of the predictive operation of the control system 40. As is clear from the figure, when processing starts, the data acquisition unit 101 reads and references the data buffered in the storage unit 11 (S10). The amount of data buffered is the amount of data corresponding to the time series length required for prediction.

[0062] If, as a result of this reference process, it is determined that the data has not accumulated to a predetermined amount (S11NO), a random action is generated, and the generated action is written to the storage unit 11 by the memory processing unit 104. This written action is output to each pump (310, 320, 330) and the agitator motor 43 (S12). Each pump (310, 320, 330) and the agitator motor 43 are controlled using the value related to the predicted action as the target value. After that, the process returns to the beginning, and the series of processes is performed again.

[0063] On the other hand, if it is determined that data has accumulated to a certain amount or more (S11YES), the prediction processing unit 103 executes prediction processing, and the prediction processing result is stored in the storage unit 11 by the storage processing unit 104 (S13). After this prediction processing, the written action is output to each pump (310, 320, 330) and the agitator motor 43 (S15). Each pump (310, 320, 330) and the agitator motor 43 controls the value related to the predicted action as a target value. After this output, the process returns to the beginning, and the series of processes is performed again.

[0064] Refer to Figures 6 and 7 to explain the details of the prediction process (S13). Figure 6 is a detailed flowchart of the prediction process (S13), and Figure 7 is a conceptual diagram of the prediction process.

[0065] When processing begins, the prediction processing unit 103 selects target agents based on information about the control devices acquired by the data acquisition unit 101, and also arranges the selected agents in an arbitrary order (S131). In this embodiment, an agent is provided corresponding to each of the N control devices, including each pump (310, 320, 330) and the agitator motor 43, and an order is assigned to these agents (i.e., agents 1 to N in Figure 7). Each agent selects action a when given a state s, based on policy π.

[0066] After the sorting order setting process, the data acquisition unit 101 obtains observed values ​​from the environment 70 for N agents. nperforms processing to acquire, and the preprocessing unit 102 processes the observation value o n performs preprocessing on to convert it into the state variable s n processing for conversion is performed (S132). This converted state variable s n is provided to each agent.

[0067] Note that this state variable s n includes various variables related to the environment 70. In the present embodiment, the state variable s n includes various parameters related to the environment 70, that is, air temperature, humidity, monitoring values of an optimization target (internal temperature and pressure in a reactor 45), and monitoring values of a control device (material quality values (such as viscosity and particle size of raw materials), etc.). Also, the state variable s n includes information on whether each control device is currently in an operating state. For example, when only the raw material A is charged into the reactor and stirred, only the agent corresponding to the pump 310 associated with the tank 31 for the raw material A and the agitator motor 43 is set as in an operating state (flag is 1), while agents corresponding to pumps (320, 330) associated with the tank 32 for the raw material B and the tank 33 for the raw material C are set as in a non-operating state (flag is 0).

[0068] According to such a configuration, since the state variable s n includes operating states of control devices corresponding to other agents, flexible control can be performed even when the operating state of the control device is changed.

[0069] After the conversion processing of the state variable s n is completed, the prediction processing unit 103 performs processing for normalizing predetermined coefficients corresponding to each agent (S133). Through this normalization, the sum obtained when adding up all N coefficients of each agent becomes 1. Note that in the present embodiment, the energy consumption coefficient c is used as the coefficient n is employed. The energy consumption coefficient c n is a value indicating the tendency of energy consumption in the control device corresponding to each agent.

[0070] With this configuration, by using normalized coefficients, it is possible to prioritize agents that are positioned later in the sequence and have received more information, thereby improving the accuracy of behavior prediction.

[0071] After the normalization process, the prediction processing unit 103 initializes the values ​​of the cumulative total energy consumption e_cum and the cumulative total coefficient c_cum (S135). In this embodiment, the values ​​of e_cum0 and c_cum0 are set to 0.

[0072] After this initialization process, the prediction processing unit 103 initializes a variable k that indicates the number of repetitions of the process described later (S136). For example, the variable k is set to 1.

[0073] Furthermore, after the initialization of variable k, the prediction processing unit 103 initializes variable n, which indicates the agent to be referenced (S137). For example, variable n is set to 1.

[0074] After the initialization process of variable n, the prediction processing unit 103 sets the state variable s for the nth agent. n , the cumulative total of energy consumed for the (n-1)th instance e_cum n-1 , the cumulative sum of the (n-1)th coefficients c_cum n-1 By inputting this into the corresponding trained model (LSTM) of the policy network 51, the agent's action a n Make a prediction (S138).

[0075] After the behavior prediction processing, the prediction processing unit 103 determines the predicted behavior a n and a predetermined input x n This is input into the trained model (gradient boosting decision tree) of the reward association value prediction model 52 corresponding to the Nth agent, and the predicted energy consumption e_hat for that agent is used as the reward association value. n The process of calculating is performed (S140). In this embodiment, the input x n This is the state variable s corresponding to the agent. n That is the case.

[0076] With this configuration, the predicted behavior a n In addition, the state variable s at that time n Also input x n Since it is input as such, reward-related values ​​can be predicted with greater accuracy.

[0077] After the calculation process, the system updates various cumulative values. Specifically, the prediction processing unit 103 updates the cumulative coefficient values ​​c_cum up to the (n-1)th value. n-1 and the (n-1)th coefficient c n-1 The sum of the nth coefficient cumulative value c_cum n-1 The process of assigning the value to is performed (S141).

[0078] Furthermore, the prediction processing unit 103 calculates the cumulative total of energy consumption up to the (n-1)th item, e_cum n-1 and the nth predicted energy consumption e_hat n The n+1th predicted energy consumption e_hat is obtained from the sum of these values. n+1 The value obtained by subtracting this is the cumulative total of energy consumed for the nth term, e_cum n The process of substituting the value into (S142) is performed. Note that the (n+1)th predicted energy consumption e_hat n+1 This refers to the value from the previous iteration in the loop using the variable k.

[0079] After these update processes, the prediction processing unit 103 determines whether all agents have been referenced, that is, whether N agents have been referenced (S143). If not all agents have been referenced yet (S143NO), the variable n is incremented by 1 (S145), and the same process (S138~S142) is performed for the next (n+1) agent. On the other hand, if it is determined that all agents have been referenced (S143YES), the process proceeds to the next stage, as the behavior prediction for all agents is considered complete.

[0080] Once all agents have been reviewed, the prediction processing unit 103 determines whether a predetermined iterative process has been completed, that is, whether the series of processes has been repeated m times (S146). If the predetermined number of repetitions has not been completed (S146NO), the variable k is incremented by 1 (S147), and the same process (S137~S143) is executed in the next (k+1) iteration.

[0081] With this configuration, the accuracy of behavioral prediction can be further improved through iterative processing.

[0082] On the other hand, if the process is repeated m times (S146YES), the prediction processing unit 103 determines the generated action a n and predicted energy consumption value e_hat n The data is provided to the memory processing unit 104 and stored in the memory unit 11 (S148), and the process ends. This stored prediction result, i.e., the generated action a n The action output processing (S15) is executed using this.

[0083] With the above configuration, each agent makes action predictions while considering the prediction results of other agents, thus enabling overall control optimization. Furthermore, since the state variables include the operating status of the control device corresponding to the agent, flexible control can be performed even if the operating status of the control device changes. In other words, it is possible to resolve the challenges of applying multi-agent reinforcement learning technology to real-world control systems, such as the need for flexible response to changes in the operating status of the control device.

[0084] (1.2.2 Reinforcement Learning Operation) Next, we will explain the machine learning process, particularly the reinforcement learning flow, used to generate the trained models used in predictive actions.

[0085] Figure 8 is a detailed flowchart of the reinforcement learning process performed by the learning processing unit 105. As is clear from the figure, when the process starts, the learning processing unit 105 initializes the agent to be referenced, that is, it initializes the variable n corresponding to the agent (S21). For example, the variable n is set to 1.

[0086] After the initialization process, the learning processing unit 105 reads the data stored in the memory unit 11 (S22). More specifically, it reads the state variable s for each batch size. n , action a n , reward r, behavior a n The next state s' n Perform the process of reading values ​​such as these.

[0087] After the reading process, the learning processing unit 105 uses the corresponding trained model of the policy network 51 to process the state s' n Based on this, perform prediction processing and predict action a' n The process of generating is performed (S23).

[0088] After this process, the learning processing unit 105 processes the reward r and the state variable s. n , action a n , state s' n , predictive behavior a' n This is input to the target Q network 62 to predict the target Q value (S25). This target Q value is used by the Q network 61 to learn, and contributes to stabilizing the learning of the Q network 61.

[0089] After the target Q value prediction process, the learning processing unit 105 receives the reward r and the state variable s. n , action a n The Q value is predicted by inputting this into the Q network 61 (S26).

[0090] After the target Q-value and Q-value prediction process, the learning processing unit 105 calculates the loss function (Q-Loss) 63 of the critic 60 based on these target Q-values ​​and Q-values ​​(S27). Subsequently, the parameters of the Q-function related to the Q-network 62 are updated using this loss function (Q-Loss) (S28). This update is performed, for example, by mini-batch stochastic gradient descent.

[0091] Once the learning process in the Q function is complete, the learning processing unit 105 sends the state variable s to the policy network 51. n By inputting action a'' n The process of predicting is performed (S30). After this prediction process, the learning processing unit 105 processes these state variables s n , action a'' n Based on this, the loss function (P-Loss) 53 of the policy network 51 (or actor 50) is calculated (S31). Subsequently, the parameters of the trained model (LSTM) related to the policy network 51 are updated using this loss function (P-Loss) (S32). This update is performed, for example, by mini-batch stochastic gradient descent.

[0092] After the policy function training process, the training unit 105 performs a training process on the target Q network 62 to update the parameters of the trained model (S33). This training process is performed using a soft update method. In this embodiment, the target Q network 62 is updated by taking a weighted average of the model parameters of the Q network 61 and the target Q network 62 using a weight coefficient τ.

[0093] Once the learning process for the target Q function is complete, the learning processing unit 105 determines whether or not all agents have been referenced (S35). If not all agents have been referenced yet (S35NO), the variable n is incremented by 1, and the series of processes is repeated (S22-S33). On the other hand, if it is determined that all agents have been referenced (S35YES), the learning process ends, and all learning results are stored in the storage unit 11 by the storage processing unit 104.

[0094] With this configuration, the results of the actions generated by the agent related to actor 50 can be used to train both actor 50 and critic 60 to maximize future rewards.

[0095] Although the detailed flowchart in Figure 8 does not mention the training or updating of the reward-related value prediction model 52, this model may also be trained sequentially or in batches. That is, a certain state variable s n Action a n The energy consumption of the target control device corresponding to each agent when this was performed. n We can observe these parameters and then perform supervised learning on the model (GBDT) based on this data.

[0096] Furthermore, the order of each step in the above flow may be changed, except for those that require the calculation results from the previous step.

[0097] (2. Variant) The present invention can be implemented in various modified forms.

[0098] In the above embodiment, the agent predicts action a n Rather, it is a predicted value of the reward-related value (e_hat n ) was converted to and provided to the next agent, but the present invention is not limited to such a configuration.Therefore, without providing a reward-related value prediction model 52, etc., simply action a n You may provide this to the next agent.

[0099] Although the above-described embodiment illustrates an example of application to a chemical plant, the present invention can be applied to a variety of applications. Therefore, it can be applied to systems that use multiple devices (agents) for control, such as HEMS (Home Energy Management Systems), district heating and cooling systems, data centers, and energy management systems (EMS) related to solar and wind power generation. It can also be applied to autonomous driving systems, robot control in factories and homes, stock trading, smart cities, customer support, and the like.

[0100] Figure 9 shows a modified example in which the control system according to the present invention is applied to a temperature control system 90 (air conditioning system) of a building 95 as an example of an energy management system (EMS). The building 95 is, for example, an office building, a commercial facility, a hospital, etc.

[0101] Building 95 is equipped with an air handling unit (AHU) 89, from which cold air is supplied. The AHU 89 provides cold air using chilled water supplied from an air-cooled chiller 81, a turbo chiller 82, and an absorption chiller 83.

[0102] Chilled water pumps 883 and 884 are provided between the turbo chiller 82 and the AHU 89, and between the absorption chiller 83 and the AHU 89, respectively.

[0103] Furthermore, the turbo chiller 82 operates on power supplied from the commercial power supply / cogeneration system 86 and performs heat exchange by circulating cooling water with the cooling tower 85. Similarly, the absorption chiller 83 operates on power supplied from the boiler / cogeneration system 88 and performs heat exchange by circulating cooling water with the cooling tower 87. For these circulations, cooling water pumps 881 and 882 are provided between the cooling towers (88 and 87) and the chillers (82 and 83), respectively.

[0104] In this system, the control devices are the chilled water pumps 883 and 884, and the cooling water pumps 881 and 882. That is, various state variables s provided by the environment. nFor example, based on temperature, humidity, wind speed, monitoring values ​​of the target to be optimized (room temperature, occupancy status, etc.), and monitoring values ​​of the control device (chilled water temperature, cooling water temperature, operating status, etc.), each agent (881-884) performs action a n In other words, in this example, we predict the pump frequency. In this case, the reward r is a value that increases as the total sum of energy consumed decreases, for example. Note that the reward is not limited to this and can be set arbitrarily, for example, it may be an indicator value for stable control (such as frequency fluctuation).

[0105] Although embodiments of the present invention have been described above, these embodiments represent only a part of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments. Furthermore, the above embodiments can be combined as appropriate, as long as no contradictions arise. [Industrial applicability]

[0106] This invention can be used in various industries and other fields that utilize machine learning technology. [Explanation of symbols]

[0107] 1. Information Processing Device 10 processors 101 Data Acquisition Unit 102 Pre-processing section 103 Prediction Processing Unit 104 Memory Processing Unit 105 Learning Processing Unit 11 Storage section 12 Communications Department 15 Display Control Unit 16 I / O Processing Unit 17 Audio Control Unit 30 tanks 31 Tank (Raw Material A) 310 Input volume control pump 32 Tanks (Raw Material B) 320 Input volume control pump 33 Tanks (Raw Material C) 330 Input volume control pump 40 Control Systems 43. Stirrer motor 45 Reactor 50 Actors 51 Policy Network 52 Reward-Related Value Prediction Models 53. Loss Function (Policy Network / Actor) 60 Critics 61 Q Network 62 Target Q Network 63. Loss Function (Critical) 70 Environment 90 Temperature control system 81 Air-cooled chiller 82 Turbo chiller 83 Absorption chiller 85 Cooling Tower 86 Power supply (commercial power supply / cogeneration) 87 Cooling Tower 88 Heat sources (boilers, cogeneration) 881 Cooling water pump 882 Cooling water pump 883 Chilled water pump 884 Chilled water pump 89 AHU (Air Handling Unit)

Claims

1. A control system comprising a plurality of control devices for controlling objects in an environment, wherein each control device is associated with a virtual agent that predicts actions related to the control of each control device, A state variable acquisition unit acquires state variables corresponding to each agent, including the operating status of the control device corresponding to each agent, from the environment described above. A sequence assigning unit that assigns a sequence to a plurality of the aforementioned agents, In the nth agent, a behavior prediction unit predicts the behavior based on the state variable corresponding to the nth agent and the reward-related cumulative value provided by the (n-1)th agent. A reward-related value prediction unit predicts the reward-related value corresponding to the nth agent based on the predicted action, A cumulative value calculation unit generates a cumulative reward-related value provided to the (n+1)th agent by subtracting the reward-related value corresponding to the (n+1)th agent from the sum of the reward-related value corresponding to the nth agent and the cumulative reward-related value provided by the (n-1)th agent, A repeating operation unit operates the aforementioned behavior prediction unit, the aforementioned reward-related value prediction unit, and the aforementioned cumulative value calculation unit sequentially for each of the target agents, thereby causing behavior to be predicted for all of the agents. A control system comprising: a control unit that controls each of the control devices based on the actions predicted by each of the agents.

2. Each of the agents is assigned a normalized coefficient. The control system according to claim 1, wherein the action prediction unit predicts the action of the nth agent based on the state variable corresponding to the nth agent, the reward-related cumulative value provided by the (n-1)th agent, and further, the sum of the coefficients related to each of the agents up to the (n-1)th agent.

3. The control system according to claim 1, wherein the reward-related value prediction unit predicts the reward-related value based on a predetermined input value assigned to each agent, in addition to the predicted behavior.

4. The control system according to claim 1, further comprising a second repeating operation unit that causes the repeating operation unit to operate a predetermined number of times.

5. The control system according to claim 1, wherein the behavior prediction unit uses a regressive neural network to predict the behavior.

6. The control system according to claim 5, wherein the recurrent neural network is an LSTM.

7. The control system according to claim 1, wherein the reward-related value prediction unit predicts the reward-related value using a decision tree.

8. The control system according to claim 7, wherein the decision tree is a gradient boosting decision tree.

9. The control system according to claim 1, wherein the agent functions as an actor in actor-critic reinforcement learning.

10. The control system according to claim 9, wherein the critic in the actor-critic type reinforcement learning is trained on an action-value function.

11. A control method for a control system, comprising a plurality of control devices for controlling objects in an environment, wherein each control device is associated with a virtual agent that predicts actions related to the control of each control device, A state variable acquisition step, which involves obtaining state variables corresponding to each agent, including the operating status of the control device corresponding to each agent, from the environment described above. A sequence assignment step, which assigns an order to a plurality of the aforementioned agents, A behavior prediction step in which, in the nth agent, behavior is predicted based on the state variable corresponding to the nth agent and the reward-related cumulative value provided by the (n-1)th agent, A reward-related value prediction step, which predicts the reward-related value corresponding to the nth agent based on the predicted action, A cumulative value calculation step, which generates a cumulative reward-related value to be provided to the (n+1)th agent by subtracting the reward-related value corresponding to the (n+1)th agent from the sum of the reward-related value corresponding to the nth agent and the cumulative reward-related value provided by the (n-1)th agent, A repeating operation step in which the aforementioned behavior prediction step, the aforementioned reward-related value prediction step, and the aforementioned cumulative value calculation step are applied sequentially to each of the target agents, thereby causing all of the agents to predict their actions; A control method comprising: a control step of controlling each of the control devices based on the actions predicted by each of the agents.

12. A control program for a control system, comprising a plurality of control devices for controlling objects in an environment, wherein each control device is associated with a virtual agent that predicts actions related to the control of each control device, A state variable acquisition step, which involves obtaining state variables corresponding to each agent, including the operating status of the control device corresponding to each agent, from the environment described above. A sequence assignment step, which assigns an order to a plurality of the aforementioned agents, A behavior prediction step in which, in the nth agent, behavior is predicted based on the state variable corresponding to the nth agent and the reward-related cumulative value provided by the (n-1)th agent, A reward-related value prediction step, which predicts the reward-related value corresponding to the nth agent based on the predicted action, A cumulative value calculation step, which generates a cumulative reward-related value to be provided to the (n+1)th agent by subtracting the reward-related value corresponding to the (n+1)th agent from the sum of the reward-related value corresponding to the nth agent and the cumulative reward-related value provided by the (n-1)th agent, A repeating operation step in which the aforementioned behavior prediction step, the aforementioned reward-related value prediction step, and the aforementioned cumulative value calculation step are applied sequentially to each of the target agents, thereby causing all of the agents to predict their actions; A control program comprising: a control step of controlling each of the control devices based on the actions predicted by each of the agents.

Images