Control device, control method, and program

Abstract

This control device comprises: a transfer learning unit that, in transfer learning using reinforcement learning, uses information about one or a plurality of transfer source models that are used for prescribed control and that have been pre-trained at a transfer source to create a plurality of models by performing transfer learning using a prescribed method for obtaining a plurality of models; and a transfer model selection unit that selects a transfer model to be output to a transfer destination, from among the created plurality of models on the basis of an indicator of at least one of a reward, a loss function, and a learning step.

Description

Control device, control method, and program

[0001] The disclosed technology relates to a control device, a control method, and a program.

[0002] There is a technology for optimizing control using reinforcement learning, which is a technology that makes it possible to select the optimal control (policy).

[0003] For example, there is a technology for optimizing behavior for controlling air conditioning in a target space (see Patent Document 1). This technology uses reinforcement learning in behavioral exploration to optimize behavior by reproducing the environment of the space to be controlled. As described in Patent Document 1, reinforcement learning is known to be effective in optimizing the control of air conditioning equipment and the like.

[0004] Patent No. 7014299

[0005] However, policy decision-making models using reinforcement learning have the problem that convergence is slow, and accuracy does not necessarily increase even as the learning steps progress.

[0006] The disclosed technology has been made in consideration of the above points, and aims to provide a control device, a control method, and a program that can reduce the computational costs associated with learning using transfer learning.

[0007] A first aspect of the present disclosure is a control device that includes: a transfer learning unit that, in transfer learning using reinforcement learning, performs transfer learning using a predetermined method to obtain multiple models using information on one or more source models that are models used for predetermined control and have been trained in advance at a source, and creates the multiple models; and a transfer model selection unit that selects a transfer model to be output to a destination from the multiple created models, based on at least one index of a reward, a loss function, and a learning step.

[0008] A second aspect of the present disclosure is a control method, in which a computer executes a process of performing transfer learning using reinforcement learning by using information on one or more source models that are used for predetermined control and have been trained in advance at the source, performing transfer learning in a predetermined manner to obtain multiple models, creating the multiple models, and selecting a transfer model to be output to the destination from the created multiple models based on at least one index of reward, loss function, and learning step.

[0009] According to the disclosed technology, the computational costs associated with learning can be reduced by using transfer learning.

[0010] FIG. 1 is a schematic diagram showing an example of a system that calculates a scenario by optimal control using reinforcement learning. FIG. 2 is a block diagram showing the configuration of a control system. FIG. 3 is a block diagram showing the hardware configuration of a control device. FIG. 4 is a diagram for explaining the relationship between the control device and a system to be controlled. FIG. 5 is a diagram showing the state of the system to be controlled. FIG. 6 is a flowchart showing the flow of control processing. FIG. 7 is a schematic diagram showing an image of the method of this embodiment.

[0011] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same reference numerals are used to designate identical or equivalent components and parts in each drawing. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0012] First, the premise and overview of the present disclosure will be described. An example of a system that is the premise of the present disclosure will be described. FIG. 1 is a schematic diagram illustrating an example of a system that calculates a scenario through optimal control using reinforcement learning. In this system, processing is performed by an environment reproduction unit and a scenario calculation unit, and a model used in the scenario calculation unit is learned using reinforcement learning. The environment reproduction unit uses people flow data as input and reproduces an environment using an environment reproduction simulator. The people flow data may be, for example, people flow vectors predicted using a prediction model from past people flow data. The reproduced environment provides energy-saving information and comfort information (such as temperature, humidity, and the amount of movement of people). This information obtained from the reproduced environment is output to the scenario calculation unit and used as reward information. The scenario calculation unit uses the reward information to learn a model based on a deep neural network (DNN), and uses the model to calculate a scenario that optimizes behavioral control. For example, a scenario can calculate the temperature by time period for each of multiple cool air outlets in a building's air conditioning system. Examples include "Cool air outlet 1, time zone 9:00, temperature 22.5°C / time zone 9:30, temperature 22.0°C..." and "Cool air outlet 2, time zone 9:00, temperature 24.1°C / time zone 9:30, temperature 24.5°C...".

[0013] The above example can refer to the method of calculating behavior using an optimal control device described in Patent Document 1. By referring to the method described in Patent Document 1, it is possible to acquire environmental data of the real space, reproduce the environment of the target space, and obtain a model that calculates optimal control measures based on the reproduced environment through machine learning (reinforcement learning), thereby selecting optimal behavior. However, when applied to an actual building, learning and inference must be performed with a computational cost lower than the energy that can be reduced by the above method, and this condition may not be met depending on the measured data and the environment.

[0014] Under the above assumptions, the learning cost of a reinforcement learning model is high, so it was necessary to reduce the learning cost.Challenges with reinforcement learning include the slow convergence of learning in a policy decision model using reinforcement learning, the fact that accuracy does not necessarily increase as the learning step progresses, and the fact that in actual operation, it is necessary to use an optimal model obtained within a limited time.

[0015] Therefore, in this embodiment, a transfer learning technique is used in which a base model is selected from a model trained over a finite time period. For example, models are trained for multiple floors in the same building, and a model trained on one floor is transferred to multiple floors. Note that, below, reinforcement learning of an air conditioning control agent is used as an example of reinforcement learning. However, this embodiment can be applied to the control of various facilities and devices in environments that can be reproduced using an environmental reproduction model, not limited to air conditioning equipment. Note that transfer learning using reinforcement learning refers to bringing a model into an environment other than the original environment (an environmental simulator in this embodiment) to speed up learning and improve accuracy. Note that, in this embodiment, a deep reinforcement learning technique is used among reinforcement learning techniques. Hereinafter, even when simply referring to reinforcement learning, deep reinforcement learning is used.

[0016] An example of a transfer learning technique that is assumed to be applied to this embodiment is shown below. The trained model at the transfer source is selected from (1) the center of the largest peak in multiple ranges where the moving average curve of the reward function, which regards the reward as a function of the learning step, is convex upward (spiky), (2) the center of the smallest peak in multiple ranges where the moving average curve of the loss function is convex downward, and (3) the final model over a limited learning period. Learning toward the transfer destination involves providing models (1), (2), and (3) as initial values, and performing (a) full-layer learning or (b) learning only on the output layer as learning means. From the results of up to six transfer learning combinations (1) to (3) and (a) and (b), the trained model and learning means at the transfer source are selected based on the larger moving average curve of the reward function and the smaller moving average of the loss value. Furthermore, environmental simulations are performed as necessary, and if none of the models meet the building criteria, additional learning is performed to further the agent's learning. By learning using the above procedure, it is possible to select the best model within a time limit and to select a suitable model that can operate even during learning. In this embodiment, by utilizing the above transfer learning method, it is possible to reduce computational costs.

[0017] The configuration of this embodiment will be described below. Fig. 2 is a block diagram showing the configuration of a control system. The control system 100 is connected to a control device 110 and a plurality of systems 102 to be controlled. The systems 102 are the systems to be controlled, and serve as a transfer source system (102A) and a transfer destination system (102B) depending on the processing mode of the control device 110. The relationship between the systems 102 and the control device 110 will be described later. Note that the system 102 and the control device 110 will be described as separate systems, but the control device 110 may also be a device that includes the system 102.

[0018] Fig. 3 is a block diagram showing the hardware configuration of the control device 110. As shown in Fig. 3, the control device 110 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0019] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a control program.

[0020] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0021] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0022] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may be a touch panel type and function as the input unit 15.

[0023] The communication interface 17 is an interface for communicating with other devices such as terminals, etc. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0024] Next, a description will be given of each functional configuration of the control device 110 in the control system 100. Each functional configuration of the control device 110 is realized by the CPU 11 reading out a control program stored in the ROM 12 or the storage 14, expanding the program in the RAM 13, and executing the program.

[0025] 1, the control device 110 includes a storage unit 112, a learning model selection unit 120, a transfer learning unit 122, and a transfer model selection unit 124. Each unit of the control device 110 will be described below with reference to its relationship with the system 102.

[0026] 4 is a diagram illustrating the relationship between the control device 110 and the system 102 to be controlled. In the system 102, a system that is the learning source of the transfer model is referred to as a transfer source system 102A, and a system that performs control using the transfer model that has undergone transfer learning is referred to as a transfer destination system 102B. The transfer source system 102A is a system that corresponds to the representative floor of the facility. The transfer destination system 102B is a system that corresponds to each floor of the representative floor.

[0027] The system 102 in FIG. 4 is a basic configuration for explaining the parts necessary for model transfer by the processing of the control device 110. As shown in FIG. 5 , the system 102 is described as including a control learning unit 1020 and a scenario calculation unit 1022. The control learning unit 1020 of the transfer source system 102A performs reinforcement learning of the air conditioning control agent for a predetermined period of time. The reinforcement learning uses environmental data and BEMS data from the environmental DB. When a control scenario created using a model currently being learned is input from the control learning unit 1020 to the environmental simulator, an environmental prediction result is output to the control learning unit 1020. The control model learned by the control learning unit 1020 is stored in the control model DB. The scenario calculation unit 1022 calculates a scenario using a control model in the control model DB. For the transfer destination system 102B, the control device 110 updates the control model DB with the transfer model. The control model is a model used to control equipment in the environment and is a model used to calculate a control scenario in the scenario calculation unit. If no transfer is made, the control model is learned in the transfer destination system 102B, but if a transfer is made, the learning of the control model is substituted by a transfer model obtained by transfer learning.

[0028] It is assumed that normal deep reinforcement learning is being performed in the source system 102A. Therefore, it is assumed that deep reinforcement learning is performed in the control learning unit of the source system 102A, and the learned control model is stored in the control model DB. The control models stored in the control model DB include (1) a unique control model when learning has converged, and (1) multiple model candidates when learning has not converged. Therefore, (1) if learning has converged, the learned control model is input to the control device 110. (2) If learning has not converged, all of the models (model candidates) currently being learned are input to the control device 110.

[0029] The learning model selection unit 120 acquires, from the control models of the transfer source system 102A, the transfer source control models (including models currently being trained), the score of the sum of the rewards and penalties for each of the control models (hereinafter simply referred to as the score), and the loss value (value output by the loss function) for each of the control models as information on the transfer source model (input of a1 in Figure 4).The learning model selection unit 120 selects a model to be used as a learning target for transfer from the transfer models input from the transfer source system 102A, for example, using a predetermined index.The loss values ​​arranged in the order of the learning steps are used as an index for the loss function.

[0030] The learning model selection unit 120 (1) inputs only one control model that has completed learning if the learning has converged, and outputs the input control model as is. Therefore, in this case, the processing of the transfer model selection unit 124 may be omitted. (2) If the learning has not converged, all models currently being learned are input together with their rewards and loss values, and an arbitrary model is selected and output to the transfer learning unit 122. The indicators of the models selected by the learning model selection unit 120 may be, for example, "the model with the largest reward value (the model with the largest score)," "the model with the smallest loss value (the value output by the loss function)," or "the model with the most learning steps within the aborted range." Any number of models, one or more, may be selected. As described above, if the learning of the transfer model is aborted midway, the learning model selection unit 120 can select a transfer model from multiple transfer models currently being learned based on at least one indicator of the reward, loss function, and learning steps.

[0031] The transfer learning unit 122 substitutes for the learning of the control model performed by the control learning unit of the transfer destination system 102B. In processing by the transfer learning unit 122, environmental data (including BEMS data) from the environmental data DB and an environmental simulator are acquired in advance from the transfer destination system 102B and stored in the storage unit 112. Using these, the transfer learning unit 122 performs additional learning using the trained model of the transfer source or a candidate model in the middle of learning as a base model.

[0032] The transfer learning unit 122 performs multiple types of reinforcement learning with the weights of the selected source model transferred (copied to the destination model) to create multiple models. Examples of multiple transfer methods include a method (first method) in which the weights of the source model are set as initial values, all weights are updated, and reinforcement learning is performed. Another method (second method) involves freezing the weights of the source model and not updating the transferred weights. These methods are all applied to each of the selected one or more source models. Therefore, the number of models (transfer model candidates) created is the number of models selected (by the learning model selection unit 120) multiplied by the number of methods applied (by the transfer learning unit 122). As described above, the transfer learning unit 122 applies the first and second methods to the source model to create multiple models.

[0033] The transfer model selection unit 124 selects a transfer model based on the reward and loss value of the transfer model created by the transfer learning unit 122. For example, it selects one "model with the largest reward in transfer learning." Alternatively, it may select one "model with the smallest loss value in transfer learning." Alternatively, it may select one "model with the most learning steps in transfer learning." In other words, the transfer model selection unit 124 selects a transfer model based on any one index of reward, loss function, or learning steps, and outputs it to the transfer destination system 102B. In the transfer destination system 102B, the output transfer model is updated as the control model in the control model DB.

[0034] Next, the operation of the control system 100 will be described. Fig. 6 is a flowchart showing the flow of the control process. In the control device 110, the CPU 11 reads a control program from the ROM 12 or the storage 14, loads it into the RAM 13, and executes it, thereby performing the control process. The process is performed via the system 102 (102A, 102B).

[0035] In step S100, the source system 102A (control learning unit 1020) performs reinforcement learning on the air conditioning control agent for a set period of time, for example, and stores the learned control model in the control DB. Here, the control learning unit 1020 performs reinforcement learning on the air conditioning control agent using the source environmental simulator for a limited period of time, such as four hours. In the reinforcement learning, the score of the sum of the rewards and penalties obtained is stored together with the learning step.

[0036] In step S102, the transfer source system 102A outputs one or more transfer source models stored in the control DB and information on the transfer source models to the control device 110.

[0037] In step S104, the CPU 11, as the learning model selection unit 120, selects a model to be used as a learning target for transfer from among the input transfer models using a predetermined index. An example of the selection method is described below. In this process, for example, based on information about the transfer source model, the scores of each learning step of the reinforcement learning performed by the control learning unit 1020 are tallied, and the learning steps are treated as a time axis to calculate a moving average of the scores, for example, every 1,000 learning steps. Then, the model closest to the maximum step around the maximum value of the calculated moving average of the scores is selected as a candidate for the air conditioning control agent. The model of the last learning step within the limited time is also selected as a candidate for the air conditioning control agent. Note that a loss value may be used instead of a reward for the score.

[0038] In step S106, the CPU 11, as the transfer learning unit 122, performs multiple types of reinforcement learning with the weights of the source model transferred, to create multiple models. The reinforcement learning in this processing step involves creating, for example, two new air conditioning control agents with the selected source model as their initial values, and performing multiple types of reinforcement learning. The first method freezes all layers except the output layer, preventing updates through learning. The second method does not freeze any layers, and updates all layers through learning. In this processing, reinforcement learning is performed using each method.

[0039] In step S108, the CPU 11, as the transfer model selection unit 124, selects a transfer model based on the rewards and loss values ​​of the multiple models created. In this processing step, for example, the scores of each learning step in which multiple models (transfer model candidates) are trained are tallied, and the moving average of the scores is calculated, for example, every 1,000 learning steps, regarding the learning steps as a time axis. Then, the model closest to the maximum step around the maximum value of the moving average of the scores is selected as the transfer model.

[0040] In step S110, the CPU 11 functions as the transfer model selection unit 124 and outputs the selected transfer model to the transfer destination model.

[0041] In step S112, the transfer destination system 102B (scenario calculation unit 1022) uses the input transfer model (control model) to calculate a control scenario for the floor corresponding to the system.

[0042] As described above, according to the control system 100 of this embodiment, it is possible to reduce the computational costs associated with learning by using transfer learning.

[0043] FIG. 7 is a schematic diagram illustrating the concept of the method of this embodiment. In cases where there are multiple spaces (floors) with similar environments, such as in an office building, the best models are selected from among models that have terminated learning in a limited time in any one to a small number of spaces. For example, an office building is expected to have multiple spaces (floors) with similar environments. Considering divisions, a 30-story office building may have 30 floors, and there may be 30 or more source systems. Therefore, when a model is transferred, the learning time on the destination floor can be reduced.

[0044] Furthermore, by transferring to another space using one or more methods, learning can be completed in less time than the total learning time required when learning multiple spaces independently. By training multiple transfer model candidates and selecting the best transfer model for each space, the total computational cost required for learning and inference can be reduced. Here is an example of reducing learning time through transfer. Let the reduction rate be α, and assume that learning can be performed with αT < T for the learning time T for one floor. In this case, learning 30 floors, which previously required 30T, can now be performed with T + 29αT < 30T. The larger α is, the greater the reduction amount, and the lower the average learning cost per floor can be.

[0045] Note that the present invention is not limited to the above-described embodiment, and various modifications can be made. For example, the index in the learning model selection unit 120 is exemplified as a score of the sum of rewards and penalties, but the present invention is not limited to this. For example, instead of a score, the index may be each of the following conditions, and the maximum or minimum value of the corresponding condition may be used as the selection criterion in the learning model selection unit 120. Furthermore, the following combinations may be used as indices. Similarly, the transfer model selection unit 124 may also use each of the following conditions and combinations as indices: -1 times the loss value output by the loss function (-1 times indicates the minimum value) - The correlation coefficient between the outside temperature and the room temperature simulated in the learning step - The correlation coefficient between the humidity in the weather information and the humidity simulated in the learning step - The number of action types selected by the model in the learning step - -1 times the absolute value of the PMV value of the environmental simulator - A value obtained by linearly combining two or more of the above indices with the moving average of the score

[0046] In the above embodiment, the control processing executed by the CPU after reading the software (program) may be executed by various processors other than the CPU. Examples of such processors include a programmable logic device (PLD) (such as a field-programmable gate array (FPGA)), a graphics processing unit (GPU), and an application-specific integrated circuit (ASIC), which are dedicated electrical circuits designed specifically to execute specific processes. The control processing may be executed by one of these processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, etc.). Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0047] In the above embodiment, the program is pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0048] The following additional notes are provided regarding the above-described embodiments.

[0049] (Supplementary Item 1) A control device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to: perform transfer learning using reinforcement learning in a predetermined manner to obtain a plurality of models using information on one or more source models that are models used for predetermined control and that have been trained in advance at the source of transfer, to create the plurality of models; and select a transfer model to be output to the destination of transfer from the plurality of created models based on at least one index of reward, loss function, and learning step.

[0050] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to perform control processing, the non-transitory storage medium comprising: in transfer learning using reinforcement learning, performing transfer learning in a predetermined manner to obtain multiple models using information on one or more source models that are models used for predetermined control and have been trained in advance at the source of transfer, thereby creating multiple models; and selecting a transfer model to be output to the destination of transfer from the multiple created models based on at least one index of reward, loss function, and learning step.

[0051] 100 Control system 102 System 102A Transfer source system 102B Transfer destination system 110 Control device 112 Storage unit 120 Learning model selection unit 122 Transfer learning unit 124 Transfer model selection unit 1020 Control learning unit 1022 Scenario calculation unit

Claims

1. A control device including: a transfer learning unit that performs transfer learning in a predetermined manner to obtain multiple models using information on one or more source models that are models used for predetermined control and that have been trained in advance at a source of transfer in transfer learning using reinforcement learning, and creates multiple models; and a transfer model selection unit that selects a transfer model to be output to a destination of transfer from the multiple models created based on at least one index of reward, loss function, and learning step.

2. The control device of claim 1, further comprising a learning model selection unit, which, when the learning of the source model is terminated midway, selects one or more source models to be subjected to the transfer learning from among a plurality of source models in the middle of learning based on at least one indicator of the reward, the loss function, and the learning step.

3. The control device described in claim 2, wherein the learning model selection unit compiles a score of the sum of the rewards and penalties for each learning as information of the source model as the index related to the reward, and calculates a moving average of the scores by regarding the learning steps as a time axis.

4. The control device according to claim 1, wherein the transfer learning unit, in the transfer learning by the predetermined method, uses as a first method a method in which the weights of the source model are set to initial values, all weights are updated, and reinforcement learning is performed, and as a second method a method in which the weights of the source model are frozen and the transferred weights are not updated, and creates multiple models by applying the first method and the second method to the source model, respectively.

5. A control method in which a computer executes a process of performing transfer learning using reinforcement learning, using information on one or more source models that are used for a specified control and have been trained in advance at the source, performing transfer learning in a specified manner to obtain multiple models, creating multiple models, and selecting a transfer model to be output to the destination from among the multiple models created based on at least one index of reward, loss function, and learning step.

6. A program for causing a computer to function as each part of the control device according to any one of claims 1 to 4.