Control device, control method, program, and control system

Abstract

To provide a control device capable of universally controlling a device to be controlled having a plurality of control patterns, a control method, a program, and a control system.SOLUTION: According to an embodiment, a control device includes an acquisition part, a determination part, and a control part. The acquisition part acquires control state information about a control state of a control object device. The determination part determines a control pattern according to the control state information. The control part controls the control object device on the basis of a reinforcement learning model corresponding to the control pattern.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Embodiments of the present invention relate to a control device, a control method, a program, and a control system. [Background technology]

[0002] Traditionally, reinforcement learning, a type of machine learning that supports the optimized operation of automated control, has been known as a technology that achieves high control accuracy. Reinforcement learning is a method that learns manipulated variables so that the controlled object achieves a better control state. The reinforcement learning model generated by reinforcement learning can appropriately control the controlled device.

[0003] Here, reinforcement learning is a learning method that explores manipulated variables through trial and error. Therefore, when there are multiple control patterns for the device being controlled, reinforcement learning is expected to acquire more sophisticated control by exploring manipulated variables through trial and error for each control pattern. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-187489 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, in reinforcement learning, which explores control parameters through trial and error, the controlled device is at risk of malfunction or damage. Therefore, there is a need for a control device that can universally control devices with multiple control patterns.

[0006] Therefore, the present invention has been made in view of the above circumstances, and aims to provide a control device, a control method, a program, and a control system that enable the general-purpose control of a device having multiple control patterns.

Means for Solving the Problem

[0007] The control device of the embodiment includes an acquisition unit, a determination unit, and a control unit. The acquisition unit acquires control state information regarding the control state of the control target device. The determination unit Past data acquired by the aforementioned acquisition unit the control state information Based on this, a new control state is predicted that includes multiple elements related to the operation of the controlled device, and the deviation between the new control state and the actual control state is calculated. correspondingly hand determines a control pattern. The control unit controls the control target device based on a reinforcement learning model corresponding to the control pattern. Output control parameters

Brief Description of the Drawings

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the outline of a control system according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of the schematic hardware configuration of the control device according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing an example of the schematic functions of the control device according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of the schematic functions of the control learning unit according to the first embodiment. <� [Figure 5] FIG. 5 is a diagram showing an example of the schematic functions of the basic control unit according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of the schematic functions of the control inference unit according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of the schematic functions of the control pattern determination unit according to the first embodiment. [Figure 8] FIG. 8 is a flowchart showing an example of the database creation process executed by the control device according to the first embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of the inference process executed by the control device according to the first embodiment.

Modes for Carrying Out the Invention

[0009] The control device, control method, program, and control system will be described in detail below with reference to the attached drawings. In the following descriptions of each embodiment and modification, parts denoted by the same reference numerals have substantially the same function, and the description of overlapping parts will be omitted as appropriate.

[0010] (First Embodiment) Figure 1 shows a schematic example of a control system 1 according to the first embodiment. The control system 1 comprises a controlled device 10 and a control device 20. The controlled device 10 and the control device 20 are connected in a communicative manner. The control system 1 shown in Figure 1 has one controlled device 10 and one control device 20. However, the control system 1 may have multiple controlled devices 10 and control devices 20.

[0011] The controlled device 10 is the device that is controlled by the control device 20. The controlled device 10 has multiple control patterns. The control pattern is a pattern of the control state of the controlled device 10.

[0012] For example, the controlled device 10 is a servo motor. In this case, the control pattern is the rotation angle of the servo motor. A control pattern is provided for each load state of the servo motor. For example, the load states include high load, medium load, and low load. The controlled device 10 is then controlled by the control pattern corresponding to the high load, medium load, and low load.

[0013] The control device 20 is a device that controls the controlled device 10. For example, the control device 20 is a computer device such as a microcomputer, a personal computer, or a server. The control device 20 acquires control state information that indicates the control state of the controlled device 10. Then, the control device 20 controls the controlled device 10 based on the control state information.

[0014] Next, the hardware configuration of the control device 20 will be described.

[0015] Figure 2 shows a schematic example of the hardware configuration of the control device 20 according to the first embodiment. The control device 20 includes a processor 21, RAM (Random Access Memory) 22, ROM (Read Only Memory) 23, storage device 24, and input / output interface 25.

[0016] The processor 21 comprehensively controls the control unit 20. For example, the processor 21 is a control circuit such as a CPU (Central Processing Unit). RAM 22 is a volatile memory that provides the working area for the processor 21. ROM 23 is a non-volatile memory that stores various programs and parameters. The processor 21 uses RAM 22 as its working area to perform various calculations according to the programs stored in ROM 23 and the storage device 24.

[0017] The storage device 24 is a device that stores information such as flash memory, HDD (Hard Disk Drive), and SSD (Solid State Drive). For example, the storage device 24 stores control programs and reinforcement learning databases. The control program is a program that implements the functions of the storage device 24. The reinforcement learning database has multiple reinforcement learning models, each set up for the control pattern of the controlled device 10.

[0018] The input / output interface 25 is an interface that performs communication with the controlled device 10. For example, the input / output interface 25 is a circuit that performs communication with the controlled device 10.

[0019] Next, the functional configuration of the control device 20 will be described.

[0020] Figure 3 is a schematic example of the functions of the control device 20 according to the first embodiment. The processor 21 of the control device 20 performs various processes using the RAM 22 as a working area, according to programs stored in the ROM 23 and storage device 24, thereby realizing the functions shown in Figure 3. Specifically, the control device 20 includes an input / output control unit 201, an operation selection unit 202, a control learning unit 210, a reinforcement learning model management unit 220, and a control inference unit 230.

[0021] Furthermore, all or part of the functions of the control device 20 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may also be transmitted via a telecommunications line.

[0022] The input / output control unit 201 performs input and output of information to the controlled device 10 by controlling the input / output interface 25. More specifically, the input / output control unit 201 acquires control state information regarding the control state of the controlled device 10. The input / output control unit 201 is an example of an acquisition unit. The control state information includes state information, control command values, control measurement values, and control patterns. The state information indicates the state of the controlled device 10 that changes as the controlled device 10 performs actions based on control by the control device 20. The control command value is information indicating the target value of the control. The control measurement value is information indicating the measurement value measured by the controlled device 10. The control pattern is a pattern indicating the control state of the controlled device 10.

[0023] Furthermore, the input / output control unit 201 outputs an operation quantity that indicates the amount of operation performed on the controlled device 10.

[0024] The operation selection unit 202 determines the recipient of the control state information based on the mode selection signal. The mode selection signal is a signal that selects the operation mode of the control device 20. The operation modes include a learning mode for generating a reinforcement learning model and an inference mode for controlling the controlled device 10 using the reinforcement learning model.

[0025] The operation selection unit 202 operates the control learning unit 210 when the mode selection signal indicates the learning mode. The operation selection unit 202 also inputs control state information to the control learning unit 210. On the other hand, the operation selection unit 202 operates the control inference unit 230 when the mode selection signal indicates the inference mode. The operation selection unit 202 also inputs control state information to the control inference unit 230.

[0026] The control learning unit 210 generates a reinforcement learning model through reinforcement learning. More specifically, the control learning unit 210 includes a reinforcement learning execution unit 211 and a basic control unit 212.

[0027] The reinforcement learning execution unit 211 generates a reinforcement learning model for each of the multiple control patterns of the controlled device 10 by performing reinforcement learning for each control pattern. The reinforcement learning execution unit 211 is an example of a learning unit. When state information, control command values, and control measurement values ​​are input to the reinforcement learning model, it outputs control parameters corresponding to the input state information, control command values, and control measurement values. The reinforcement learning execution unit 211 also outputs the control parameters generated by the reinforcement learning model to the basic control unit 212. The control parameters are parameters that adjust the operating values ​​of the controlled device 10.

[0028] The reinforcement learning execution unit 211 performs reinforcement learning for each control state and control pattern, such as the operating environment and operating conditions. As a result, the reinforcement learning execution unit 211 generates a reinforcement learning model that is suitable for the control state, such as the operating environment and operating conditions.

[0029] The basic control unit 212 controls the device to be controlled 10. For example, the basic control unit 80 controls the device to be controlled 10 using PI (Proportional Integral) control or PID (Proportional Integral Derivative) control.

[0030] The basic control unit 212 is configured with the control parameters output from the reinforcement learning execution unit 211. In other words, the basic control unit 212 is configured with the control parameters that need to be adjusted according to the control state of the controlled device 10. The basic control unit 212 also calculates the manipulated amount according to the control parameters. The basic control unit 212 then outputs the manipulated amount to the input / output control unit 201.

[0031] The input / output control unit 201 outputs the manipulated variable input from the basic control unit 212 to the controlled device 10. As a result, the controlled device 10 performs an action based on the manipulated variable. The controlled device 10 then outputs control status information corresponding to the action to the control device 20.

[0032] Furthermore, the control learning unit 210 calculates a reward based on the control state information output from the controlled device 10. The control learning unit 210 then trains itself to maximize the reward. The control learning unit 210 performs reinforcement learning by repeatedly executing this process. In this way, the control learning unit 210 generates a reinforcement learning model.

[0033] The reinforcement learning model management unit 220 manages the reinforcement learning model. More specifically, when a reinforcement learning model is generated by the control learning unit 210, the reinforcement learning model management unit 220 obtains the state information, control command values, and control measurement values ​​used to generate the reinforcement learning model from the control state information. Then, the reinforcement learning model management unit 220 associates the reinforcement learning model generated by the reinforcement learning execution unit 211 with the control pattern and stores it in the reinforcement learning model database. The reinforcement learning model management unit 220 is an example of a memory control unit.

[0034] Furthermore, when the reinforcement learning model management unit 220 receives a request for a reinforcement learning model from the control inference unit 230, it retrieves a reinforcement learning model from the reinforcement learning model database that corresponds to the control pattern included in the request. The reinforcement learning model management unit 220 then outputs the reinforcement learning model to the control inference unit 230.

[0035] The control inference unit 230 controls the controlled device 10 based on a reinforcement learning model. In other words, the control inference unit 230 infers a control variable appropriate to the control state of the controlled device 10. More specifically, the control inference unit 230 includes a control pattern determination unit 231, a reinforcement learning inference unit 232, and a basic control unit 233.

[0036] The control pattern determination unit 231 determines a control pattern according to the state information, control command value, and control measurement value included in the control state information. The control pattern determination unit 231 is an example of a determination unit. More specifically, the control pattern determination unit 231 predicts the operation content of the controlled device 10. Furthermore, based on the predicted operation content, the control pattern determination unit 231 determines one control pattern from multiple control patterns of the controlled device 10. In other words, the control pattern determination unit 231 predicts a control pattern suitable for the current step.

[0037] The reinforcement learning inference unit 232 outputs control parameters corresponding to the control pattern. More specifically, the reinforcement learning inference unit 232 requests a reinforcement learning model corresponding to the control pattern predicted by the control pattern determination unit 231 from the reinforcement learning model management unit 220. The reinforcement learning inference unit 232 also obtains control parameters by inputting control state information into the reinforcement learning model obtained from the reinforcement learning model management unit 220. Finally, the reinforcement learning inference unit 232 outputs the obtained control parameters to the basic control unit 233.

[0038] The basic control unit 233 has the same functions as the basic control unit 212 of the control learning unit 210. That is, the basic control unit 233 controls the controlled device 10 based on a reinforcement learning model corresponding to the control pattern determined by the control pattern determination unit 231. The basic control unit 233 is an example of a control unit. The basic control unit 233 sets the control parameters output from the reinforcement learning inference unit 232. That is, the basic control unit 233 controls the controlled device 10 based on a reinforcement learning model that outputs control parameters to adjust the operating values ​​of the controlled device 10.

[0039] Furthermore, the basic control unit 233 calculates manipulated variables according to the control parameters. That is, the basic control unit 233 controls the controlled device 10 based on a reinforcement learning model that outputs control parameters when control command values ​​and control measurement values ​​included in the control state information are input. The basic control unit 233 then outputs manipulated variables to the controlled device 10 via the input / output control unit 201.

[0040] Next, the control learning unit 210 will be described in more detail.

[0041] Figure 4 shows a schematic example of the functions of the control learning unit 210 according to the first embodiment. The reinforcement learning execution unit 211 includes a reinforcement learning unit 2111, a memory control unit 2112, and a reward calculation unit 2113.

[0042] The reinforcement learning unit 2111 performs reinforcement learning for each control pattern. When state information, control command values, and control measurement values ​​are input, the reinforcement learning unit 2111 performs reinforcement learning that outputs control parameters corresponding to the state information, control command values, and control measurement values. In other words, the reinforcement learning unit 2111 learns the control parameters in a way that maximizes the reward.

[0043] The memory control unit 2112 stores the data output during reinforcement learning in the memory device 24 for later use. The memory control unit 2112 also retrieves data from the memory device 24 during reinforcement learning.

[0044] The reward calculation unit 2113 calculates the reward in reinforcement learning. For example, the reward calculation unit 2113 calculates the reward based on state information, control command values, and control measurement values in reinforcement learning. Further, the reward calculation unit 2113 may calculate the reward including the operation amount calculated by the basic control unit 212.

[0045] Here, the reinforcement learning by the control learning unit 210 will be described in more detail.

[0046] For example, the reinforcement learning unit 2111 executes reinforcement learning by a policy gradient method which is a kind of direct policy search method (Direct Policy Search: DPS). However, the reinforcement learning executed by the reinforcement learning unit 2111 is not limited to the policy gradient method, and other search algorithms may be used.

[0047] The policy gradient method is a learning method for expressing the action probability of a reinforcement learning agent by a neural network. To search for a policy, three elements of state, action, and reward are used. The state at discrete time step t is s t , the action is a t , and the stochastic policy function is π(a t | s t ). The reinforcement learning unit 2111 observes the state s t and executes the action a t , and as a result of the transition, receives a reward. The stochastic policy function π(a t | s t ) that characterizes the stochastic policy in action selection represents the probability of selecting the action a t in the state s t . That is, the reinforcement learning unit 2111 executes the action a t according to the policy probability π corresponding to the state s t . Thereby, according to the stochastic policy function π(a t | s t ), a transition is made from the state s t to the state s t+1 .

[0048] The reward is the destination state s t+1 Based on the control command value and control measurement value of step t+1 included in, or other state information or manipulated variables, the reward calculation unit 2113 calculates the policy probability π(a t |s t Information about ) is not provided in advance. The purpose of reinforcement learning is to learn the policy probability π in order to achieve value maximization. That is, the reinforcement learning unit 2111 learns the evaluation value V based on the total discounted reward, as shown in equation (1) below. π Learn the stochastic policy function π that maximizes (s).

[0049]

number

[0050] Here, γ is the discount rate (where 0 ≤ γ ≤ 1), and it indicates the importance of the future reward. E[] represents the expected value.

[0051] Here, TD_err is calculated as the enhancement signal. TD_err is an evaluation value V based on state transitions. π This shows the change in (s). In the learning of the reinforcement learning unit 2111, TD_err is used as the action evaluation in the stochastic policy function π(a t |s t This improves the state s. In other words, when TD_err is positive, the reinforcement learning unit 2111 considers that it has transitioned to a good state, so state s t This increases the probability of selecting action at in state s. Conversely, when TD_err is negative, the reinforcement learning unit 2111 checks state s t Reduce the probability of selecting action at in this context.

[0052] The stochastic policy function π(a) characterizes the action selection rate. t |s t The stochastic policy function π(a) is expressed using the policy parameter vector θ. t |s t ) is expressed including the policy parameter vector θ.

[0053] The reinforcement learning unit 2111 changes the action selection probability by adjusting the policy parameter vector θ. For example, the reinforcement learning unit 2111 updates the policy parameter θ to maximize the objective function, as shown in equation (2) below. η is the learning rate, and η > 0.

[0054]

number

[0055] As a result, the reinforcement learning unit 2111 determines the control parameters of the basic control unit 80 suitable for step t+1 based on the improved stochastic policy function π obtained through learning, for the state s of step π.

[0056] The memory control unit 2112 processes the data {s} acquired by the reinforcement learning unit 2111 through reinforcement learning. t ,a t ,r t ,s t+1 The} are stored in the replay buffer as action transitions. Then, the reinforcement learning unit 2111 repeatedly randomly samples the data stored in the replay buffer and uses it during learning.

[0057] The reinforcement learning unit 2111 outputs a reinforcement learning model once it has finished reinforcement learning according to the control pattern. The reinforcement learning model management unit 220 then stores the control pattern as an index in the reinforcement learning model database.

[0058] Next, the basic control unit 212 will be described in more detail.

[0059] Figure 5 shows a schematic example of the functions of the basic control unit 212 according to the first embodiment. The basic control unit 212 includes a parameter setting unit 2121 and a basic control calculation unit 2122.

[0060] The parameter setting unit 2121 sets the basic control calculation unit 2122 using the control parameters calculated by the reinforcement learning execution unit 211. That is, when the basic control calculation unit 2122 performs PID control, the parameter setting unit 2121 sets the control parameters {K} calculated by the reinforcement learning execution unit 211. P , K I , K D Set} to the basic control calculation unit 2122.

[0061] The basic control calculation unit 2122 calculates the manipulated variable of the controlled device 10. More specifically, once the parameter setting unit 2121 has completed setting the control parameters, the basic control calculation unit 2122 calculates the manipulated variable according to the control parameters based on the control command value and the control measurement value. For example, the basic control calculation unit 2122 calculates the manipulated variable ops using the following formula (3).

[0062]

number

[0063] The basic control calculation unit 2122 outputs the calculated manipulated variable to the controlled device 10 via the input / output control unit 201.

[0064] Next, the control reasoning unit 230 will be described in more detail.

[0065] Figure 6 shows a schematic example of the functions of the control inference unit 230 according to the first embodiment. The control inference unit 230 includes a control pattern determination unit 231, a reinforcement learning inference unit 232, and a basic control unit 233.

[0066] The control pattern determination unit 231 determines the control pattern of the controlled device 10. The control pattern determination unit 231 determines the control pattern of the controlled device 10 from a set of multiple control patterns, a set of past operation details M of the controlled device 10, and the operation details M of a certain step of the controlled device 10. T The operation details of the step immediately preceding a certain step of the controlled device 10 M T-1Based on this, the control pattern determination unit 231 predicts the operation content of the controlled device 10 for each of the multiple control patterns. Based on the predicted operation content, the control pattern determination unit 231 determines one control pattern from the multiple control patterns. The control pattern determination unit 231 then outputs the predicted control pattern to the reinforcement learning model management unit 220.

[0067] Here, "operation details" refers to the state of the controlled device 10 controlled by the control device 20. In other words, "operation details" refers to the state of the controlled device 10 indicated by the control state information. Furthermore, "steps" refer to, for example, the outputs of each manipulated variable in the output flow of manipulated variables sequentially output by the basic control unit 233.

[0068] The reinforcement learning model management unit 220 retrieves a reinforcement learning model corresponding to the control pattern input from the control pattern determination unit 231 from the reinforcement learning model database. The reinforcement learning model management unit 220 then outputs the retrieved reinforcement learning model to the reinforcement learning inference unit 232.

[0069] The reinforcement learning inference unit 232 outputs control parameters using the reinforcement learning model output from the reinforcement learning model management unit 220. More specifically, the reinforcement learning inference unit 232 controls the controlled device 10 based on the reinforcement learning model that outputs control parameters to adjust the operating values ​​of the controlled device 10. That is, when the reinforcement learning inference unit 232 inputs state information, control command values, and control measurement values ​​into the reinforcement learning model, it outputs the control parameters output from the reinforcement learning model to the basic control unit 233.

[0070] The basic control unit 233 has the same functions as the basic control unit 212 of the control learning unit 210. That is, when the control parameter setting by the parameter setting unit 2331 is completed, the basic control calculation unit 2332 calculates an manipulated variable according to the control parameter based on the control command value and the control measurement value. The basic control calculation unit 2332 then outputs the calculated manipulated variable to the controlled device 10 via the input / output control unit 201.

[0071] Next, the control pattern determination unit 231 will be described in more detail.

[0072] Figure 7 shows a schematic example of the functions of the control pattern determination unit 231 according to the first embodiment. The control pattern determination unit 231 includes a control target motion prediction unit 2311, a control target motion deviation calculation unit 2312, and a control pattern selection unit 2313.

[0073] The control target operation prediction unit 2311 predicts multiple control patterns and the operation content M' of the T-1 step of the control target device 10. T-1 Based on the set M of past operations of the controlled device 10, the operation content of step T {M1'} is determined for each control pattern. T ,M2' T ,…,MN' T It predicts the}. The controlled object motion prediction unit 2311 is an example of a prediction unit.

[0074] For example, the control target motion prediction unit 2311 predicts the motion content of step T {M1'} for each control pattern based on the difference in changes between past motion content and the motion content of the previous step using a linear prediction. T ,M2' T ,…,MN' T This predicts the}. Note that the control target motion prediction unit 2311 may predict using linear or nonlinear functions, not limited to this method.

[0075]

number

[0076] In formula (4), the deviation in the actions between step T-1 and step T is predicted based on the deviation in the actions between step T-2 and step T-1. M1' T-2 This is included in the set M of past operating details of the controlled device 10. a is a hyperparameter.

[0077] The control target operation deviation calculation unit 2312 calculates the actual operation content M of step T-1 of the control target device 10. TThe operation content M' predicted by the control target operation prediction unit 2311. T The deviation is calculated. The controlled object operation deviation calculation unit 2312 is an example of a calculation unit. Also, if there are multiple elements in the operation content M (for example, the control pattern is predicted by multiple elements such as the current, voltage, and speed of a servo motor), the set M1 is a set M1 = {O11, O12, ..., O1} composed of m elements. m} becomes M1' ERR This is the sum of the standard scores corresponding to the weight of each element.

[0078]

number

[0079] Weight 1 in formula (5) i This is element O1 of operation content M1. i These are weighting coefficients that fit the criteria.

[0080] The control pattern selection unit 2313 selects one control pattern from n types of control patterns based on the deviation calculated by the controlled object operation deviation calculation unit 2312. The control pattern selection unit 2313 is an example of a selection unit. For example, the control pattern selection unit 2313 is {M1 ERR ,M2 ERR ,…,MN ERR Select the control pattern with the smallest deviation within the given range.

[0081]

number

[0082] Next, various processes performed by the control device 20 according to the first embodiment will be described.

[0083] Figure 8 is a flowchart showing an example of a database creation process performed by the control device 20 according to the first embodiment. The database creation process is the process of creating a database that has multiple combinations of control patterns and reinforcement learning models.

[0084] The input / output control unit 201 receives input of multiple control patterns from the controlled device 10 (step S1). As a result, the input / output control unit 201 prepares multiple control patterns.

[0085] The control learning unit 210 selects one control pattern from multiple control patterns (step S2).

[0086] The control learning unit 210 performs reinforcement learning to output control parameters corresponding to the control state information when control state information for the selected control pattern is input (step S3). More specifically, the reinforcement learning execution unit 211 generates control parameters. The basic control unit 212 calculates the manipulated variable after setting the generated control parameters. The basic control unit 212 also outputs the calculated manipulated variable to the controlled device 10 via the input / output control unit 201. When an operation is executed based on the manipulated variable output to the controlled device 10, the control learning unit 210 acquires control state information via the input / output control unit 201. The control learning unit 210 then calculates a reward according to the control state information. In the repetition of this process, the control learning unit 210 performs reinforcement learning to output control parameters that maximize the reward.

[0087] The control learning unit 210 determines whether or not reinforcement learning is complete (step S4). If reinforcement learning is not complete (step S4; No), the control learning unit 210 performs reinforcement learning in step S3.

[0088] If reinforcement learning is completed (Step S4; Yes), the control learning unit 210 generates a reinforcement learning model (Step S5). The reinforcement learning model management unit 220 stores the reinforcement learning model generated by the control learning unit 210 in the reinforcement learning model database (Step S6). Specifically, the reinforcement learning model management unit 220 stores the neural network configuration of the reinforcement learning model and the weight coefficients of the neural network in the reinforcement learning model database.

[0089] The control learning unit 210 determines whether the generation of reinforcement learning models corresponding to each of the multiple control patterns has been completed (step S7). If the generation of all reinforcement learning models has not been completed (step S7; No), the control learning unit 210 proceeds to step S2. The control learning unit 210 then executes processing corresponding to the control patterns for which reinforcement learning models have not yet been generated.

[0090] If the generation of all reinforcement learning models is complete (Step S7; Yes), the control learning unit 210 determines whether or not a change has occurred in the control state of the controlled device 10, such as the operating environment or operating conditions (Step S8). If a change has occurred (Step S8; Yes), the control learning unit 210 proceeds to Step S1 and generates a reinforcement learning model suitable for the control state, such as the operating environment or operating conditions, after the change.

[0091] If no change occurs (Step S8; No), the control learning unit 210 terminates the database creation process.

[0092] Figure 9 is a flowchart showing an example of an inference process performed by the control device 20 according to the first embodiment. The inference process is a process that calculates the manipulated variable using a reinforcement learning model stored in a reinforcement learning model database.

[0093] The input / output control unit 201 acquires control status information of the controlled device 10 (step S11).

[0094] The control pattern determination unit 231 determines a set of multiple control patterns, a set of past operation details M of the controlled device 10, and the operation details M of a certain step of the controlled device 10. T The operation details of the step immediately preceding a certain step of the controlled device 10 M T-1 Based on this, the operation of the controlled device 10 is predicted for each of the multiple control patterns (step S12).

[0095] The controlled object operation deviation calculation unit 2312 calculates the deviation between the operation content of the controlled device 10 and the predicted operation content (step S13).

[0096] The control pattern selection unit 2313 determines the control pattern based on the calculated deviation (step S14).

[0097] The reinforcement learning model management unit 220 retrieves a reinforcement learning model corresponding to the determined control pattern from the reinforcement learning model database (step S15).

[0098] The reinforcement learning inference unit 232 sets the acquired control parameters in the basic control unit 233 by inputting control state information to the acquired reinforcement learning model (step S16).

[0099] The basic control unit 233 calculates the manipulated variable (step S17).

[0100] The basic control unit 233 outputs the calculated manipulated variable to the controlled device 10 via the input / output control unit 201 (step S18).

[0101] The control inference unit 230 terminates the database creation process.

[0102] As described above, the control device 20 according to the first embodiment determines one control pattern from among the multiple control patterns of the controlled device 10. The control device 20 also sets control parameters based on the reinforcement learning model generated for each control pattern. The control device 20 then controls the controlled device 10 according to the control parameters. In this way, the control device 20 has a reinforcement learning model for each of the multiple control patterns of the controlled device 10, and controls the controlled device 10 using the reinforcement learning model corresponding to the control pattern. Therefore, the control device 20 can universally control a controlled device having multiple control patterns.

[0103] Furthermore, the program executed by the control device 20 of this embodiment is provided as an installable or executable file, recorded on a computer-readable recording medium such as a DVD (Digital Versatile Disk), USB (Universal Serial Bus) memory, or SSD (Solid State Drive).

[0104] Furthermore, the program may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the program may be configured to be provided or distributed via a network such as the Internet.

[0105] Alternatively, the program may be configured to be pre-installed in ROM or the like before being provided. [Explanation of Symbols]

[0106] 1...Control system, 10...Controlled device, 20...Control device, 21...Processor, 22...RAM (Random Access Memory), 23...ROM (Read Only Memory), 24...Storage device, 25...Input / Output interface, 201...Input / Output control unit, 202...Motion selection unit, 210...Control learning unit, 220...Reinforcement learning model management unit, 230...Control inference unit, 211...Reinforcement learning execution unit, 212, 233...Basic control unit, 231...Control pattern determination unit, 232...Reinforcement learning inference unit, 2111...Reinforcement learning unit, 2112...Memory control unit, 2113...Reward calculation unit, 2121, 2331...Parameter setting unit, 2122, 2332...Basic control calculation unit, 2311...Controlled device motion prediction unit, 2312...Controlled device motion deviation calculation unit, 2313...Control pattern selection unit.

Claims

1. An acquisition unit that acquires control state information regarding the control state of the controlled device, A determination unit predicts a new control state, which includes multiple elements relating to the operation of the controlled device, based on the past control state information acquired by the acquisition unit, and determines a control pattern according to the deviation between the new control state and the actual control state. A control unit that outputs control parameters based on a reinforcement learning model corresponding to the control pattern and controls the controlled device, A control device equipped with the following features.

2. The aforementioned determination unit, A prediction unit that predicts the operation content for each control pattern of the controlled device, A calculation unit calculates the deviation between the operation content for each control pattern predicted by the prediction unit and the operation content of the controlled device, The system includes a selection unit that selects the control pattern based on the deviation calculated by the calculation unit, The control device according to claim 1.

3. The control unit controls the controlled device based on the reinforcement learning model that outputs the control parameters for adjusting the operating values ​​of the controlled device in accordance with the control pattern determined by the determination unit. The control device according to claim 1.

4. The control unit controls the controlled device based on the reinforcement learning model that outputs the control parameters when the control state information is input. The control device according to claim 1.

5. A learning unit that performs reinforcement learning for each of the multiple control patterns of the controlled device, The system further comprises a memory control unit that stores the reinforcement learning model generated by the reinforcement learning performed by the learning unit in a memory unit. A control device according to any one of claims 1 to 4.

6. The learning unit performs reinforcement learning for each control state of the controlled device and each control pattern determined by the decision unit. The control device according to claim 5.

7. Obtain control status information regarding the control status of the controlled device, Based on the acquired past control state information, a new control state including multiple elements related to the operation of the controlled device is predicted, and a control pattern is determined according to the deviation between the new control state and the actual control state. Based on the reinforcement learning model corresponding to the control pattern, control parameters are output, and the controlled device is controlled. A control method that includes the following.

8. Computers, An acquisition unit that acquires control state information regarding the control state of the controlled device, A determination unit predicts a new control state, which includes multiple elements relating to the operation of the controlled device, based on the past control state information acquired by the acquisition unit, and determines a control pattern according to the deviation between the new control state and the actual control state. A control unit that outputs control parameters based on a reinforcement learning model corresponding to the control pattern and controls the controlled device, A program to make it work.

9. A control system comprising a controlled device having multiple control patterns and a control device that controls the controlled device, The control device is An acquisition unit that acquires control state information regarding the control state of the controlled device, A determination unit predicts a new control state, which includes multiple elements relating to the operation of the controlled device, based on the past control state information acquired by the acquisition unit, and determines a control pattern according to the deviation between the new control state and the actual control state. A control unit that outputs control parameters based on a reinforcement learning model corresponding to the control pattern and controls the controlled device, A control system equipped with the following features.

Images