Method and system for adapting PID coefficients through reinforcement learning

Abstract

The system and method are used to adapt the coefficients of a proportional-integral-derivative (PID) controller through reinforcement learning. The approach for adapting the PID coefficients can include an outer loop of reinforcement learning where the PID coefficients are adjusted to changes in the environment, and an inner loop of PID control to quickly react to changing errors. The outer loop can learn and adapt to changes in the environment and can be set to run only at a predetermined frequency. The outer loop can use summary statistics about the error terms and any other information sensed about the environment to calculate an observation. The observation can be used to evaluate the next action, for example by feeding this to a neural network representing the policy. The resulting action is the coefficients of the PID controller and tunable parameters such as filters.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION Embodiments of the present invention generally relate to machine learning systems and methods, and more particularly, to systems and methods for adapting the coefficients of a proportional-integral-derivative (PID) controller using machine learning. [Background technology]

[0002] (Discussion of Prior Art and Related Information) While the following background information is expected to further educate the reader about additional aspects of the prior art, it may represent examples (e.g., without limitation, approaches, facts, or common knowledge) of certain aspects of the prior art, and should not be construed as limiting the invention or any embodiment thereof to anything stated or implied in or inferred from the text.

[0003] A PID controller is a control mechanism used to achieve a specific control value given a certain value of the current error. PID controllers have many parameters (coefficients) that need to be adjusted for specific situations. PID controllers are often combined with filters for the desired value, such as low-pass filters. These filters also have parameters that need to be adjusted, such as the pole of the low-pass filter. Guidelines exist for adjusting these parameters, but they often require fine tuning, either manually or through automatic tuning. Additionally, these parameters are specific to the conditions under which they are operating and will need to be readjusted if those conditions change.

[0004] Figure 1 illustrates a schematic diagram of a typical PID controller. The PID controller 100 continuously calculates an error value, which is the difference between a desired setpoint (SP) and a measured process variable (PV), and applies a correction based on proportional, integral, and derivative terms. The controller attempts to minimize the error over time by adjusting a controlled variable, such as a process temperature or reagent control, to a new value determined by a weighted sum of the control terms.

[0005] The term P is proportional to the current value of the SP-SV error e(t). For example, if the error is large and positive, the control output will also be proportionally large and positive, taking into account the gain factor K. When proportional control is used alone, the error generates a proportional response, resulting in an error between the setpoint and the actual process value. Without error, there is no corrective response.

[0006] The term I corresponds to the value of the past SP-PV error, which is integrated over time to produce the I term. For example, if there is a residual SP-PV error after applying proportional control, the integral term attempts to remove the residual error by adding a control effect due to the cumulative value of the past error. Once the error is removed, the integral term stops increasing. As a result, as the error gets smaller, the proportional effect decreases, but this is offset by an increasing integral effect.

[0007] Term D is a best estimate of the future trend of the SV-PV error and is based on the current rate of change of the SV-PV error. This is sometimes called anticipatory control because it effectively attempts to reduce the effect of the SP-PV error by creating a control influence that is driven by the rate of change of the error. The more rapid the change, the greater the control or damping effect.

[0008] Loop tuning can be performed to generate an optimal control function. The tuning constant, denoted below as K, must be derived for each control application because it depends on the response characteristics of the complete loop external to the controller.

[0009] From the above, it is clear that there is a need for an algorithm that can use machine learning to efficiently adapt PID coefficients. Summary of the Invention

[0010] An alternative approach to solving control problems is to treat them as learning problems. Reinforcement learning (RL) models the problem as a continuous decision making problem and learning about the long-term effects of actions. To handle complex problems, RL approaches typically use function approximations, which help generalize and speed up learning. In particular, neural networks are often used as function approximators. While RL approaches can learn to control complex systems in a variety of scenarios, these function approximators are often expensive to compute. Especially in systems with limited computational resources, these function approximators can be impossible to evaluate in real time.

[0011] On the other hand, a PID controller can provide rapid changes and be computationally easy. Aspects of the present invention combine the strengths of both the approach of using a PID controller that operates in real time and the RL approach that adapts the PID coefficients to changing conditions. Among other things, the RL step can calculate the PID and filter coefficients. The RL step can operate at a low frequency, reducing computational cost and resources. Thus, the PID step can operate at a high frequency to maintain control.

[0012] An embodiment of the present invention provides a reinforcement learning algorithm for automatically adjusting proportional-integral-derivative (PID) coefficients, and an algorithm that performs the steps of operating a PID controller at a first frequency and operating a reinforcement learning model at a second frequency to minimize an error between a variable setpoint and a process output. The reinforcement learning model performs the steps of receiving sensed information on an environment for calculating summary statistics and observations about an error term from the PID controller, selecting an action including varying the PID coefficients based on the observations by applying the observations to a policy, and updating the policy with a reward function derived from the sensed information after the policy operates on the PID controller by updating the PID coefficients.

[0013] An embodiment of the present invention further provides a method for automatically adjusting coefficients of a proportional-integral-derivative controller, comprising operating a reinforcement model to perform the steps of receiving summary statistics for an error term from a PID controller and receiving sensed information on the environment for calculating an observation, selecting an action based on the observation by applying the observation to a policy, the action including varying the PID coefficients, and updating the policy with a reward function derived from the sensed information after the policy operates on the PID controller by updating the PID coefficients.

[0014] An embodiment of the present invention also provides a method for automatically adjusting coefficients of a proportional-integral-derivative controller, comprising operating a PID controller at each time step to minimize the error between a variable setpoint and a process output, and operating a reinforcement learning model after multiple time steps. The reinforcement learning model performs the following steps: receiving summary statistics about the error term from the PID controller, receiving sensed information on the environment to calculate observations, applying the observations to a policy to select an action, including varying the PID coefficients based on the observations, and updating the policy with a reward function derived from the sensed information after the policy operates on the PID controller.

[0015] These and other features, aspects, and advantages of the present invention will become better understood from the following figures, description, and claims. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram of a conventional PID controller. [Figure 2] 1 is a diagrammatic representation of a control system of a preferred embodiment of the present invention. [Figure 3] 1 is a flow chart illustrating the method of a preferred embodiment of the present invention.Unless otherwise indicated, illustrations shown in the figures are not necessarily to scale. DETAILED DESCRIPTION OF THE INVENTION

[0017] Several embodiments of the present invention are illustrated by way of example, not limitation, in the accompanying drawings. Like reference numerals indicate similar elements. The present invention and its various embodiments may be better understood from the following detailed description in which illustrated embodiments are described. It is to be clearly understood that the illustrated embodiments are set forth by way of example only and do not limit the invention as ultimately defined in the claims. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" are intended to include the plural as well as the singular. It will be understood that the terms "comprise" and / or "comprising," when used herein, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0018] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention pertains. Terms as defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with the meaning in the context of the relevant art and this disclosure, and should not be interpreted in an idealized or overly formal sense unless specifically defined herein.

[0019] In the description of the invention, it will be understood that several techniques and steps are disclosed, each of which has individual advantages and each of which can also be used in conjunction with one or more, or in some cases all, of the other disclosed techniques. Therefore, for the sake of clarity, this description will refrain from repeating every possible combination of each step in an unnecessary manner. Nevertheless, the specification and claims should be read with the understanding that such combinations are fully within the scope of the invention and claims.

[0020] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details.

[0021] The present disclosure is to be considered as an exemplification of the invention and is not intended to limit the invention to the specific embodiments illustrated in the drawings or in the following description.

[0022] A "computer" or "computing device" may refer to one or more machines and / or one or more systems capable of receiving structured input, processing the structured input according to prescribed rules, and producing the results of the processing as output. Examples of computers or computing devices may include computers, stationary and / or portable computers, computers with single processors, multiprocessors, or multicore processors that may operate in parallel and / or non-parallel, general purpose computers, supercomputers, mainframes, superminicomputers, minicomputers, workstations, microcomputers, servers, clients, interactive televisions, web appliances, telecommunications equipment with internet access, hybrid combinations of computers and interactive televisions, portable computers, tablet personal computers (PCs), personal digital assistants (PDAs), portable telephones, application-specific hardware that mimics a computer and / or software such as a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific instruction-set processor (ASIP), a chip, multiple chips, a system-on-chip, or a chipset, and generally, any device that can accept data, process data output, storage, computing power, logic, and control units.

[0023] "Software" or "application" may refer to prescribed rules for operating a computer. Examples of software or applications may include one or more computer-readable language code segments, graphical or textual instructions, applets, pre-compiled code, interpreted code, compiled code, and computer programs.

[0024] The example embodiments described herein may be implemented in an operating environment (e.g., software) that includes computer-executable instructions installed on a computer, in hardware, or in a combination of software and hardware. The computer-executable instructions may be written in a computer programming language or embodied in firmware logic. When written in a programming language conforming to recognized standards, such instructions may be executed on a variety of hardware platforms and for interfacing with a variety of operating systems. Software program code for carrying out operations for aspects of the present invention may be written in any combination of one or more suitable programming languages, including, but not limited to, object-oriented programming languages, and / or conventional procedural languages, and / or programming languages ​​such as, for example, Hypertext Markup Language (HTML), Dynamic HTML, Extensible Markup Language (XML), Extensible Stylesheet Language (XSL), Document Style Semantics and Specification Language (DSSSL), Cascading Style Sheets (CSS), Synchronized Multimedia Integration Language (SMIL), Wireless Markup Language (WML), Java™, Jini™, C, C++, Smalltalk, Python, Perl, UNIX Shell, Visual Basic or Visual Basic Script, Virtual Reality Markup Language (VRML), ColdFusion™, or other compilers, assemblers, interpreters, or other computer languages ​​or platforms.

[0025] Computer program code for performing operations for aspects of the present invention can be written in any combination of one or more programming languages, including object-oriented languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural languages ​​such as the "C" programming language or similar programming languages. The program code may execute entirely on the user computer, partially on the user computer as a standalone software package, partially on the user computer and a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection can be made to an external computer (e.g., through the Internet using an Internet Service Provider). The program code can also be distributed among multiple computing units, each unit handling a portion of the overall computation.

[0026] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to create a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing device, create means for implementing the function / acts specified in the block or blocks of the flowchart and / or block diagrams.

[0027] These computer program instructions may further be stored on a computer-readable medium capable of directing a computer, other programmable data processing apparatus, or other device to function in a particular manner, such that the instructions stored on the computer-readable medium produce an item of manufacture including instructions that implement the function / acts identified in a block or blocks of the flowcharts and / or block diagrams.

[0028] Additionally, although process steps, method steps, algorithms, etc. may be described in a sequential order, such processes, methods, and algorithms may be configured to work in other orders. In other words, any order or sequence of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. Steps of processes described herein may be performed in any order that is practical. Additionally, some steps may be performed simultaneously.

[0029] It will be readily apparent that the various methods and algorithms described herein may be implemented, for example, by appropriately programmed general-purpose computers and other computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from a memory or similar device and executes those instructions, whereby process execution is defined by those instructions. Furthermore, programs implementing such methods and algorithms may be stored and transmitted using a variety of known media.

[0030] The term "computer-readable medium," as used herein, refers to any medium that participates in providing data (e.g., instructions) which may be read by a computer, a processor, or a similar device. Such media may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include dynamic random access memory (DRAM), which typically constitutes main memory. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise a system bus coupled to a processor. Transmission media may include or transmit acoustic waves, light waves, and electromagnetic radiation, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, a DVD, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, a RAM, a PROM, an EPROM, a FLASHEEPROM, any other memory chip or cartridge, a carrier wave as described below, or any other medium from which a computer can read.

[0031] Various forms of computer-readable media may be involved in carrying the sequence of instructions to a processor. For example, the sequence of instructions may (i) be delivered to the processor from RAM, (ii) be carried over a wireless transmission medium, or (iii) be arranged in any of a number of formats, standards, or protocols, such as Bluetooth, TDMA, CDMA, 3G, 4G, 5G, etc.

[0032] Embodiments of the present invention may include apparatus for performing the operations disclosed herein. The apparatus may be specially constructed for the desired purposes, or it may include a device selectively activated or reconfigured by a program stored in the device.

[0033] Unless expressly stated otherwise, and as will be apparent from the description and claims that follow, throughout this specification, descriptions utilizing terms such as "processing," "operating," "calculating," "determining," and the like should be understood to refer to actions and / or the processor of a computer or computing system, or similar electronic computing device that manipulates and / or transforms data physically represented as electronic, registers, and / or memory quantities of a computing system into other data similarly represented as other physical quantities, such as memory, registers, or information storage, transmission, or display devices of the computing system.

[0034] In a similar manner, the term "processor" may refer to any device or part of a device that processes electronic data to convert this electronic data from registers and / or memory into other electronic data that can be stored in registers and / or memory or transmitted to an external device to cause a physical change or the operation of an external device.

[0035] Algorithms, or more generally, are considered to be any self-consistent sequence of acts or operations producing a desired result. These involve physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, properties, terms, numbers, or the like. It should be understood, however, that all of these and similar terms are to be associated with the appropriate physical quantities and merely convenient labels applied to these quantities.

[0036] Broadly speaking, embodiments of the present invention provide systems and methods for adapting the coefficients of a proportional-integral-derivative (PID) controller through reinforcement learning. The approach for adapting the PID coefficients can include a reinforcement learning outer loop, where the PID coefficients are adjusted to changes in the environment, and a PID control inner loop for rapid response to changes in error. After a predetermined number of steps, the outer loop can learn and adapt as the environment changes and can be configured to operate only at a predetermined frequency. The outer loop can use summary statistics about the error term and any other information about the environment to calculate an observation. The observation can be used to evaluate the next action, for example, by feeding the summary statistics to a neural network representing a policy. The resulting action is the coefficients of the PID controller and tunable parameters of things like filters.

[0037] Training can be updated using a reward function obtained from similar information, such as observations that sum the error across freqRL steps of the inner loop. Over time, the machine learning algorithm converges to effective PID coefficients and filter parameters for a given state of the environment. The reward function can be, for example, minimizing one or more errors, minimizing control variables, or minimizing overshoot. Once the policy is trained, it can be deployed to a production environment to help control the PID parameters.

[0038] However, the outer loop works slowly and does not directly output a new control. The inner loop helps to quickly adapt to changes in the error term, ensuring smooth control. The PID loop is easy to compute and requires very little storage, allowing it to be easily computed at each time step. Unlike a standard RL loop, there is an additional step that uses the error of the PID loop to update the observation and reward. Algorithm 1 below describes this algorithm in detail.

[0039] The algorithm described here provides fast adaptation through a PID controller and automatically updates the PID parameters as the environment demands. The algorithm combines the advantages of reinforcement learning, such as autotuning, with the advantages of a PID controller, such as fast computation and smooth control. The algorithm can operate efficiently while the control adapts to new conditions.

[0040] TIFF0007757425000001.tif216168

[0041] FIG. 2 illustrates a schematic diagram of a system 200 including a reinforcement learning outer loop 202 that can receive as sensed input information an environment 204, summary statistics 206 about the error term of a PID controller 208, and a reward function 210 based on prior changes made by the RL outer loop 202. The RL outer loop 202 can run at each freqRL step. However, the PID controller 208 can run at a greater frequency to maintain control for smooth operation. The RL outer loop 202 can use information from the environment 204 or observations such as the summary statistics 206 to generate predictions of how changes in the PID coefficients (and, e.g., the adjustable characteristics of the filter) will affect output processing and error minimization. In some embodiments, the PID inner loop can run 100-10,000 times more frequently than the outer RL learning loop. For example, the RL loop can run once per second, while the PID loop can run every 1 / 1000 of a second.

[0042] Referring to FIG. 3, a method 300 according to an exemplary embodiment of the present invention is illustrated. The method 300 can be used to automatically adjust the coefficients of a proportional-integral-derivative controller. The method 300 includes step 302 of operating a PID controller to minimize the error between a variable setpoint and a process output. The method 300 can further include step 304 of operating a reinforcement learning model. The reinforcement model can perform step 306 of receiving summary statistics from the PID controller to calculate an observation. The observation can then direct the method 300 to step 308 of selecting an action based on the observation via a policy. The policy can predict the outcome of the action, and the action can include changing the PID coefficients. The method 300 can also perform step 310 of updating the policy with a reward function. The reward function can be derived from sensed information after the policy operates on the PID controller by updating the PID coefficients.

[0043] Numerous variations and modifications can be made by those skilled in the art without departing from the spirit and scope of the present invention. Accordingly, it should be understood that the illustrated embodiments are for example purposes only and that they should not be taken as limiting the invention as defined by the following claims. For example, it should be understood that the invention may include fewer, more, or different combinations of the disclosed elements, without being hindered by the fact that the claimed elements are set forth below in certain combinations.

[0044] The words used herein to describe the present invention and its various embodiments should be understood to include not only their generally defined sense of meaning, but also generic structures, components, or operations that represent a single species by specific definition herein.

[0045] The definitions of words or elements in the claims below are therefore defined herein to not only include combinations of elements literally defined. In this sense, it is therefore anticipated that equivalent substitutions of one or more elements can be made for any one of the elements in the claims below, and that a single element can be substituted for two or more elements. Although elements may be described above as functioning in certain combinations and initially claimed as such, it should be expressly understood that one or more elements of a claimed combination may in some cases be omitted from the combination, and that a claimed combination may lead to subcombinations or variations of the subcombinations.

[0046] Slight variations of the claimed subject matter, now known or later devised, as seen by one of ordinary skill in the art, are expressly contemplated to be equivalent to the scope of the claims. Obvious substitutions now or hereafter known to one of ordinary skill in the art are defined as being within the scope of the defined elements.

[0047] The claims should thus be understood to include what is specifically illustrated and described above, what is conceptually equivalent, what is obviously substituted, and what also incorporates the essential ideas of the present invention.

Claims

1. A reinforcement learning process for automatically adjusting proportional-integral-derivative (PID) coefficients, comprising: operating a PID controller at a first frequency to minimize an error between a variable setpoint and a process output; the first frequency being 100 to 10,000 times greater than the second frequency, and operating a reinforcement learning model at the second frequency; receiving summary statistics about the error term from the PID controller and sensed information about the environment and calculating an observation; selecting an action, including varying PID coefficients, based on the observations by applying the observations to a trained policy; training the policy with a reward function derived from sensed information after application of the policy to the PID controller by updating the PID coefficients with changed PID coefficients; and A reinforcement learning process involving

2. The process of claim 1 , wherein the reward function is derived from summary statistics for an error term from the PID controller.

3. The process of claim 1 , wherein the action comprises varying an adjustable parameter of a filter.

4. The process of claim 1 , wherein the PID controller operates continuously in real time.

5. The process of claim 1 , further comprising deploying the trained policy to a production environment.

6. The process of claim 1 , wherein the reward function is based on one or more of minimizing error, minimizing change in control variables, and minimizing overshoot.

7. 1. A method for automatically adjusting coefficients of a proportional-integral-derivative (PID) controller, comprising: operating the PID controller at a first frequency to minimize an error between a variable setpoint and a process output; the first frequency being 100 to 10,000 times greater than the second frequency, and operating a reinforcement learning model at the second frequency; receiving summary statistics for the error term from the PID controller and information sensed in the environment for calculating an observation; selecting an action, including varying the PID coefficients, based on the observations by applying the observations to a trained policy; training the policy with a reward function derived from sensed information after application of the policy to the PID controller by updating the PID coefficients with changed PID coefficients; and running a reinforcement learning model that executes A method comprising:

8. The method of claim 7 , wherein the reward function is derived from summary statistics for an error term from the PID controller.

9. The method of claim 7 , wherein the action comprises an adjustable parameter of a filter.

10. The method of claim 7 , wherein the PID controller operates continuously in real time.

11. 1. A method for automatically adjusting coefficients of a proportional-integral-derivative (PID) controller, comprising: operating a PID controller at each time step operating at a first frequency to minimize an error between a variable setpoint and a process output; after a number of said time steps operating at a second frequency, said first frequency being 100 to 10,000 times greater than said second frequency; receiving summary statistics for the error term from the PID controller and information sensed in the environment to calculate an observation; selecting an action, the action including varying the PID coefficients based on the observation by subjecting the observation to a trained policy that predicts an outcome of the action; training the policy with a reward function derived from sensed information after application of the policy to the PID controller by updating the PID coefficients; and running a reinforcement learning model that executes A method comprising:

12. The method of claim 11 , wherein the reward function compares the prediction of the policy over the multiple time steps to an actual summary statistic for the error term.

13. The method of claim 11 , wherein the action comprises varying an adjustable parameter of a filter.

14. The method of claim 11 , wherein the PID controller operates continuously in real time.

Images