Method and device for adaptive reinforcement learning control of a technical system

Abstract

A method (200) of controlling a technical system comprises: providing (211) a reinforcement learning (RL) agent, which has been trained to control the technical system in a nominal operating condition; using the RL agent, generating (212) a tentative control action to be applied to the technical system in its current state; and assessing (213) whether a current operating condition of the technical system is nominal. If the current operating condition of the technical system is nominal, the tentative control action is applied (214). If instead the current operating condition of the technical system is non-nominal, an alternative control action, which approximates the tentative control action and is consistent with the non-nominal current operating condition, is applied (216).

Description

METHOD AND DEVICE FOR ADAPTIVE REINFORCEMENT LEARNING CONTROL OF A TECHNICAL SYSTEM TECHNICAL FIELD

[0001] The present disclosure relates to the field of automatic control, and more precisely to automatically controlling a technical system on the basis of control actions which are output from a reinforcement learning (RL) agent. In particular, the present disclosure relates to a method where the control actions are occasionally adapted to changes in the operating condition of the technical system, yet without having to modify the RL agent in itself.BACKGROUND

[0002] Reinforcement learning (RL) algorithms have been used to solve optimal control problems for more than a decade. For example, US20130262353A1 discloses a control architecture including an action network and a critic network, where the action network determines a control action that minimizes a cost function determined by the critic network.

[0003] Many of the currently available RL-based control solutions are constrained by having predefined configurations and / or predefined boundary conditions, which were established during an offline training phase. However, real-world scenarios frequently experience changes in operational conditions due to maintenance, aging or failures, and such changes might translate into the significant physical limitations of the process. When such an RL-based control solutions is applied to the real-world scenario, it may be stopped from using the full capability of the action space. More precisely, control actions that were previously feasible may become infeasible after changes in the process, leading the RL agent to generate uncertain control actions that can be dangerous and harmful. When the uncertain control actions are applied within the real-world scenario, the resulting behavior may trigger a safety system to perform a process shutdown.

[0004] From the field of process industry, figure 3 illustrates an example of a technical system where the action space is sensitive to changes in operational conditions. Here, two parallel pipelines stemming from upstream units come together to supply a mainline towards a downstream unit. A similar setup may be found in a cracking furnace within a chemical plant, where a mixture of fuels fromtwo parallel lines — hydrogen gas and methane — feeds the burners. Maintaining a consistent pressure within the mixture line is important to ensuring the continuous operation of the burners. In such a setup, a significant pressure drop in the mainstream line can prompt the shutdown of the burners, abruptly halting furnace operations. This scenario initiates a chain reaction of problems, including extensive maintenance procedures and costly downtime. Furthermore, production halts can cause significant financial problems for the process, making the situation even harder. In this scenario, if any of the valves 151, 152 experiences reduced operational capacity due to factors like clogging, maintenance or other issues, the process controller must adjust its actions to ensure the output flow at the mixing point 153 maintains a setpoint level. This is necessary in order to keep the pressure of the mixture lines within a feasible range for the downstream burners.

[0005] The example in figure 3 highlights how constraints on action variables, such as valve openings, can frequently fluctuate within an industrial process. Similar situations may arise in sectors such as cement, oil and gas, pulp and paper, marine operations and beyond. Some conventional optimal control algorithms, including Model Predictive Control (MPC), are able to adapt to variations in the operational conditions by adjusting boundary conditions and / or solving new optimization problems in subsequent iterations. RL agents, however, struggle to adapt effectively.

[0006] To make up for this shortcoming of RL agents, it may be considered to retrain the RL agent, whereby new limitations and constraints can be accommodated. For example, US20180281180A1 discloses an RL agent which controls a robot that picks workpieces from a storage and hands them over to an operator. The RL agent is continually trained at runtime, wherein rewards are calculated based on the duration of each handover event. Although useful in theory, approaches of this kind are often impractical due to their high computational costs, as well as the need for time and adequate training data.

[0007] Another possible way of adapting the RL agent to an evolving process may be transfer learning. While generally recognized as very effective, transfer learning has significant requirements for high-quality historical data, and it is computationally heavy since the non-frozen layers of the RL agent must be retrained. Such data quality and computational resources are not always available in a real-world scenario

[0008] For all these reasons, a novel RL-based control solution which is robust to changing operating conditions of the system under control would be very desirable and have significant commercial potential.SUMMARY

[0009] One objective of the present disclosure is to make available a method and a device for controlling a technical system on the basis of control actions generated by an RL agent in such manner as to tolerate non-nominal operating conditions of the technical system. A further objective is to make available such a method and a device, in which the RL agent can be used unchanged despite a switch from a nominal to a non-nominal operating condition. In particular, it should be possible to handle the non-nominal operating conditions without modifying the RL agent. In particular, it should be possible to handle the non-nominal operating conditions without having recourse to a retrained version of the RL agent; rather, the method and device should be able to process the deviations away from nominal operating conditions in real time. Preferably it should be possible to handle the non-nominal operating conditions while neither modifying the RL agent, nor having recourse to a retrained version of the RL agent. A further objective is to make available a method and a device for RL-based control of a technical system configured to replace a control action with an alternative control action when the technical system’s operating condition is non-nominal, wherein the alternative control action is determined using preexisting elements from the RL agent. A further objective is to make available a method and device for RL-based control of a technical system which are, at runtime, independent of historical data.

[0010] At least some of these objectives are achieved by the present invention, as defined by the independent claims. The dependent claims are directed to advantageous embodiments of the invention.

[0011] In a first aspect of the present disclosure, there is provided a method of controlling a technical system. The method comprises: providing an RL agent, which has been trained to control the technical system in a nominal operating condition; using the RL agent, generating a tentative control action to be applied to the technical system in its current state; and assessing whether a current operating condition of the technical system is nominal. If the current operating condition of the technicalsystem is nominal, the tentative control action is applied. If the current operating condition of the technical system is non-nominal, an alternative control action is applied instead, which approximates the tentative control action and is consistent with the non-nominal current operating condition.

[0012] In the terminology of the present disclosure, an “operating condition” (or status) of a technical system may correspond to a known or estimated state of the technical system, to an observable of the technical system, or to an environmental parameter of the technical system. Depending on which of these applies, there is a corresponding test for assessing whether the current operating condition is nominal or non-nominal. Further in the terminology herein, a control action is considered “consistent with” the non-nominal operating condition if the control action belongs to a constrained action space corresponding to the non-nominal operating condition.

[0013] In the control method according to the first aspect, a control action generated by the RL agent is not used indiscriminately, but only after it has been assessed whether the current operating condition of the technical system is nominal. The control action is otherwise replaced with an alternative control action, which approximates the tentative control action and is consistent with the non-nominal current operating condition. As the inventors have realized and verified, this is an efficient and computationally lean way of handling non-nominal operating conditions of the technical system. A particular benefit of the control method according to the first aspect is that the RL agent can be used unchanged, even when the operating condition is non-nominal; instead, the control method can be executed without modifying the RL agent, let alone using a retrained version of the RL agent.

[0014] In some embodiments, the control method further comprises a step of determining the alternative control action by selection from a first set B(a1) of control actions which approximate the tentative control action a1and are consistent with the non-nominal current operating condition. The requirement for consistency with the non-nominal current operating condition can be met by selecting the alternative control action from the first set’s intersection with a constrained action space Ā.

[0015] In the terminology of the present disclosure, an “action space” of the technical system models the set of technically admissible control actions, e.g., in the sense that they are well-defined from the point of view of the technical system and / orcan be applied substantially without causing an error in the technical system and / or are within the abilities of the actuators associated with the technical system. The nominal action spaceof the technical system corresponds to control actions suitable for those real or simulated states which are nominal operating conditions of the technical system and on which the RL agent has been trained. The constrained action space Ā replaces the nominal action space A of the technical system when the operating condition is non-nominal. The constrained action space Ā (in particular, the extents of the constrained action space Ā) may be variable as a function of the non-nominal current operating condition.

[0016] Constructively, in these embodiments, the selection of the alternative control action from the first set B(a1) of control actions can be guided by outputs obtained by evaluating the RL agent’s Q function for at least some of the control actions in the first set B(a1). Alternatively, the selection of the alternative control action from the first set B(a1) of control actions can be guided by iteratively maximizing the Q function, wherein the maximization is initialized with the tentative control action and is constrained to be consistent with the non-nominal current operating condition.

[0017] In a second aspect of the present disclosure, there is provided a control device for controlling a technical system. The control device comprises: an input interface for obtaining information indicative of a current state of the technical system; an RL agent, which has been trained to control the technical system in a nominal operating condition; an output interface for feeding information indicative of a control action to be carried out by actuators associated with the technical system; and processing circuitry configured to perform the above-outlined method according to the first aspect.

[0018] The control device according to the second aspect generally shares the effects and advantages of the control method according to the first aspect, as outlined above, and it can be implemented with a corresponding degree of technical variation.

[0019] The present disclosure further relates to a computer program containing instructions for causing a computer, or the control device in particular, to carry out the method according to the first aspect. The computer program may be stored or distributed on a data carrier. As used herein, a “data carrier” maybe a transitory datacarrier, such as modulated electromagnetic or optical waves, or a non-transitory data carrier. Non-transitory data carriers include volatile and non-volatile memories, such as permanent and non-permanent storage media of magnetic, optical or solid-state type. Still within the scope of “data carrier”, such memories may be fixedly mounted or portable.

[0020] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order presented, unless this is explicitly stated.BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Aspects and embodiments are now described, by way of example, with reference to the accompanying drawings, on which:figure 1 shows an industrial system together with a control device which incorporates an RL agent, according to embodiments herein;figure 2 is a flowchart of a method of controlling a technical system using control actions generated by an RL agent, according to embodiments herein;figures 3 to 5 are block diagrams of technical systems, which are possible use cases of the teachings disclosed herein;figure 6 illustrates how the action space of a technical system (vertical axis) varies in response to changes in an operating condition (horizontal axis); andfigures 7 to 9 illustrate different configurations of a nominal action spaceand a constrained action space < A.DETAILED DESCRIPTION

[0022] The aspects of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, on which certain embodiments of the invention are shown. These aspects may, however, be embodied in many different forms and should not be construed as limiting; rather, theseembodiments are provided by way of example so that this disclosure will be thorough and complete, and to fully convey the scope of all aspects of the invention to those skilled in the art. Like numbers refer to like elements throughout the description.

[0023] Figure 1 shows a control device 100 arranged to control a technical system 150. The control device 100 includes an input interface 110 configured to obtain information indicative of a current state of the technical system 150. Such information may include values of one or more observables (or output variables) of the technical system 150, which the input interface 110 receives in the form of a digital or analog signal conveyed by an electric or optical medium. The control device 100 further comprises an output interface 130 for feeding information indicative of a control action to be carried out by the actuators associated with the technical system 150. The information indicative of the control action maybe a control signal (or input variable). In the present disclosure, the terms state x, observable y and control signal u refer to the generally applicable state-space representation of a linear or nonlinear dynamic system:where x, y, u are vectors with one or more elements, A, B, C, D are constant or timevariable matrices, and f, g are time-invariant or time-variable functions. The state x evolves in state space S over time, in accordance with linear system equations (1a) or nonlinear system equations (1b), in a way that depends on the values that the one or more state variables have at any given instant and further on the values of externally imposed control signals u, which lie in an action space A of the system. The observables y depend on the state-variable values and may also have a direct dependence on the control signals u.

[0024] The control device 100 further comprises processing circuitry 140 and a memory 160 for storing one or more computer programs 161 with instructions executable by the processing circuitry 140. The processing circuitry 140 may for example be general-purpose (programmable) circuitry with one or more processing cores, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or a system-on-chip.

[0025] Also comprised in the control device 100 is a reinforcement learning (RL) agent 120. The present disclosure will endeavor to use terminology and concepts in a manner consistent with the usage in the monographR. Sutton and A. G. Barto, “Reinforcement Learning: An Introduction”, 2ndedition, MIT Press, Cambridge MA (2018).In line with the architecture of figure 11.1 of Sutton (2018), implementations of the control device 100 may include an RL agent 120 structured as follows:- an actor network, which is a neural network trained to approximate a policy, and- a critic network, which is a neural network trained to approximate a Q function (or state-action value function).The policy π(x) defines which action a to take in each state x. When an action is taken, the environment transitions to a new state x' and the agent receives a reward r. The reinforcement learning problem can be modeled as a Markov Decision Process (MDP), which is defined by the tuple (S; A; T; R; γ), where 5 is the state space,is the action space, T is a state transition model (or evolution operator, cf. above equations 1a and 1b), R is a reward model, and y is a discount factor. This model can also be considered to represent the RL agent’s interaction with the training environment. At every time step t, the goal of the agent is to choose an action a that maximizes the discounted return,In so-called Q-learning, the RL agent 120 is trained to learn the optimal action-value function Q*(x, a), which is defined asQ*(x, a) = max IE[Rt|xt= x, at= a, π],71where IE denotes expected value. From the optimal action-value function, the policy is derived as perπ(x) = argmax Q*(x, a). (2) a

[0026] In the control device 100, the control signals u may be derived from the control actions a that the RL agent 120 outputs, e.g., by translating, converting orreformatting them into a signal or command format that is understood by actuators in the technical system 150.

[0027] The technical system 150 may be any system that has structural and functional means configured to perform a task in an industry branch. Example industry branches include: agriculture and forestry; fishing; mining and quarrying; manufacture of food products, beverages, tobacco; manufacture of textiles; manufacture of leather; manufacture of wood products; manufacture of pulp, paper, paper products, printed products; manufacture of coke, refined petroleum products, nuclear fuel; manufacture of chemicals, chemical products; manufacture of rubber and plastics; manufacture of non-metallic mineral products; manufacture of metals and metal products; manufacture of machinery; manufacture of electric and optic equipment; manufacture of transport equipment; electricity gas and water supply; construction; wholesale and retail trade; repair of motor vehicles; transport, storage and communication; real estate and building activities; education; healthcare; domestic work. A task in the field of an industry branch is a task which occurs in the industry branch (e.g., such that known building blocks and other common knowledge of practitioners in the industry branch can be relied upon) and / or which shall be solved in a manner consistent with the practices in the industry branch (as regards dimensional accuracy, chemical purity, minimum hygiene requirements, safety, etc.). The technical system 150 may in particular be an industrial process or an industrial system.

[0028] To illustrate, figure 3 shows a technical system 150 where a pressurized hydrogen gas pipeline joins a pressurized methane pipeline at a mixing point 153 after passing through respective controllable valves 151, 152. The settings of the two valves 151, 152 (or two linear combinations of the settings, such as their sum and difference) are the control variables. Suitable observables may be the pressure and composition of the outgoing mixture. The observables maybe used as state variables in the system model. The system dynamics are influenced by the two incoming pressures, and these may optionally be used as further state variables.

[0029] Figure 4 illustrates a technical system 150 with two independent control signals (two actuators) and a single process variable. The control signals correspond to the settings of two independent valves 154, 155, located on a main pipeline and a parallel smaller pipeline, which are to be coordinated accurately and stably, and preferably over a wide range of flow speeds.

[0030] The structure seen in figure 4 may be considered to be motivated by the need to handle a large flow from time to time, which makes a large valve 155 necessary. However, such equipment may struggle to achieve the requisite high precision or responsiveness. To address this problem, a portion of the flow is diverted into the parallel smaller pipeline fitted with a smaller, more accurate and / or more responsive valve 154. The output flow is the sum of the main flow and the parallel flow, wherein finer adjustments on a shorter time scale can preferably be carried out by the smaller valve 154. In such a use case, if one valve encounters operational issues like clogging or needs temporary maintenance, the control system should adapt to the altered operational ranges of these control valves to guarantee the desired accuracy, ultimately to maintain the setpoint output flow. Unlike an RL agent 120, conventional optimal MPC algorithms possess the capability to adapt to such variations. MPC achieves this by recalibrating boundary conditions and iteratively solving new optimization problems. In contrast, RL agents face challenges in effectively adapting to process capability changes, which is what the present teachings address.

[0031] Figure 5 shows an alternative layout of a flow regulation system for a similar use case as that of figure 4. Here, there is a single pipeline, on which the larger valve 156 is arranged upstream of the smaller valve 157. This way, the larger valve 156 is configured to adaptively moderate (throttle) the incoming flow, such that the resulting flow and pressure are in a range that the smaller valve 157 can handle. This could relax the requirements on the smaller valve 157 compared to the corresponding valve 154 in figure 4, which is exposed to the full incoming pressure. Because of the serial arrangement of the valves 156, 157 in the present system 150, however, their mutual interaction in cases of full or partial failure will be felt more strongly.

[0032] Although the examples in figure 3 to 5 are related to process industry and the regulation of fluid flow more precisely, the teachings presented herein are applicable and useful in a much wider context, including technical systems 150 in any of the industry branches enumerated above. Furthermore, even though the actuators seen in figures 3 to 5 (i.e., the valves 151, 152, 154, 155, 156, 157) may be considered as integral parts of their respective technical systems, it is understood that the teachings herein are equally applicable to a technical system 150 with external actuators, i.e., where the actuators influence the technical system 150 while being merely associated with it.

[0033] Figure 2 is a flowchart illustrating a method 200 of controlling a technical system on the basis of control actions generated by an RL agent. The method 200 may be executed by any general-purpose computer which has at its disposal an RL agent 120 and which is configured to exchange observables and control signals with a technical system 150. In particular, the method 200 can be understood as a configured behavior of the control device 100, e.g., a behavior defined by computer-readable instructions in the computer program 161.

[0034] While training is not a mandatory step of the method 200, it is assumed that the RL agent 120 has initially been trained, in a step 210, on the basis of real or simulated states which are nominal operating conditions of the technical system 150 under consideration. Said real or simulated states can be recognized as nominal operating condition of the technical system 150 if the states lie in an associated nominal state range Nx. Further, said real or simulated states can be recognized as nominal operating condition of the technical system 150 if the states give rise to observable values which lie in an associated nominal observable range Ny. The output of the training step 210 may include a neural network or weight values of a neural network.

[0035] The control method 200 further comprises a step 211 of providing the RL agent 120. Said act of providing may include ensuring that the control device 100 or another entity that executes the method 200 has a data structure representing the RL agent 120 at its disposal. For example, neural networks functioning as the actor network and the critic network may be initialized (e.g., in runtime memory) and weight values determined during a preceding training step 210 maybe assigned to it. It is recalled that the training step 210 is not a mandatory step in all embodiments of the control method 200; instead, the RL agent 120 or its parameters maybe obtained from an external entity.

[0036] In a next step 212, the RL agent 120 is requested to generate a tentative control action a₁ to be applied to the technical system in its current state x₁. Generating the tentative control action a₁ may correspond to evaluating equation 2 for x = x₁, which is an operation of feeding the current state of the technical system 150 to the RL agent 120. A suitably trained RL agent 120 will return a tentative control action a which normally lies in the nominal action space of the technical system 150. The nominal action space of the technical system corresponds to control actions suitable for those real or simulated states which are nominaloperating conditions of the technical system and on which the RL agent 120 has been trained. This is to say, in nominal operating conditions, the control actions in 풜 are well-defined from the point of view of the technical system 150 and / or they can be applied substantially without causing an error in the technical system 150 and / or they are within the abilities of the actuators 151, 152, 154, 155, 156, 157 associated with the technical system 150.

[0037] The execution flow of the method 200 proceeds to a decision point 213, where it is assessed whether a current operating condition of the technical system 150 is nominal. In some embodiments of the method 200, this assessment may include assessing (substep 213.1) whether a current value of an observable y of the technical system 150 is in an associated nominal observable range Ny. Alternatively, in other embodiments, this assessment may include assessing (substep 213.2) whether a current value of a state x of the technical system 150 is in an associated nominal state range Nx. Further alternatively, the assessment in point 213 may include whether an environmental parameter of the technical system 150 is in the nominal state range Nx.

[0038] As explained above under step 210, a state can be recognized as nominal operating condition of the technical system 150 if the state lies in the nominal state range Nx. Further, said state can be recognized as nominal operating condition of the technical system 150 if the state gives rise to an observable value which lies in the nominal observable range Ny. The nominal observable range Nymaybe determined theoretically (e.g., from specified capabilities of the actuators), by simulations and / or by observing the technical system 150 under consideration while it is known to operate in nominal operating conditions. Similarly, the nominal state range Nxcan be determined theoretically (e.g., from specifications of the system components), by simulations and / or by estimating states of the technical system 150 under consideration on the basis of observations thereof while the technical system 150 is known to operate in nominal operating conditions. In specifications of the system components, environmental specifications (e.g., acceptable physicochemical operating ranges) and machine-parameter specifications (e.g., allowable ranges of parameters relating to conditions of technical components) may be relevant.

[0039] If it is found in point 213 that the current operating condition is nominal, the tentative control action a is applied, by an execution of a step 214. Depending onthe characteristics of the actuators associated with the technical system 150, the tentative control actioncan be fed via the output interface 130 in its preexisting format, or the tentative control actionmay undergo a translation, conversion or reformatting into a form that is understood to the actuators. Step 214 may include a conversion from a high-level command into a machine-level instruction. Preferably, the operations within step 214 preserve the substantive content of the tentative control action a as far as possible.

[0040] If it is instead found in point 213 that the current operating condition is non-nominal, step 216 is executed, in which an alternative control action is applied, which is consistent with the non-nominal current operating condition. In some embodiments, a preconfigured fallback control action is applied which is independent of the current state x. The preconfigured fallback control action maybe one which is always acceptable in the technical system 150, or one which is acceptable in a very wide range of operating conditions. In other embodiments, in addition to or instead of the preconfigured fallback control action, the alternative control action approximates the tentative control action and is consistent with the non-nominal current operating condition.

[0041] A suitable alternative control action, which approximates the tentative control action and is consistent with the non-nominal current operating condition, maybe determined in a step 215. In some embodiments, this step 215 includes a selection (substep 215.1) from a first set B(a1) of control actions which approximate the tentative control action a1and are consistent with the non-nominal current operating condition. The alternative control action maybe considered to approximate the tentative control action a if it is close - with respect to a suitable norm or distance function in the action space- to the tentative control action a. The alternative control action may be considered to be consistent with the non-nominal current operating condition if it lies in a constrained action space < A. Figure 8 geometrically illustrates the concepts of the tentative control action a, the corresponding first set B(ax), the action space c / Z and the constrained action space < A in a two-dimensional example case, where the two components of the control signal correspond to the coordinate axes u1, u2of the plane. Figure 8 further illustrates a second set H(a₁) of points in the (u1, u2) plane which are close to the tentative controlaction a₁. The first set B(a₁) may be understood as the intersection of the second set H(a₁) and the constrained action space Ā.

[0042] In general, the constrained action space < A is independent of the nominal action space < A. At least the following cases are possible:a) The constrained action space < A is a subset of the nominal action space < A.Then, the nominal action space < A contains at least one control action not contained in the constrained action space < A, as shown in figure 7A.b) The nominal action space < A is a subset of the constrained action space < A.Then, the constrained action space < A contains at least one control action not contained in the nominal action space < A, as shown in figure 7B.c) The constrained action space < A partially overlaps with the nominal action space c / Z. Here, the nominal action space < A contains at least one control action not contained in the constrained action space < A, and the constrained action space < A contains at least one control action not contained in the nominal action space < A, as shown in figure 7C.The action spaces < A, < A maybe finite or infinite sets. Figure 9 shows an example where the action spaces < A, < A have a semi-infinite extent with respect to the u coordinate.

[0043] In general, the constrained action space < A is variable as a function of the non-nominal current operating condition, and particularly in the sense that non-nominal different operating conditions give rise to different appearances of the constrained action space < A.

[0044] Figure 6 illustrates this variability in a case where the operating condition corresponds to a state of the technical system 150. For simplicity of the drawing, the state is shown as a one-dimensional quantity, and likewise the action space < A, < A shown as a one-dimensional range. In real implementations, both the state space and action space maybe multi-dimensional. The state x is recognized as a nominal operating condition of the technical system 150 if the state lies in the nominal state range Nx, and then any control action in the nominal action space c / Z (set 301 in figure 6) can be applied. When the technical system 150 is in a first non-nominal state x = the control actions in a first constrained action space A(x^) are available (set302). When the technical system 150 is in a second non-nominal state x = x2, the control actions in a second constrained action space c / ?(x2) are available (set 303).

[0045] In substep 215.1, the selection may be guided by evaluating the Q function in the RL agent 120 for at least some of the control actions in the first set B(ax). An evaluation may include feeding the control actions in the first set B(ax) to the critic network which approximates the Q function. To provide a sufficient basis for the selection, the evaluations can be organized in many different ways, including:a) evaluate the Q function for all control actions in the first set B(ax) and pick the global optimum;b) evaluate the Q function for some control actions in the first set B(ax), until a Q value is found which is better than a preconfigured absolute threshold;c) evaluate the Q function for some control actions in the first set B(ax) and pick the best one of these.Under option c, the control actions in the first set B(a₁) for which the Q function is evaluated may correspond to a preconfigured number of randomly sampled control actions, wherein the random sampling may correspond to a uniform distribution (the randomly sampled control actions are uniformly distributed in action space), or it may correspond to a known or estimated probability distribution that is characteristic of the technical system 150 under consideration.

[0046] With reference to the same concepts, other embodiments of the method 200 comprise a substep 215.2 wherein the alternative control action is determined by maximizing the Q function. Substep 215.2 is an alternative way of determining the alternative control action and may thus replace substep 215.1 described above.

[0047] In particular, substep 215.2 may include an iterative maximization of the Q function, wherein the maximization is initialized with the tentative control action and is constrained to be consistent with the non-nominal current operating condition. Recalling that consistency with the non-nominal current operating condition can be ensured by a restriction to a constrained action space < A corresponding to said non-nominal operating condition, the maximization can be written as follows:where Q(x1, a) is the restriction of the Q function (in particular, of the optimal action-value function) to the current state x of the technical system 150, || • || is a norm (or distance function) in action space, and K is the radius of a neighborhood of the tentative control action a. The neighborhood represents those alternative control actions that approximate the tentative control action a. It is assumed that the Q function assigns a higher Q value to a more desirable action and vice versa. A sign change may be needed in implementations where the Q function is defined such that a more desirable action obtains a lower Q value, so that the Q function is minimized or the maximization is applied to -Q(x, a).

[0048] To find a numerical solution to the maximization problem (3), an iterative or non-iterative optimization solver can be applied. An iterative solver can be modeled as a function F such thatvn+1= F(vn), n > 1, (4) where the sequence v1, v2, v3,... approaches a solution of the problem (3) monotonically or non-monotonically. Preferably, the iterative maximization (4) is initialized with the tentative control action, v = a. If the Q function is chosen as a convex function, convergence will be given under broad circumstances.

[0049] The iterative maximization (4) may for example be a gradient-descent maximization of Q(x, a), or a gradient-descent minimization applied to -Q(x₁, a). In the case of a gradient-descent maximization, one hasF(vn) = vn+ γn∇Q(x1, vn), (5) where ∇Q(x1, vn) is (a numerical approximation of) the gradient at vnof Q(x, a), and γn> 0 is a step size at step n. The step size maybe constant or variable with respect to n. The constraint a G < A may be enforced by verifying before each iteration of the iterative maximization (5) that the right-hand side of equation 5 is an element of the constrained action space c / ?; if this is not true, the step size can be adjusted or the maximization may be allowed to run for some more steps. The final solution (to be used as alternative control action) should at any rate be required to lie in the constrained action space < A.

[0050] In the special case where <7? c <4 and the tentative control action a g < A, the iterative maximization (5) may be initialized with an arbitrary element in < A.Then, the iterative maximization (5) will reach the best solution of (3) within c / ? eventually, although this may require a somewhat greater number of iterations than in the general case where the maximization (5) can be initialized with the tentative control action, v = a.

[0051] For most of the iterative numerical optimization solvers described in the literature, there is at least one convergence test or at least one optimality test. In particular, such tests exist for the gradient-descent method. These convergence or optimality tests can be used for implementing the teachings herein. More precisely, this allows a verification that the solver has converged for the current iterate and / or that the current iterate is optimal, wherein the current iterate can be adopted as the final solution (to be used as alternative control action).

[0052] As can be seen from the above description, the method 200 is performed without modifying the RL agent 120, whether by retraining, transfer learning or finetuning.

[0053] After this stage, as shown in figure 2, the execution flow of the method 200 may optionally loop back to step 212, in which a further tentative control action a2is generated for the then current state x2of the technical system 150. The step 213 of assessing whether the current operating condition is nominal may be repeated for each new execution of step 212 in some embodiments, or it maybe performed more rarely in other embodiments. For example, step 213 may be repeated only for each A th repetition of step 212, where N > 2 is a preconfigured integer, or step 213 maybe repeated only if a preconfigured time period T213has elapsed since the previous execution of step 213. The embodiments where step 213 is performed more rarely than step 212 are suitable for a technical system 150 which is expected to respond to a change in operating conditions (e.g., external or environmental factors) by a relatively longer time constant than the time constant which governs variations in the state of the technical system 150.

[0054] The aspects of the present disclosure have mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.

Claims

CLAIMS1. A method (200) of controlling a technical system (150), comprising: providing (211) a reinforcement learning agent, RL agent (120), which has been trained to control the technical system in a nominal operating condition;using the RL agent, generating (212) a tentative control action (a₁) to be applied to the technical system in its current state;assessing (213) whether a current operating condition of the technical system is nominal; and- if the current operating condition of the technical system is nominal, applying (214) the tentative control action;- if the current operating condition of the technical system is non-nominal, applying (216) an alternative control action, which approximates the tentative control action and is consistent with the non-nominal current operating condition.

2. The method (200) of claim 1, further comprising:determining (215) the alternative control action by selection (215.1) from a first set (B(a₁)) of control actions which approximate the tentative control action (a₁) and are consistent with the non-nominal current operating condition.

3. The method (200) of claim 2, wherein the RL agent (120) includes a Q function, and said selection (215.1) includes evaluating the Q function for at least some of the control actions in the first set.

4. The method (200) of claim 1, further comprising:determining (215) the alternative control action by an iterative maximization (215.2) of the Q function, wherein the maximization is initialized with the tentative control action and is constrained to be consistent with the non-nominal current operating condition.

5. The method (200) of claim 4, wherein the iterative maximization (215.2) includes executing a numerical optimization solver, such as a Gradient Descent solver.

6. The method (200) of any of claims 2 to 5, wherein a control action is considered consistent with the non-nominal current operating condition if the control action belongs to a constrained action space (c / l), which is independent of a nominal action space (c / Z) of the technical system (150).

7. The method (200) of claim 6, wherein:the nominal action space (c / Z) contains at least one control action not contained in the constrained action space (c / l); and / orthe constrained action space (c / ?) contains at least one control action not contained in the nominal action space (c / Z).

8. The method (200) of claim 6 or 7, wherein the constrained action space (c / ?) is variable as a function of the non-nominal current operating condition.

9. The method (200) of any of the preceding claims, which is executed without modifying the RL agent (120).

10. The method (200) of any of the preceding claims, wherein assessing (213) whether the current operating condition of the technical system (150) is nominal includes at least one of:assessing (213.1) whether an observable (y) of the technical system is in an associated nominal observable range (Ny)assessing (213.2) whether a state (x) or a state estimate or an environmental parameter of the technical system is in an associated nominal state range (Nx).

11. The method (200) of any of the preceding claims, further comprising initially training (210) the RL agent (120) on the basis of real or simulated states which are nominal operating conditions of the technical system (150).

12. The method (200) of any of the preceding claims, wherein the tentative control action (ax) is generated (212) by feeding the current state of the technical system to the RL agent (120).

13. A control device (100) for controlling a technical system (150), the control device comprising:an input interface (110) for obtaining information indicative of a current state of the technical system;a reinforcement learning agent, RL agent (120), which has been trained to control the technical system in a nominal operating condition;an output interface (130) for feeding information indicative of a control action to be carried out by actuators (151, 152; 154, 155; 156, 157) associated with the technical system; andprocessing circuitry (140) configured to perform the method (100) of any of the preceding claims.

14. A computer program (161) containing instructions to cause the control device (100) of claim 13 to carry out the method (100) of any of the preceding claims.

Images