METHOD FOR COMPUTER-AID CONTROL OF A TECHNICAL SYSTEM, IN PARTICULAR A ENERGY GENERATION PLANT

Description

[0001] The invention relates to a method for computer-aided control of a technical system, in particular a power generation plant.

[0002] Various methods for controlling power generation plants are known in the prior art. Determining and implementing an optimal control method are particularly advantageous for efficient control of a power generation plant.

[0003] DE 10 2016 010 796 A1 discloses a method for calibrating a simulation model for calculating the functionality of an exhaust aftertreatment system. An adapted evolutionary algorithm or an adapted swarm-intelligence algorithm can be used for the calibration. For example, a covariance matrix evolution strategy can be used as an adapted evolutionary algorithm, and a particle swarm optimization can be used as an adapted swarm-intelligence algorithm.

[0004] WO 2010 / 121695 A1 discloses a control system for a complex process, in particular for controlling a combustion process in a power plant, wherein a controlled system and at least one controller are provided. The control system is hierarchically divided into different levels, wherein the first level represents the complex, real process to be controlled and is implemented by the controlled system. The second level is represented by an interface to the process and is implemented by a process control system. The third level is represented by the control of the process and is implemented by the at least one active controller. The fourth level is represented by a higher-level supervisor and is implemented by a main controller. The main controller trains simulators that simulate the controlled system. In addition, the main controller develops a set of controllers through tests on the simulators.In addition, the control in the third level can be divided into sub-areas among several active controllers, which are assigned to sub-processes of the complex process.

[0005] The object of the invention is to provide and use an improved method for computer-aided control of a technical system.

[0006] The problem is solved by independent claims 1, 6 and 7. Dependent claims 2-5 specify further developments of the invention.

[0007] One advantage of the described method is that it enables improved control of the technical system. This is achieved by determining an optimization data set based on an operating data set and a system model of the technical system using a gradient-free optimization method. The optimization method uses an open method to determine an optimization data set. Based on the optimization data set, a selection procedure is used to select relevant parameters of the technical system that enable more advantageous control of the technical system than other parameters of the technical system.

[0008] Using the selected relevant parameters, a control method for the technical system is determined. The determined control method is used to control the technical system. The term control refers to both controlling and regulating. The proposed method allows an optimized control method for the technical system to be determined with less computational effort.

[0009] In one embodiment, the selection method used is an adaptive mutual information feature selection (AMIFS) method. Using this method, more suitable relevant parameters can be selected for controlling the technical system than with other selection methods. Thus, the more precise selection of the relevant parameters further improves the control method.

[0010] In a further embodiment, the control method is determined based on the selected relevant parameters using a model-based reinforcement learning method. This allows an optimized control method for the technical system to be determined quickly and precisely.

[0011] Using the model-free reinforcement learning method, the control method is determined based on the selected, more relevant parameters. The model-free reinforcement learning method can also be used to determine a control method optimized for the technical system.

[0012] In one embodiment, a swarm optimization method is used as the optimization method. A particle swarm optimization method can be used, which is well suited for the described method.

[0013] The above-described properties, features and advantages of this invention, as well as the manner in which they are achieved, will become clearer and more clearly understood in connection with the following description of the embodiments, which are explained in more detail in connection with the drawings, in which FIG 1 shows a schematic representation of various methods for determining the control method, FIG 2 shows a schematic representation for determining a system model on the basis of an operating data set, FIG 3 shows a schematic representation of a first process status for a swarm optimization method, FIG 4 shows a schematic representation of a second process status for the swarm optimization method, and FIG 5 shows a schematic structure of a system for carrying out the method.

[0014] FIG 1 shows, in a schematic representation, various methods for generating a control method for a technical system. At a first program point 1, a predefined operating data set is provided for controlling the technical system. The operating data set is determined, for example, experimentally or assigned predefined values. In a simple embodiment, random values ​​are used for control during the creation of the operating data set at the first program point 1. The operating data set is transmitted from the first program point 1 directly to a fourth program point 4.

[0015] In a second program point 2, a system model 3 for the technical system is determined based on the operational data set using regression training and a machine learning method.

[0016] For example, a neural network, in particular a recurrent neural network, can be used to determine the system model 3 for the technical system, for example, a power generation plant. The power generation plant can be designed, for example, as a gas turbine or a wind turbine. The determined system model 3 is transmitted to the fourth program point 4.

[0017] At the fourth program point 4, a control data set 5 is determined using the operating data set and the system model 3 with the aid of a gradient-free optimization method, in particular with the aid of a swarm optimization method. The control data set 5 comprises, for at least one state of the system, an assigned optimal value for at least one control parameter, in particular assigned values ​​for a plurality of control parameters. The control data set preferably comprises the assigned optimal values ​​of control parameters for a plurality of states of the system. Thus, the control data set describes values ​​for control parameters for controlling the technical system, in particular for different states of the technical system, in order to achieve a desired optimal behavior of the technical system.

[0018] The control data set 5 is used at a fifth program point 6 to determine relevant control parameters using a selection procedure. The selection procedure at the fifth program point 6 is designed to select a limited number of relevant technical parameters of the technical system that enable better control of the technical system than other technical parameters of the technical system.

[0019] Various methods can be used to select the relevant parameters. For example, methods based on mutual information regarding the control parameters can be used. For example, an AMIFS method can be used to select the specified number of more relevant parameters from the available parameters of the technical system. The AMIFS method (Adaptive Feature Selection by Using Mutual Information) is described, for example, by Michel Tesmer and Pablo A. Estévez in 2004 IEEE, 0-7803-8359-1 / 04, pages 303 to 308. In addition, other selection methods can be used to select the specified number of more relevant parameters, as described, for example, by Isabelle Guyon et al. in "An Introduction to Variable and Feature Selection," Journal of Machine Learning Research 3(2003) 1157-1182.Furthermore, an MIFS or an MIFS-U method can also be used to select the specified number of more relevant parameters. However, experiments have shown that the AMIFS method is particularly suitable for selecting the more relevant parameters in technical systems such as power generation plants. The MIFS and MIFS-U methods are described, for example, by R. Battiti in "Using Mutual Information for Selecting Features in Supervised Neural Net Learning," IEEE Transaction on Neural Networks, Volume 5, Issue 4, Pages 537 to 550, July 1994.

[0020] At the fifth program step 6, a number of relevant parameters 7 are determined from the set of available parameters using one of the previously described methods. Using the determined relevant parameters 7, an optimized control method with improved values ​​for the technical system is determined at a sixth program step 8 using a model-based learning method. For example, the system model 3 determined at the second program step 2 is used as a model for the model-based learning method.

[0021] The control method can be created in the form of functions, tables, characteristic curves, etc. For example, the control method can be implemented as a data set, with the data set containing a value for a control parameter for several states of the system. Furthermore, the control method can be implemented as a data set, with the data set containing values ​​for several control parameters of the technical system for each state of the technical system.

[0022] Using the relevant parameters 7, an optimized control method for the technical system can alternatively be determined at a seventh program point 9 and taking into account the operating data set provided at the first program point 1 using a model-free reinforcement learning method, in particular using a reinforcement machine learning method.

[0023] The further control method can be created in the form of functions, tables, characteristic curves, etc. For example, the further control method can be implemented as a data set, wherein the data set contains a value for a control parameter for several states of the system. Furthermore, the further control method can be implemented as a data set, wherein the data set contains values ​​for several control parameters of the technical system for each state of the technical system.

[0024] At a following eighth program point 10, either the control method determined by the sixth program point 8 or the further control method determined by the seventh program point 9 is used for controlling the technical system by a control unit.

[0025] FIG 2 shows in a schematic representation a possible procedure with which a system model 3 for the technical system is determined at the second program point 2 based on the operating data provided by the first program point 1.

[0026] The description of the technical system using a state space S, an action space A, and a stochastic transition function P corresponds to the well-known Markov decision process. It is assumed that the technical system can be described using such a process. For this process, there exists a reward function c: S x A x S → R, where R represents the space of rewards in the form of real numbers rt that the system receives for choosing an action at in state st, and where the system is transitioned to state st+1.

[0027] The described method is applicable to any type of technical system whose dynamic behavior can be described by a state space S and an action space A with a stochastic transition function P( St , at , st+1 ). Here, st , st+1 are states of the technical system at times t and t+1, respectively. Furthermore, at denotes a control action acting on the technical system at time t. Furthermore, a t+1 denotes a control action acting on the technical system at time t+1.

[0028] Each state of the technical system is characterized by a plurality of state variables or environmental variables. The environmental variables are measurable state variables of the technical system, for example, gas pressure, gas temperature, combustion chamber acceleration, and the like for a gas turbine. Similarly, the state variables are, for example, wind speed, rotor blade speed, rotor blade rotational speed, rotor blade acceleration, and the like for a wind turbine.

[0029] The actions at are control parameters, i.e. manipulated variables of the technical system at time t, which in turn influence later states of the technical system. A state st can have several state variables or be characterized by several state variables. Analogous to the state st, an action at can also comprise a plurality of action variables, and an action can thus be characterized by the change of several manipulated variables. An example of a manipulated variable that can be changed in a technical system is the setting of valves in a gas turbine. In the case of a wind turbine, for example, this could be the angular position of the rotor blades or the orientation of the rotor itself. However, all other control parameters of the technical system, in particular of the wind turbine or the gas turbine, can also be used as action variables.Technical systems are often designed in such a way that the state space and the action space overlap, i.e., a control variable in the technical system also characterizes the state of the technical system.

[0030] The goal is now to determine an optimal rule for all actions in a given future period that maximizes the expected cumulative reward function or the average reward function c for each state of the given period. Maximizing the reward function c corresponds to one possible design of an action selection rule. The reward function is defined in particular so that it reflects desired properties of the technical system. In the simplest case, the reward function could, for example, have a maximum value for a desired state of the technical system or for a desired sequence of states of the technical system in the specified future period. In the following, it is assumed that the technical system is described by a Markov decision process with discrete time steps, where the state spaces S and A are continuous.

[0031] FIG 2 shows a schematic representation of a method for determining a system model for a technical system. In the method according to FIG 2 Table 11 shows an operating data set that is provided in accordance with the first program item 1. The operating data set was measured or recorded, for example, on the technical system for a specified period of time. Here, s(t) denotes the states of the technical system, a(t) denotes the action specified for state s(t), and r(t) denotes the reward achieved for action a(t) at state s(t). S(t+1) represents the state that results from state s(t) after action a(t).

[0032] Based on the available operating data set, the system model 3 is adapted, e.g. by supervised machine learning, until the measured states and the states calculated with the system model 3 match as closely as possible.

[0033] Each action at can have multiple action variables x 0 , x 1 , ..., x I-1 . The action variables thus represent at least one or more manipulated variables. In the processing block 12, the action at is now applied to the state st in a first processing step 13. A system model 3 is used for this purpose. In the first processing step 13, the next state s t+1 that then arises is determined. In addition, a reward rt is determined for the transition between the state st and the state s t+1. The reward rt is passed on to a summation block 14.

[0034] In addition, in a second processing step 15, the action a t+1 is applied to the state s t+1. System model 3 is used again. In the second processing step 15, a state s t+2 is determined based on the state s t+1 and the action a t+1. Furthermore, a reward r t-1 is determined for the second processing step 15. The reward r t+1 is fed to the summation block 14.

[0035] For a specified number of states s t+(T-1), the corresponding actions a t+2 to a t+T-1 are executed with corresponding processing steps, and the corresponding rewards r t+2 to r t+T-1 are determined and forwarded to the summation block 14. System model 3 is used in each case.

[0036] In summation block 14, the rewards rt , r t+1 , ..., r t+T-1 are summed up, whereby a weighting factor γ k< can preferably be used. The weighting factor γ k< is 1 for k = 0 and decreases with the size of k. γ can take on values ​​between 0 and 1. In this way, states further in the future, which are more uncertain, have a smaller influence on the sum of the rewards. The sum of the rewards determined in summation block 14 can be used to optimize the model, i.e. to optimize the actions a t+k for the respective states s t+k. For this purpose, the individual actions at can be changed in iterative steps. This optimizes a trajectory.

[0037] In the fourth program item 4 of the FIG 1 The operating data set from the first program point 1 and the system model determined by the second program point 2 based on the operating data set are processed. At the fourth program point 4, an improved control data set 5 is determined based on the operating data set and the system model using a gradient-free optimization method, in particular a swarm optimization method. Depending on the selected implementation, a particle swarm optimization method can be used.

[0038] A method for determining a second control data set based on the first control data set using a swarm optimization method is known, for example, from WO 2015 / 043806 A1. Furthermore, a swarm optimization method is known from "Reinforcement Learning with Particle Swarm Optimization Policy (PSO-P) in Continuous State and Action Spaces" by Daniel Hein et al., Volume 7, Issue 3, July-September 2016, International Journal of Swarm Intelligence Research, pages 23 to 42.

[0039] FIG 3 shows a schematic representation of a swarm optimization process. A second table 17 is shown with an operational data set, which was recorded or measured, for example, on the technical system. One column shows the states s(t) at time t, the actions a(t) for the state s(t) at time t, the resulting states s(t+1) at time t+1, and the rewards r(t). Depending on the chosen embodiment, the rewards r(t) can also be determined using this process.

[0040] Furthermore, a second processing block 12 is provided with processing steps 13, 15, 16 and a summation block 14. In the processing steps 13, 15, 16, the FIG 2 The system model 3 determined is used to calculate the resulting states and resulting rewards from a measured state and the possible actions. In addition, a fitness function f st (x) is determined, where an action vector x contains the actions at , a t+1 to a t+T-1.

[0041] In this method, the vector x for the state variables is iteratively modified according to the swarm optimization procedure until a vector x for the state variables is determined that has a maximum reward sum for the specified number of actions, i.e., a maximum fitness function f st (x). Details of the described method are described in the article "Reinforcement Learning with Particle Swarm Optimization Policy (PSO-P) in Continuous State and Action Spaces," by Daniel Hein et al., International Journal of Swarm Intelligence Research, Volume 7, Issue 3, July-September 2016, pages 23 to 42.

[0042] For example, the action sequences x are listed in a third table 18 in order with the corresponding fitness function f(x).

[0043] FIG 4 shows the procedure according to FIG 3 , where after determining the best action sequence x according to the third table 18, the state s(t) and the associated action a(t) of the best action sequence are written into a fourth table 19.

[0044] Then, when determining the action a(t), the procedure according to FIG 3 run through again and in doing so again a variation of the action sequence x is carried out, whereby the action a(t) remains unchanged. The optimal action sequence x resulting from this variation is used to determine the second optimal action a(t+1) and enter it into the fourth table 19. In this way, a fourth table 19 is created in which optimized actions a(t) to a(t+T-1) have been determined for the states s(t) to s(t+T-1). The fourth table 19 is then transferred as a second control data set 5 to the fifth program point 6. The following part of the method, in which a second control data set is used, is not part of the invention and is therefore only explained by way of example.

[0045] The fifth program point 6 uses a selection procedure to determine, on the basis of the second control data set 5, a specified number of selected parameters that enable better control of the technical system than the specified number of other parameters.

[0046] As already explained above, one of several methods can be used for the selection process. For example, methods that utilize mutual information to select the parameters can be used, as described, for example, by Jorge R. Vergara and Pablo A. Estévez in "A Review of Feature Selection Methods Based on Mutual Information." Preferably, an adaptive mutual information selection method (AMIFS) is used, as already described.

[0047] Thus, as a result of the fifth program point 6, a predetermined number of more relevant, i.e. more important, parameters of the technical system for optimized control are determined.

[0048] The more relevant parameters 7 are then processed either using a model-based machine learning method according to the seventh program point 8 or using a model-free machine learning method according to the eighth program point 9 in order to determine a control method which is used in the ninth program point 10 to control the technical system.

[0049] For example, a reward function (rt , r t+1 , ...) can represent an optimization criterion such as the lowest possible pollutant emissions for gas turbines or the highest possible power generation for wind turbines. However, other optimization criteria can also be considered when determining the reward function.

[0050] For the calculation of mutual information, the following parameters can be used as classes for a gas turbine, for example: ambient pressure, ambient temperature, gas turbine temperature, exhaust gas temperature, valve positions, etc. For a wind turbine, the following parameters can be used as classes: rotor speed, power generation output, rotor blade angle of attack, rotor orientation, time of day, air pressure, etc.

[0051] FIG 5shows a schematic representation of a computing unit 20 connected via control lines to actuators 22 of a power generation system 21. Furthermore, the computing unit 20 is connected via sensor lines to sensors 23. The sensors 23 are designed to detect parameters, in particular operating parameters and environmental parameters, of the power generation system 21. The computing unit 20 is designed to control or regulate the power generation system according to the determined control method according to the ninth program point 10.

[0052] Although the invention has been illustrated and described in detail by the preferred embodiment, the invention is not limited to the disclosed examples and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention, which scope is defined solely by the following claims.

Claims

1. Method for the computer-based control of a technical system for achieving a predetermined technical behaviour of the technical system, wherein an operating dataset with states of the technical system and the actions defined for the states, which exist as variables of the technical system in control parameters, is provided (1) to control the system, wherein a system model (3) is provided (2) to describe the mode of operation of the technical system, wherein on the basis of the operating dataset and on the basis of the system model with an optimisation method, a control dataset with at least one value which is optimal for achieving an optimal behaviour of the technical system is determined for at least one control parameter for at least one state of the system by means of a reward function, wherein on the basis of the control dataset with the at least one optimal value with the aid of a selection method, a fixed number of more relevant parameters of the technical system are selected (6), which enable a more advantageous control of the technical system than the fixed number of other parameters of the technical system, wherein a control method for the technical system is determined with the selected, more relevant parameters and either with the aid of a model-based (8) or with the aid of a model-free (9) reinforcing learning method, wherein the technical system is controlled (10) with the aid of the control method.

2. Method according to claim 1, wherein the selection method is an adaptive alternating information feature selection method (AMIFS).

3. Method according to one of the preceding claims, wherein the control method is determined on the basis of the selected relevant parameters with the aid of a model-based reinforcing learning method.

4. Method according to one of the preceding claims, wherein a swarm optimisation method is used as an optimisation method.

5. Method according to claim 4, wherein a particle swarm optimisation method is used as a swarm optimisation method.

6. Computing unit, which is connected to sensors by way of sensor lines and to actuators by way of control lines and is embodied to execute a method according to one of the preceding claims.

7. Computer program product with a program code stored on a machine-readable carrier for carrying out a method according to one of claims 1 to 5 during its execution.