METHOD AND CONTROL DEVICE FOR CONTROLLING A TECHNICAL SYSTEM

Description

[0001] Data-driven machine learning methods are increasingly being used to control complex technical systems, such as gas turbines, wind turbines, steam turbines, engines, robots, chemical reactors, milling machines, production plants, cooling systems, or heating systems. In particular, artificial neural networks are trained using reinforcement learning methods to generate a state-specific control action for a given state of the technical system, thereby optimizing the performance of the technical system. Such a control agent optimized for controlling a technical system is often referred to as a policy or, for short, an agent.

[0002] Successful optimization of a control agent typically requires large amounts of operational data from the technical system as training data. The training data should cover the operating states and other operating conditions of the technical system as representatively as possible.

[0003] In many cases, such training data is available in the form of databases containing operating data recorded on the technical system. Such stored training data is often referred to as batch training data or offline training data. Experience shows that training success generally depends on the extent to which the possible operating conditions of the technical system are covered by the batch training data. Accordingly, it can be expected that control agents trained with batch training data will behave poorly in operating conditions for which only limited batch training data was available.

[0004] To improve control behavior in areas of the state space that are poorly covered by training data, a recursive learning method is proposed in the publication "Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization" by Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu at https: / / arxiv.org / abs / 2006.03647 (accessed October 8, 2021). However, this method yields stochastic policies that can sometimes output very different and unpredictable control actions for the same operating state. While this method allows for efficient exploration of the state space, such stochastic policies are not permissible on many technical systems because they cannot be reliably validated in advance.The disclosure DE 10 2016 224207 A1 (SIEMENS AG [DE]) relates to a computer-implemented method for controlling a technical system, wherein training data is read in, wherein a respective training data set comprises a state data set specifying a state of the technical system, an action data set specifying a control action and a performance value of the technical system resulting from an application of the control action, as well as a first machine learning module which is trained on the basis of the training data to reproduce a resulting performance value on the basis of a state data set and an action data set, and a control agent for controlling the technical system.

[0005] It is an object of the present invention to provide a method and a control device for controlling a technical system, which allow an exploration of a state space of the technical system and thereby use deterministic control agents.

[0006] This object is achieved by a method having the features of patent claim 1, by a control device having the features of patent claim 12, by a computer program product having the features of patent claim 13 and by a computer-readable storage medium having the features of patent claim 14.

[0007] To control a technical system, training data is read in. Each training data set comprises a state data set specifying a state of the technical system, an action data set specifying a control action, and a performance value of the technical system resulting from the application of the control action. The term "control" also refers to regulating the technical system. Using the training data, a first machine learning module is trained to reproduce a resulting performance value based on a state data set and an action data set. Furthermore, state data sets are each fed to a plurality of different deterministic control agents, and the resulting output data is fed into the trained first machine learning module as action data sets.Depending on the performance values ​​output by the trained first machine learning module, several of the control agents are then selected. According to the invention, the technical system is controlled by the selected control agents, with additional state data sets, action data sets, and performance values ​​being recorded and added to the training data. Using the thus supplemented training data, the above method steps are repeated starting with the training of the first machine learning module.

[0008] To carry out the method according to the invention, a corresponding control device, a computer program product and a computer-readable, preferably non-volatile storage medium are provided.

[0009] The method according to the invention and the control device according to the invention can be carried out or implemented, for example, by means of one or more computers, processors, application-specific integrated circuits (ASICs), digital signal processors (DSPs) and / or so-called "field programmable gate arrays" (FPGAs).

[0010] A particular advantage of the invention is that, due to the control of the technical system by several different and iteratively adapted control agents, an effective exploration of the state space of the technical system is enabled. In this way, the negative effects of areas of the state space poorly covered by training data can be effectively reduced. At the same time, by limiting the scope to deterministic control agents, validation problems associated with stochastic control agents can be avoided.

[0011] Advantageous embodiments and further developments of the invention are specified in the dependent claims.

[0012] According to an advantageous embodiment of the invention, a second machine learning module can be trained, or trained using the training data, to reproduce an action data set using a state data set. The control agents can each be compared with the second machine learning module, wherein a distance quantifying a difference between the respective control agent and the second machine learning module is determined. In this way, a control agent with a smaller distance to the second machine learning module can be preferentially selected over a control agent with a greater distance. In particular in artificial neural networks, a distance between neural weights of a respective control agent and neural weights of the second machine learning module can be determined as the distance. Preferably, a possiblyweighted Euclidean distance between vector representations of these neural weights can be determined.

[0013] In particular, the distance of a respective control agent can be compared to a threshold. The respective control agent can then be excluded from the selection if the threshold is exceeded. A precondition for the selection of a control agent can thus be formulated as |N2 - NP| <= TH, where TH represents the threshold, N2 represents a vector of neural weights of the second machine learning module, and NP represents a vector of neural weights of the respective control agent. In this way, the control of the technical system can be limited to control agents that do not differ too much from the second machine learning module. In many cases, this can effectively exclude control agents that issue inadmissible or severely detrimental control actions.

[0014] Advantageously, the threshold can be increased when repeating the process steps. This can improve the exploration of the state space of the technical system during iterative training.

[0015] According to a further advantageous embodiment of the invention, the plurality of control agents can be generated by a model generator. In this context, several of the generated control agents can be compared with other control agents, wherein a distance quantifying a difference between the compared control agents is determined in each case. Advantageously, a control agent with a greater distance to one or more other control agents can then be preferentially selected and / or generated over a control agent with a smaller distance. This makes it possible to provide control agents that exhibit a high degree of diversity among themselves. In this way, different regions of the state space can be explored in parallel and thus more efficiently.

[0016] According to an advantageous development of the invention, the control agents can be trained using the training data to reproduce an action data set specifying a performance-optimizing control action based on a state data set. In addition to selection, such additional training can, in many cases, further improve the performance of the selected control agents.

[0017] According to a further advantageous development of the invention, a respective training data set can comprise a subsequent state data set that specifies a subsequent state of the technical system resulting from the application of a control action. The first machine learning module can then be trained using the training data to reproduce a resulting subsequent state data set based on a state data set and an action data set. Furthermore, a second machine learning module can be trained using the training data to reproduce an action data set based on a state data set. The trained first machine learning module can then determine a subsequent state data set for an action data set output by the respective control agent and feed it into the trained second machine learning module.The resulting subsequent action dataset can then be fed into the trained first machine learning module together with the subsequent state dataset. Finally, the resulting performance value for the subsequent state can be considered when selecting control agents. In this way, a state and a control action can be extrapolated or predicted step by step into the future, allowing a control trajectory spanning several time steps to be determined. Such an extrapolation is often referred to as a rollout or virtual rollout. A cumulative performance over several time steps can then be calculated for the control trajectory and preferably assigned to the control action at the beginning of the trajectory. Such cumulative performance is often referred to as a "return" in the context of reinforcement learning.To calculate the return, the performance values ​​determined for future time steps can be discounted, i.e., they can be given weights that decrease for each time step.

[0018] An embodiment of the invention is explained in more detail below with reference to the drawings, each of which illustrates in schematic form: Figure 1 a control device according to the invention when controlling a technical system, Figure 2 training of a first machine learning module for the technical system, Figure 3 training a second machine learning module for the technical system, Figure 4 a determination of the performance of a control agent and Figure 5 an optimization of control agents for the technical system.

[0019] Insofar as the same or corresponding reference symbols are used in different figures, these reference symbols designate the same or corresponding entities, which can be implemented or configured in particular as described in connection with the relevant figure.

[0020] Figure 1 illustrates a control device CTL according to the invention controlling a technical system TS, e.g., a gas turbine, a wind turbine, a steam turbine, a chemical reactor, a milling machine, a machine tool, a manufacturing plant, a factory, a robot, a motor vehicle, an engine, a cooling system, a heating system, an energy plant, or another machine, device, or system. In particular, a component or subsystem of a composite technical system can also be considered a technical system TS.

[0021] The technical system TS has a sensor system SK for preferably continuously detecting and / or measuring system states or subsystem states of the technical system TS. For the present exemplary embodiment, it is assumed that a machine tool is controlled as the technical system TS.

[0022] The control device CTL is in Figure 1 external to the technical system TS and coupled with it. Alternatively, the control unit CTL can also be fully or partially integrated into the technical system TS.

[0023] The control device CTL has one or more processors PROC for executing method steps of the control device CTL as well as one or more memories MEM coupled to the processor PROC for storing the data to be processed by the control device CTL.

[0024] Furthermore, the control device CTL has one or more deterministic control agents P1, P2, ... for controlling the technical system TS. In the present exemplary embodiment, several control agents P1, P2, ... are provided, each implemented as an artificial neural network and trainable using reinforcement learning methods. Such a control agent is often also referred to as a policy agent or, for short, an agent. A deterministic control agent is characterized by the fact that it outputs the same output data for the same input data.

[0025] The control agents P1, P2, ... and thus the control device CTL are trained and / or selected in advance using training data and are thus configured for optimized control of the technical system TS. The training data for training and / or selecting the control agents P1, P2, ... are taken from a database DB, in which the training data is stored in the form of a large number of training data sets TD. The stored training data sets TD are recorded in advance on the technical system TS or a similar technical system and / or generated using simulation or data-driven methods.

[0026] Based on this training data, the control agents P1, P2, ... are trained and / or selected to determine a control action for a given state of the technical system TS that optimizes the performance of the technical system TS. The performance to be optimized can, in particular, relate to a power output, a yield, a speed, a weight, a runtime, a precision, an error rate, a resource consumption, a degree of effectiveness, an efficiency, a pollutant emission, a stability, a wear and tear, a service life, a physical property, a mechanical property, an electrical property, a constraint to be met, or other target variables of the technical system TS or one of its components that are to be optimized.

[0027] The process of this training is explained in more detail below.

[0028] After training is complete, the trained and / or selected control agents P1, P2, ... can be used for optimized control of the technical system TS. For this purpose, current operating states and / or other operating conditions of the technical system TS are continuously measured or otherwise determined by the sensor system SK and transmitted in the form of status data sets S from the technical system TS to the control device CTL. Alternatively or additionally, the status data sets S can be determined at least partially by simulation using a simulator, in particular a digital twin of the technical system TS.

[0029] A respective state data set S specifies a state of the technical system TS and is preferably represented by a numerical state vector. The state data sets S can include measurement data, sensor data, environmental data, or other data arising during the operation of the technical system TS or influencing its operation, in particular data on actuator positions, forces occurring, power, pressure, temperature, valve positions, emissions, and / or resource consumption of the technical system TS or one of its components. In production plants, the state data sets S can also relate to product quality or other product properties.

[0030] The state data sets S transmitted to the control device CTL are fed as input data to one or more of the trained control agents P1, P2, ... The control agents P1, P2, ... can be used alternately to control the technical system TS or can be controlled depending on a state or other operating conditions of the technical system TS.

[0031] Based on a supplied state data set S, a currently controlling control agent P1 or P2, ... generates a performance-optimizing control action in the form of an action data set A. The action data set A specifies a control action that can be performed on the technical system TS. In particular, the action data set A can be used to specify manipulated variables of the technical system TS, e.g., for adjusting a gas supply in a gas turbine or for executing a movement trajectory in a robot.

[0032] The generated action data records A are transmitted by the control unit CTL to the technical system TS and executed by it. In this way, the technical system TS is controlled in a manner optimized for the current operating state.

[0033] Figure 2 illustrates the training of a first machine learning module NN1 for the technical system TS. The first machine learning module NN1 is data-driven and is designed to model a state transition when applying a control action to a given state of the technical system TS. Such a machine learning module is often also referred to as a transition model or system model.

[0034] The first machine learning module NN1 can be implemented in the control device CTL or entirely or partially externally thereto. In the present exemplary embodiment, the first machine learning module NN1 is implemented as an artificial neural network, in particular as a feedforward neural network.

[0035] The first machine learning module NN1 is to be trained using the training data sets TD contained in the database DB to predict as accurately as possible a subsequent state of the technical system TS resulting from the application of the control action as well as a resulting performance value based on a respective state and a respective control action.

[0036] The first machine learning module NN1 is trained using the training data sets TD stored in the database DB. Each training data set TD comprises a state data set S, an action data set A, a subsequent state data set S', and a performance value R. As already mentioned above, the state data sets S each specify a state of the technical system TS, and the action data sets A each specify a control action that can be performed on the technical system TS. Accordingly, a respective subsequent state data set S' specifies a subsequent state resulting from the application of the respective control action to the respective state, i.e., a system state of the technical system TS assumed in a subsequent time step. The respective associated performance value R ultimately quantifies the respective performance of an execution of the respective control action in the respective state.As already indicated above, the performance value R can, in particular, refer to a resulting performance, a resulting emission value, a resulting resource consumption, a resulting product quality, and / or other operating parameters of the technical system TS resulting from the execution of the control action. In the context of machine learning, such a performance value is also referred to by the terms reward, or – complementarily – cost or loss.

[0037] To train the first machine learning module NN1, state data sets S and action data sets A are fed to it as input data. The first machine learning module NN1 should be trained so that its output data reproduces a resulting subsequent state and a resulting performance value as accurately as possible. Training is preferably carried out using a supervised machine learning method.

[0038] Training is generally understood as the optimization of a mapping of input data, here S and A, from a machine learning module to its output data. This mapping is optimized during a training phase according to predefined learned and / or to-be-learned criteria. Criteria that can be used include, in particular, a prediction error for prediction models and the success of a control action for control models. Through training, for example, the network structures of neurons in a neural network and / or the weights of connections between the neurons can be adjusted or optimized so that the predefined criteria are met as closely as possible. Training can thus be viewed as an optimization problem.For such optimization problems in the field of machine learning, a variety of efficient optimization methods are available, in particular gradient-based optimization methods, gradient-free optimization methods, backpropagation methods, particle swarm optimizations, genetic optimization methods and / or population-based optimization methods.

[0039] In particular, artificial neural networks, recurrent neural networks, convolutional neural networks, perceptrons, Bayesian neural networks, autoencoders, variational autoencoders, Gaussian processes, deep learning architectures, support vector machines, data-driven regression models, K-nearest neighbor classifiers, physical models or decision trees can be trained in this way.

[0040] In the case of the first machine learning module NN1, as mentioned above, state data sets S and action data sets A from the training data are fed to it as input data. For each pair (S, A) of input data sets, the first machine learning module NN1 outputs an output data set OS' as a predicted subsequent state data set and an output data set OR as a predicted performance value. The training of the first machine learning module NN1 aims to ensure that the output data sets OS' match the actual subsequent state data sets S' as well as the output data sets OR match the actual performance values ​​R as closely as possible.

[0041] For this purpose, a deviation D1 is determined between the output data sets (OS', OR) and the corresponding data sets (S', R) contained in the training data. The deviation D1 can be understood as a reproduction error or prediction error of the first machine learning module NN1. The reproduction error D1 can be determined, in particular, by calculating a Euclidean distance between the respective representing vectors, e.g., according to D1 = (OS' - S') 2< + (OR - R) 2< .

[0042] The reproduction error D1 is, as in Figure 2indicated by a dashed arrow, is fed back to the first machine learning module NN1. Using the fed-back reproduction error D1, the first machine learning module NN1 is trained to minimize the reproduction error D1, at least on average. A variety of efficient optimization methods are available for minimizing the reproduction error D1. By minimizing the reproduction error, the first machine learning module NN1 is trained to predict a resulting subsequent state and a resulting performance value as accurately as possible for a given state and a given control action.

[0043] Figure 3illustrates the training of a second machine learning module NN2 for the technical system TS. The second machine learning module NN2 can be implemented in the control device CTL or entirely or partially externally. In the present exemplary embodiment, the second machine learning module NN2 is implemented as an artificial neural network, in particular as a so-called feedforward network. The second machine learning module NN2 is data-driven and is to be trained using the training data sets TD stored in the database DB to predict an associated control action based on a respective state.

[0044] For this purpose, the state data sets S contained in the training data sets TD are fed to the second machine learning module NN2 as input data, from which the second machine learning module NN2 derives output data OA. The second machine learning module NN2 is to be trained such that the output data OA derived from a respective state data set S of a respective training data set TD reproduces the respective action data set A contained in the same training data set TD as accurately as possible, at least on average. For this purpose, a deviation D2 between the output data sets OA and the corresponding action data sets A is determined. The deviation D2 can be understood as the reproduction error of the second machine learning module NN2. The reproduction error D2 can be determined in particular by calculating a Euclidean distance between the respective representative vectors, e.g., according to D2 = (OA - A) 2< .

[0045] The reproduction error D2 is, as in Figure 3 indicated by a dashed arrow, is fed back to the second machine learning module NN2. Based on the fed-back reproduction error D2, the second machine learning module NN2 is trained to minimize this reproduction error D2, at least on average. The same optimization methods can be applied here as in the case of the first machine learning module NN1.

[0046] By minimizing the reproduction error D2, the second machine learning module NN2 is trained to output a corresponding control action for a given state.

[0047] Figure 4 illustrates a determination of the performance of a control agent P by a performance evaluator PEV. The performance evaluator PEV comprises the machine learning modules NN1 and NN2, which are trained as described above.

[0048] To illustrate successive processing steps, Figure 4 Several instances of the trained machine learning modules NN1 and NN2 are shown schematically. The different instances can, in particular, correspond to different calls or evaluations of routines by means of which the machine learning modules NN1 and NN2 are implemented.

[0049] To evaluate the performance of the control agent P, state data sets S are fed into the control agent P as input data. The resulting output data A of the control agent P, which can be interpreted as action data sets, is then transmitted to the performance evaluator PEV together with the corresponding state data sets S.

[0050] The performance evaluator PEV uses the trained machine learning modules NN1 and NN2 to predict an overall performance RET for a respective pair (S, A) of a state data set S and an action data set A. This overall performance quantifies the performance of the technical system TS resulting from the application of the control action A, accumulated over several time steps into the future. Such an overall performance is often referred to as "return" in the technical field of machine learning, especially reinforcement learning.

[0051] To determine the overall performance RET, the respective action data set A, together with the respective state data set S, is fed into the trained first machine learning module NN1, which then predicts a subsequent state and outputs a subsequent state data set S' specifying this, as well as an associated performance value R. The subsequent state data set S' is in turn fed into the trained second machine learning module NN2, which then derives an action data set A' for the subsequent state. The action data set A', together with the respective subsequent state data set S', is fed into another instance of the first machine learning module NN1, which then predicts another subsequent state and outputs a subsequent state data set S" specifying this, as well as an associated performance value R'.

[0052] The above process steps can be repeated iteratively, with performance values ​​being determined for subsequent states. The iteration can be terminated if a termination condition is met, e.g., if a predefined number of iterations is exceeded. In this way, a control trajectory spanning several time steps and progressing from subsequent state to subsequent state with associated performance values ​​R, R', R'', ... can be determined.

[0053] The determined performance values ​​R, R', R", ... are fed into a performance function PF.

[0054] The performance function PF determines the total performance RET accumulated over a given control trajectory. The determined total performance RET is then assigned to the control action, here A, at the beginning of the control trajectory. The total performance RET thus evaluates the ability of the control agent P to determine a favorable, performance-optimizing control action, here A, for a given state data set S. The total performance RET is preferably calculated by the performance function PF as the performance discounted over the future time steps of a control trajectory.

[0055] For this purpose, the performance function PF preferably calculates a weighted sum of the performance values ​​R, R', R'', ..., whose weights are multiplied with each time step into the future by a discount factor W < 1. In this way, the overall performance RET can be calculated according to RET = R + R' * W + R" * W 2< + ... For the discount factor W, for example, a value of 0.99 or 0.9 can be used.

[0056] The predicted overall performance RET is finally output by the performance evaluator PEV as an evaluation result.

[0057] Figure 5 illustrates an optimization of control agents for the technical system TS. For this purpose, a large number of training data sets TD are fed to the control unit CTL, which are then used to train the machine learning modules NN1 and NN2, as described above.

[0058] The control device CTL further comprises a model generator MG for generating deterministic control agents. The latter are preferably implemented as artificial neural networks. As such, the control agents are specified by a connection structure of their neurons as well as by weights of connections between the neurons. A respective control agent can thus be generated by the model generator MG by generating and outputting a set of neural weights and / or a data set specifying a neuron connection. For the present embodiment, it is assumed that the model generator MG generates the different deterministic control agents in the form of different sets of neural weights.

[0059] During the generation of control agents, a comparison CMP1 is performed between various generated control agents. During this comparison CMP1, a distance between the respective control agents is determined, which quantifies the difference or similarity between the respective control agents. Preferably, a possibly weighted Euclidean distance between the neural weights of the compared control agents is calculated as the distance. Depending on the determined distances, the model generator MG preferentially outputs those control agents that are further apart from other generated or output control agents. In this way, control agents that exhibit a high degree of diversity or dissimilarity can be output.This fundamentally enables better exploration of areas of a technical system's state space that are less covered by training data. Further methods for generating control agents with high diversity are known, for example, from the publication "Promoting Diversity in Particle Swarm Optimization to Solve Multimodal Problems" by Cheng S., Shi Y., and Qin Q. in Neural Information Processing (eds: Lu BL., Zhang L., Kwok J.), ICONIP 2011, Lecture Notes in Computer Science, vol. 7063, Springer, Berlin, Heidelberg, (https: / / doi.org / 10.1007 / 978-3-642-24958-7_27).

[0060] In the present embodiment, the model generator MG generates a plurality of different deterministic control agents P1, P2, ... A plurality of state data sets S of the training data are fed to the generated control agents P1, P2, ... as input data. For each state data set S, the control agent P1 outputs a respective action data set A1, and the control agent P2 outputs a respective action data set A2. The other generated control agents output further action data sets analogously.

[0061] Each action data set A1 from the control agent P1 is fed into the performance evaluator PEV, which predicts an overall performance value RET1 for this action data set A1, as described above. Similarly, the performance evaluator PEV determines a corresponding overall performance value RET2 for each action data set A2. Action data sets from other generated control agents are processed analogously.

[0062] The predicted overall performance values ​​RET1, RET2, etc. are fed by the performance evaluator PEV into a selection module SEL of the control unit CTL. The selection module SEL serves the purpose of selecting performance-optimizing control agents SP1, ..., SPK from the generated control agents P1, P2, ...

[0063] For this purpose, the control agents P1, P2, ...and the trained machine learning module NN2 in the form of their neural weights (not shown). Based on the neural weights, the selection module SEL performs a comparison CMP2 between a respective control agent P1, P2, ... and the trained machine learning module NN2. In this comparison CMP2, a distance between a respective control agent P1, P2, ... and the trained machine learning module NN2, which determines a difference or similarity of the respective control agent P1, P2, ... to the trained machine learning module NN2.

[0064] The determined distances are each compared with a predefined threshold TH. If a respective distance exceeds the threshold TH, the respective control agent is excluded from the selection by the selection module SEL. In this way, only those of the control agents P1, P2, ...selected that do not differ too much from the trained machine learning module NN2 and thus exhibit similar control behavior. To the extent that the trained machine learning module NN2 is trained to output control actions stored in the training data, control agents that output inadmissible or severely detrimental control actions can be excluded.

[0065] From the remaining set of control agents similar to the machine learning module NN2, the selection module SEL continues to select those control agents that exhibit, at least on average, the highest overall performance values. In the present embodiment, a number of K highest-performing control agents SP1, ... SPK are selected and output by the selection module SEL in this way.

[0066] The select control agents SP1, ... SPK will then, as in connection with Figure 1As explained, they are used to control the technical system TS. The selected control agents SP1, ... SPK can, in particular, take over control of the technical system TS alternately and / or cyclically. As long as the selected control agents SP1, ... SPK are deterministic control agents, the technical system TS is controlled deterministically, which significantly simplifies validation of the control compared to stochastic control. At the same time, the diversity of the selected control agents SP1, ... SPK allows for efficient exploration of the state space or action space of the technical system TS.

[0067] During the control of the technical system TS by the selected control agents SP1, ... SPK, further state data records ES, action data records EA, subsequent state data records ES' and performance values ​​ER for the technical system TS are recorded by the sensor system SK and, if necessary, by additional recording devices and added to the training data in the database DB.

[0068] With the training data thus supplemented, the training of the machine learning modules NN1 and NN2, the resulting generation and selection of the deterministic control agents, and the supplementation of the training data are then iteratively repeated. The selected control agents SP1, ... SPK can be transmitted to the model generator MG in order to drive the generation of control agents in the next iteration step toward the selected control agents SP1, ... SPK, in the sense of a genetic or population-based optimization process. Furthermore, the threshold TH can be successively increased from iteration to iteration in order to explore larger areas of the state space or control action space of the technical system TS. In this way, the state space or control action space can be efficiently explored and enriched with operationally relevant training data.

[0069] In particular, the above iteration can be run during regular operation of the technical system TS to continuously adapt control agents to environmental or wear-related changes in the technical system TS in an optimized manner. Furthermore, the training data can be continuously supplemented in a manner representative of these changes.

[0070] In addition to a selection of the control agents P1, P2, ..., they can also be trained during operation using the training data sets TD to determine a performance-optimizing control action in the form of an action data set for a given state data set S. For this purpose, as in Figure 5 As indicated by dashed arrows, the total performance values ​​RET1 are returned to the control agent P1 and the total performance values ​​RET2 are returned to the control agent P2. The same applies to other generated control agents.

[0071] The processing parameters, in this case the neural weights of a respective control agent P1, P2,..., are each adjusted or configured in such a way that the respective overall performance values ​​are maximized, at least on average. A variety of efficient standard methods can be used to implement the training. The training described above can, in many cases, further improve the performance of the selected control agents SP1, ..., SPK.

Claims

1. Computer-implemented method for controlling a technical system (TS), wherein a) training data are read in, a respective training dataset (TD) comprising a state dataset (S) that specifies a state of the technical system, an action dataset (A) that specifies a control action, and a performance value (R) of the technical system that results from an application of the control action, b) a first machine learning module (NN1) is trained, using the training data, to use a state dataset (S) and an action dataset (A) to reproduce a resulting performance value (R), c) a multiplicity of different deterministic control agents (P1, P2,...) are each supplied with state datasets (S), and resulting output data are fed into the trained first machine learning module (NN1) as action datasets, d) performance values output by the trained first machine learning module (NN1) are taken as a basis for selecting multiple instances of the control agents, e) the technical system is respectively controlled by the selected control agents (SP1, ..., SPK), with further state datasets (ES), action datasets (EA) and performance values (ER) being captured and added to the training data, and f) method steps b) to e) are repeated using the thus augmented training data.

2. Method according to Claim 1, characterized in that - a second machine learning module (NN2) is trained, or is trained using the training data, to use a state dataset (S) to reproduce an action dataset (A), - the control agents (P1, P2,...) are each compared with the second machine learning module (NN2), a respective distance that quantifies a dissimilarity between the respective control agent (P1, P2,...) and the second machine learning module (NN2) being ascertained, and - a control agent having a lesser distance from the second machine learning module (NN2) is selected preferentially over a control agent having a greater distance.

3. Method according to Claim 2, characterized in that the distance of a respective control agent is compared with a threshold value (TH), and in that the respective control agent is excluded from the selection if the threshold value (TH) is exceeded.

4. Method according to Claim 3, characterized in that the threshold value (TH) is increased when the method steps are repeated.

5. Method according to one of Claims 2 to 4, characterized in that the second machine learning module (NN2) and the control agents (P1, P2,...) are artificial neural networks, and in that a divergence between neural weights of the second machine learning module (NN2) and neural weights of a respective control agent is ascertained and is quantified by the respective distance.

6. Method according to one of the preceding claims, characterized in that - the multiplicity of control agents (P1, P2,...) are generated by a model generator (MG), - multiple instances of the generated control agents are compared with other control agents, a respective distance that quantifies a dissimilarity between the compared control agents being ascertained, and - a control agent having a greater distance from one or more other control agents is selected preferentially over a control agent having a lesser distance.

7. Method according to one of the preceding claims, characterized in that the control agents (P1, P2,...) are trained, using the training data, to use a state dataset (S) to reproduce an action dataset that specifies a performance-optimizing control action.

8. Method according to one of the preceding claims, characterized in that - a respective training dataset (TD) comprises a subsequent state dataset (S') that specifies a subsequent state of the technical system (TS) resulting from an application of a control action, - the first machine learning module (NN1) is trained, using the training data, to use a state dataset (S) and an action dataset (A) to reproduce a resulting subsequent state dataset (S'), - a second machine learning module (NN2) is trained, using the training data, to use a state dataset (S) to reproduce an action dataset (A), - the trained first machine learning module (NN1) ascertains a subsequent state dataset (S') for an action dataset output by the respective control agent and feeds said subsequent state dataset into the trained second machine learning module (NN2), - a resultant subsequent action dataset (A') is fed into the trained first machine learning module (NN1) together with the subsequent state dataset (S'), and - a resultant performance value (R') for the subsequent state is taken into consideration for the selection of the control agents (P1, P2,...).

9. Method according to one of the preceding claims, characterized in that the control agents (P1, P2,...) are selected and / or generated using a population-based optimization method, a gradient-free optimization method, a particle swarm optimization and / or a genetic optimization method.

10. Method according to one of the preceding claims, characterized in that the first machine learning module (NN1), a second machine learning module (NN2) and / or the control agents (P1, P2,...) comprise an artificial neural network, a recurrent neural network, a convolutional neural network, a multilayer perceptron, a Bayesian neural network, an autoencoder, a variational autoencoder, a Gaussian process, a deep learning architecture, a support vector machine, a data-driven trainable regression model, a k nearest neighbours classifier, a physical model and / or a decision tree.

11. Method according to one of the preceding claims, characterized in that the technical system (TS) is a gas turbine, a wind turbine, a steam turbine, a chemical reactor, a milling machine, a machine tool, a production plant, a factory, a robot, a motor, a cooling plant, a heating plant or another machine, another device or another plant.

12. Control device (CTL) for controlling a technical system (TS) having means for carrying out the steps of a method according to one of the preceding claims.

13. Computer program product comprising commands which cause a control device according to Claim 12 to carry out a method according to one of claims 1 to 11 when the program is run by a computer.

14. Computer-readable storage medium having a computer program product according to Claim 13.