METHOD FOR CONTROLLING A MACHINE BY A LEARNING-BASED CONTROL AGENT AND CONTROL DEVICE

Description

[0001] Data-driven machine learning methods, particularly reinforcement learning, are increasingly used to control complex technical systems such as robots, motors, production plants, power supply systems, gas turbines, wind turbines, steam turbines, milling machines, and other machinery. In these methods, control agents, especially artificial neural networks, are trained using large amounts of training data to generate a control signal optimized for a given state of the technical system according to a predefined objective function. Such a control agent trained to control a technical system is often referred to as a policy or simply as an agent.

[0002] The objective function allows for the evaluation of performance to be optimized, such as power output, efficiency, resource consumption, yield, emissions, product quality, and / or other operating parameters of the technical system. Such an objective function is also frequently referred to as a reward function, cost function, or loss function.

[0003] Using such a performance-optimizing control agent can significantly increase the performance of a controlled technical system in many cases. However, the productive use of such a control agent also carries risks because the generated control signals are often untested and therefore potentially unreliable. This is especially true under operating conditions that are not adequately covered by the training data. Furthermore, a user is often less familiar with such an optimized control sequence and therefore less able to utilize their expertise to monitor the correct behavior of the technical system.

[0004] EP 3 940 596 A1 discloses a computer-implemented method for training a learning-based control agent for a technical system, wherein the control agent is configured by the training to control the technical system in an optimized manner.

[0005] US 2015 / 370227 A1 discloses a method for controlling a target system by a weighted combination of several control strategies.

[0006] The object of the present invention is to provide a method and a control device for controlling a machine by means of a learning-based control agent, which allow for more efficient and / or more reliable control.

[0007] This problem is solved by a method with the features of claim 1, by a control device with the features of claim 11, by a computer program product with the features of claim 12, and by a computer-readable storage medium with the features of claim 13.

[0008] To control a machine using a learning-based control agent, a performance evaluator and an action evaluator are provided. The performance evaluator determines the performance for controlling the machine using a control signal, and the action evaluator determines any deviation from a predefined control sequence based on the same control signal. Furthermore, a multitude of weight values ​​are generated to weight the performance against the deviation. These weight values, along with a multitude of status signals, are fed into the control agent. A resulting output signal from the control agent is fed as a control signal into both the performance evaluator and the action evaluator. The performance determined by the performance evaluator is weighted against a deviation determined by the action evaluator according to the respective weight value using an objective function. The control agent is then trained to output a control signal optimizing the objective function based on a state signal and a weight value. To control the machine, an operating weight value and an operating state signal of the machine are fed into the trained control agent, and a resulting output signal from the trained control agent is fed to the machine. During machine control, the weight value is successively changed such that the performance is weighted increasingly more heavily than the deviation.

[0009] To carry out the method according to the invention, a corresponding control device, a computer program product and a computer-readable, preferably non-volatile storage medium are provided.

[0010] The method and control device according to the invention can be carried out or implemented, for example, by means of one or more computers, processors, application-specific integrated circuits (ASICs), digital signal processors (DSPs) and / or so-called "Field Programmable Gate Arrays" (FPGAs).

[0011] A particular advantage of the invention lies in the fact that, by simply entering or changing an operating weight value, different control strategies can be set during the operation of machine M. These strategies are optimized for different weightings of performance versus deviations from a predefined control sequence. In this way, the trained control agent can be temporarily configured during operation to prioritize proximity to the predefined control sequence over machine performance. This can be particularly advantageous during safety-critical operating phases of the machine. Conversely, in less critical operating phases, performance can be weighted more heavily than proximity to the predefined control sequence to favor performance optimization.

[0012] In particular, at the beginning of a control system, for example during the commissioning of the trained control agent or the machine, deviations from the predefined control sequence can initially be weighted more heavily than performance. This reduces operational risk due to untested control sequences. If and to the extent that the control sequences then prove reliable, performance optimization can be gradually prioritized, thus generally increasing performance.

[0013] Advantageous embodiments and further developments of the invention are specified in the dependent claims.

[0014] According to a further advantageous embodiment of the invention, the performance evaluator can determine the respective performance and / or the action evaluator can determine the respective deviation based on a respective state signal. In many cases, the additional consideration of a state signal allows for a more precise determination of the respective performance and / or deviation.

[0015] Furthermore, a performance value can be read in for a multitude of state signals and control signals, quantifying the performance resulting from applying a respective control signal to a machine state specified by a respective state signal. The performance evaluator can then be trained to reproduce a corresponding performance value based on a state signal and a control signal. In this way, a performance evaluator can be implemented using machine learning methods, particularly supervised learning.

[0016] Furthermore, the performance evaluator can be trained to determine cumulative performance over a future period using a Q-learning method and / or another Q-function-based reinforcement learning method. A variety of efficient numerical optimization methods are available for implementing Q-learning and Q-function-based methods.

[0017] According to a further advantageous embodiment of the invention, a plurality of state signals and control signals can be read in. The action evaluator can then be trained to reproduce the control signal based on a state signal and a control signal after information reduction, whereby a reproduction error is determined. The deviation can then be determined based on the reproduction error. The preceding feature is based on the observation that reproduction errors in state and control action domains well covered by training data are often smaller than in poorly covered domains. Thus, the reproduction error can generally be understood as a measure of the extent to which a respective state-specific control action is covered by the training data.Since the training data is based on a predefined control sequence in many cases, the reproduction error can be used as a relatively accurate measure of the deviation from the predefined control sequence.

[0018] Preferably, the deviation can be determined by the action evaluator using a variational autoencoder, a generative opponent network, and / or by a comparison with predefined control signals, particularly one dependent on the state signal. Using an autoencoder, or in particular a variational autoencoder, allows for a more information-reduced and / or less redundant representation of the input state signals and / or control signals. Furthermore, a measure of how precisely the control signals, in particular, can be reconstructed from the reduced quantities can be easily determined. As already indicated above, the reproducibility of control signals can be used as a measure of the deviation from a predefined control sequence.

[0019] Furthermore, a gradient-based optimization method, such as gradient descent, stochastic optimization, particle swarm optimization, and / or genetic optimization, can preferably be used to train the control agent, performance evaluator, and / or action evaluator. A variety of efficient implementations are available for these methods.

[0020] An embodiment of the invention is explained in more detail below with reference to the drawing. The drawings illustrate, in schematic form: Figure 1 shows a control device according to the invention when controlling a machine, Figure 2 shows a training of a transition model for performance determination, Figure 3 shows a training of an action evaluator, and Figure 4 shows a training of a control agent of the control device.

[0021] Insofar as the same or corresponding reference symbols are used in different figures, these reference symbols denote the same or corresponding entities, which may be described, implemented or designed in particular as in connection with the figure in question.

[0022] Figure 1 Figure 1 illustrates a control device CTL according to the invention when controlling a machine M, e.g., a robot, a motor, a production plant, a factory, a power supply unit, a gas turbine, a wind turbine, a steam turbine, a milling machine, or another device or system. In particular, a component or subsystem of a machine or system can also be considered a machine M.

[0023] The machine M has a sensor system SK for preferably continuously detecting and / or measuring system states or subsystem states of the machine M.

[0024] The CTL control unit is in Figure 1 The control unit CTL is displayed externally to machine M and coupled to it. Alternatively, the control unit CTL can also be fully or partially integrated into machine M.

[0025] The CTL control unit has one or more PROC processors for executing process steps of the CTL control unit, as well as one or more MEM memory units coupled to the PROC processor for storing the data to be processed by the CTL control unit.

[0026] Furthermore, the CTL control unit features a learning-based control agent POL, which is trained or trainable using reinforcement learning methods. Such a control agent is often also referred to as a policy or simply as an agent. In the present embodiment, the control agent POL is implemented as an artificial neural network.

[0027] The control agent POL, and thus the control unit CTL, is trained in advance using predefined training data and configured for optimized control of machine M. Optionally, the control agent POL can also be further trained during the operation of machine M, using status signals, control signals, and / or performance data from the running machine M as training data.

[0028] The training of the control agent POL focuses in particular on two control criteria. On the one hand, the machine M controlled by the control agent POL should achieve the highest possible performance, while on the other hand, the control should not deviate too much from a predefined control sequence SP.

[0029] Performance can refer to, for example, power output, efficiency, resource consumption, yield, pollutant emissions, product quality and / or other operating parameters of machine M.

[0030] A predefined control sequence SP can be specified, for example, by predefined sequences of states and / or control actions. Furthermore, a predefined control sequence SP can also be considered during training by using training data from machine M, a similar machine, or a simulation of machine M while it was controlled by a reference control agent. In this case, the coverage of the states and / or control actions traversed by machine M when controlled by the control agent POL by the training data can serve as a criterion for the controller's proximity to the reference control agent and thus to the predefined control sequence SP. Preferably, a verified, validated, and / or rule-based control agent can be used as the reference control agent.

[0031] According to the invention, the control agent POL is to be trained such that the relative weighting of the two optimization criteria, performance and proximity to the predefined control sequence SP, can be changed by a weight value during the operation of the trained control agent POL. A procedure for this training is explained in more detail below.

[0032] To optimize the control of machine M by the trained control agent POL, operating state signals SO, i.e., state signals determined during the operation of machine M, are transmitted to the control unit CTL. The operating state signals SO each specify a state, in particular an operating state of machine M, and are preferably represented by a numerical state vector or a time series of state values ​​or state vectors.

[0033] The operating status signals SO can represent measurement data, sensor data, environmental data, or other data generated during or influencing the operation of machine M. Examples include data on temperature, pressure, settings, actuator positions, valve positions, pollutant emissions, utilization, resource consumption, and / or the performance of machine M or its components. In a production plant, the operating status signals SO can also relate to product quality or other product characteristics. The operating status signals SO can be measured, at least partially, by sensors SK or determined through simulation using a machine simulator.

[0034] Furthermore, the control unit CTL reads in an operating weight value WO, e.g., from a user USR of machine M. This operating weight value WO weights the performance of machine M against any deviation of machine M, controlled by the control agent POL, from the predefined control sequence SP. Alternatively, the operating weight value WO can also be generated by the control unit CTL depending on an operating condition or operating phase of machine M.

[0035] The operating status signals SO transmitted to the control unit CTL are fed into the trained control agent POL as input signals, along with the operating weight value WO. Based on each supplied operating status signal SO and operating weight value WO, the trained control agent POL generates a corresponding optimized operating control signal AO. The latter specifies one or more control actions to be performed on the machine M. The generated operating control signals AO are transmitted from the control agent POL or the control unit CTL to the machine M. The transmitted operating control signals AO control the machine M in an optimized manner by executing the control actions specified by the operating control signals AO.

[0036] The operating weight value WO allows the trained control agent POL to be temporarily configured during machine M's operation to prioritize proximity to the predefined control sequence SP over machine M's performance when determining optimized operating control signals AO. This can be particularly advantageous during safety-critical operating phases of machine M. Conversely, in less critical operating phases, performance can be weighted more heavily than proximity to the predefined control sequence SP to favor performance optimization.

[0037] In particular, at the beginning of a control system, for example during the commissioning of the trained control agent POL or the machine M, the proximity to the predefined control sequence SP can initially be weighted more highly than performance. This reduces operational risk due to untested control sequences. If and to the extent that the control sequences then prove reliable, performance optimization can be gradually prioritized, thus generally increasing performance.

[0038] Figure 2 This illustrates the training of a transition model NN to determine the performance of machine M. The transition model NN is part of a performance evaluator PEV, which is explained in more detail below.

[0039] The transition model NN is data-driven and trainable, and is designed to model a state transition when a control action is applied to a predefined state of the machine M. Such a transition model is often also referred to as a dynamic model or system model. The transition model NN can be implemented within the control unit CTL or wholly or partially externally. In the present embodiment, the transition model NN is implemented as an artificial neural network.

[0040] The transition model NN is to be trained using training data sets TD contained in a database DB to predict as accurately as possible a subsequent state of the machine M resulting from the application of the control action and a resulting performance value, based on a given state and a given control action.

[0041] In the present embodiment, the training data sets TD originate from a controller of machine M, a similar machine, or a simulation of machine M by a reference control agent. As already indicated above, the control operations performed by the reference control agent serve, among other things, to specify a predefined control sequence SP, which is reflected in the training data sets TD.

[0042] The training data sets TD each comprise a state signal S, a control signal A, a subsequent state signal S', and a performance value R. As mentioned above, the state signals S each specify a state of the machine M, and the control signals A each specify a control action that can be performed on the machine M. Accordingly, each subsequent state signal S' specifies a subsequent state of the machine M resulting from applying the respective control action to the respective state. In particular, the same operating data of the machine M can be specified by a state signal S and a subsequent state signal S' as by the operating state signals SO mentioned above. Finally, the associated performance value R quantifies the performance of executing the respective control action in the respective state.The performance value R can relate in particular to the currently resulting power, efficiency, resource consumption, yield, emissions, product quality, and / or other operating parameters of the machine M resulting from the current control action. In the context of machine learning, such a performance value is also referred to as a reward. -- Complementary to this - referred to as costs or losses.

[0043] To train the transition model NN, state signals S and control signals A are fed into the transition model NN as input signals. The transition model NN should be trained so that its output signals reproduce each resulting subsequent state and each resulting performance value as accurately as possible. Training is preferably carried out using a supervised machine learning method.

[0044] In this context, training is generally understood as the optimization of a mapping of input signals, here S and A, of a machine learning model, here NN, to its output signals. This mapping is optimized during a training phase according to predefined, learned, and / or to-be-learned criteria. For control models, these criteria can include the success of a control action, or for prediction models, a prediction error. Through training, for example, the network structures of neurons in a neural network and / or the weights of connections between neurons can be adjusted or optimized to best meet the predefined criteria. Training can thus be understood as an optimization problem. A multitude of efficient optimization methods are available for such optimization problems in the field of machine learning.Optimization is always understood to mean approaching an optimum.

[0045] In particular, artificial neural networks, recurrent neural networks, convolutional neural networks, perceptrons, Bayesian neural networks, autoencoders, variational autoencoders, deep learning architectures, support vector machines, data-driven regression models, k-nearest neighbor classifiers, physical models, or decision trees can be trained.

[0046] In the case of the NN transition model, state signals S and control signals A from the training data are fed into it as input signals, as mentioned above. For each input data set (S, A) containing a state signal S and a control signal A, the NN transition model outputs an output signal OS` as the next state signal and an output signal OR as a performance value.

[0047] The goal of training the transition model NN is to ensure that the output signals OS' correspond as closely as possible to the actual successor state signals S', and that the output signals OR correspond as closely as possible to the actual performance values ​​R, at least on average. To this end, a deviation DP is determined between each output signal pair (OS', OR) and its corresponding pair (S', R) contained in the training data. The deviation DP represents a prediction error of the transition model NN. The deviation DP can be determined, for example, by calculating a Euclidean distance between the respective representing vectors, e.g., according to DP = (OS' - S')² + (OR - R)².

[0048] The deviation DP is, as in Figure 2The deviation, indicated by a dashed arrow, is fed back to the transition model NN. Based on the returned deviation DP, the transition model NN is trained to minimize this deviation DP and thus the prediction error, at least on average. A variety of efficient optimization methods are available for minimizing the deviation DP, such as gradient-based optimization, stochastic gradient descent, particle swarm optimization, and / or genetic optimization. By minimizing the deviation DP, the transition model NN is trained to predict, as accurately as possible, a resulting subsequent state and a resulting performance value for a given state and control action.

[0049] Figure 3This illustrates data-driven training of an action evaluator (VAE). The action evaluator (VAE) can be implemented in the control unit (CTL) or wholly or partially externally.

[0050] The action evaluator (VAE) is used to determine a deviation of a controller from a predefined control sequence (SP) and is implemented in this embodiment by a variational autoencoder. As mentioned above, the predefined control sequence (SP) is defined, in a sense, by the training data obtained using the reference control agent.

[0051] In the present embodiment, the variational autoencoder VAE serves to evaluate the reproducibility of each state-specific control action. This reproducibility proves to be a relatively accurate measure of the similarity of a given state-specific control action to the control actions present in the training data, or as a measure of the probability of its occurrence in the training data.

[0052] In this embodiment, the variational autoencoder VAE is implemented as a feed-forward neural network and comprises an input layer IN, a hidden layer H, and an output layer OUT. In addition to the hidden layer H, the variational autoencoder VAE may have further hidden layers. A characteristic of an autoencoder is that the hidden layer H is significantly smaller, i.e., it contains fewer neurons than the input layer IN or the output layer OUT.

[0053] The variational autoencoder VAE is to be trained using training datasets TD read from the database DB. Based on input state signals S and control signals A, and after information reduction due to the smaller hidden layer H, the VAE aims to reproduce the input control signals A as accurately as possible, at least on average. In particular, the reproduction error DR is also to be determined.

[0054] For this purpose, the state signals S and control signals A from the training data are fed into the input layer IN as input signals and processed by the layers IN, H, and OUT. The processed data is then output by the output layer OUT as output signals OA, which should reproduce the input control signals A as accurately as possible according to the training objective. Preferably, state signals S are also fed into the hidden layer H.

[0055] Since the input signals, here S and A, must pass through the smaller hidden layer H and be largely reconstructible from the smaller or reduced data set present there, according to the training objective, a data-reduced representation of the input signals is obtained in the hidden layer H. The variational autoencoder VAE thus learns an efficient encoding or compression of the input signals. By also feeding state signals S directly into the hidden layer H, it can be achieved that the control signals A, in particular, are encoded, reduced, or represented more effectively in the hidden layer H.

[0056] In the hidden layer H, the training process realizes a so-called latent parameter space or a latent representation, in particular of the control signals A and the control actions specified by them. The data present in the hidden layer H correspond to an abstract and information-reduced representation of the state-specific control actions contained in the training data.

[0057] To achieve the above training goal, an optimization procedure is performed that adjusts the processing parameters of the variational autoencoder VAE such that the reproduction error DR is minimized. The reproduction error, or reproduction uncertainty, DR, can be determined in particular as a difference between the output signals OA and the input control signals A, according to DR = (OA - A) 2< .

[0058] To train the variational autoencoder VAE or to optimize its processing parameters, the calculated distances DR are used – as in Figure 3 Indicated by a dashed arrow – traced back to the variational autoencoder VAE. A variety of efficient standard procedures can be used for the actual execution of the training.

[0059] After successful training, the trained variational autoencoder VAE can be used to evaluate any pair of state signals S and control signals A by assessing how well the respective control signal A can be reconstructed from the information-reduced representation in the hidden layer H. It is expected that state-control-action pairs that occur frequently in the training data will exhibit a lower reproduction error DR than those that occur rarely. The reproduction error DR of the trained variational autoencoder VAE can therefore be used as a measure of how well a given state-control-action pair is covered by the training data, i.e., how frequently or how likely it occurs there, or how much it deviates from the predefined control sequence SP.

[0060] Figure 4This illustrates the training of the control agent POL using the trained transition model NN and the trained action evaluator VAE. To illustrate the successive work steps, the following are shown in Figure 4 Each diagram shows multiple instances of the control agent POL, the trained transition model NN, and the trained variational autoencoder VAE. These different instances can correspond to various calls or evaluations of routines by which the control agent POL, the trained transition model NN, or the trained action evaluator VAE is implemented.

[0061] As mentioned above, the control agent POL is to be trained to output a control signal A for each state signal S of the machine M. This control signal is optimized both with respect to the resulting performance of the machine M and with respect to its proximity to, or deviation from, the predefined control sequence SP. Obviously, proximity to the predefined control sequence SP can also be represented by a negatively weighted deviation from the predefined control sequence SP, and vice versa. The optimization aims for higher performance and greater proximity to, or smaller deviation from, the predefined control sequence SP. The weighting of the two optimization criteria, performance and proximity / deviation, should be adjustable via a weight value W.

[0062] For the present embodiment, it is assumed that the weight value W is adjustable between 0 and 1. With a weight value of W=1, the trained control agent POL should output a control signal that optimizes performance exclusively, while with a weight value of W=0, a control signal that minimizes deviations from the predefined control sequence SP should be output. For weight values ​​between 0 and 1, the two optimization criteria should be weighted proportionally.

[0063] To train the control agent POL for different weight values, a generator GEN of the control unit CTL preferably generates a large number of weight values ​​W randomly. In the present embodiment, the generated weight values ​​W are in the interval from 0 to 1.

[0064] According to the invention, the weight values ​​W generated by the generator GEN are fed into both the control agent POL to be trained and into a target function TF to be optimized by the training.

[0065] Furthermore, for the training of the control agent POL, large quantities of state signals S from the database DB are fed into the control agent POL as input signals.

[0066] In parallel, the state signals S are also fed into the trained transition model NN and the trained action evaluator VAE as input signals.

[0067] Each state signal S specifies a state of the machine M. Based on this state, the trained transition model NN predicts subsequent states of the machine M resulting from the application of control actions. Furthermore, the trained variational autoencoder VAE evaluates the respective state and its resulting subsequent states.

[0068] The control agent POL derives an output signal A from the respective state signal S and outputs it as a control signal. This control signal A, along with the respective state signal S, is then fed into the trained transition model NN, which predicts a subsequent state and outputs a subsequent state signal S1 specifying this state, as well as an associated performance value R1.

[0069] Furthermore, the control signal A, together with the respective state signal S, is fed into the trained variational autoencoder VAE, which determines and outputs a reproduction error D0 for the control signal A.

[0070] The subsequent state signal S1 is fed into the control agent POL, which derives a control signal A1 for the subsequent state from it. The control signal A1, together with the respective subsequent state signal S1, is fed into the trained transition model NN, which predicts a further subsequent state and outputs a subsequent state signal S2 specifying this state, as well as an associated performance value R2.

[0071] Furthermore, the control signal A1, together with the subsequent state signal S1, is fed into the trained variational autoencoder VAE, which determines and outputs a reproduction error D1 for the control signal A1.

[0072] As mentioned above, the reproduction errors D0 and D1 can be used as a measure of the deviation of the control signals A and A1 from the specified control sequence SP.

[0073] The preceding procedural steps can be repeated iteratively, determining performance values ​​and reproduction errors for subsequent states. The iteration can be terminated if a termination condition is met. eg Upon exceeding a predetermined number of iterations, a control trajectory spanning several time steps and progressing from successive state to successive state, extrapolated into the future, can be determined, along with associated performance values ​​R1, R2, ... and reproduction errors D0, D1, ... Such an extrapolation is often referred to as a roll-out or virtual roll-out.

[0074] From the performance values ​​R1, R2, ... of a given control trajectory, a cumulative overall performance RET of that control trajectory is determined over several time steps. Such a cumulative overall performance is often referred to as the return in the context of reinforcement learning. The overall performance RET is preferably assigned to the respective state signal S and / or control signal A at the beginning of the respective control trajectory and thus evaluates the ability of the control agent POL to determine, for the respective state signal S, a control signal A that initiates a control sequence with high performance over several time steps.

[0075] To determine the overall performance RET, the performance values ​​R1, R2,... determined for future time steps are discounted, i.e., assigned decreasing weights for each time step. In this embodiment, the overall performance RET is calculated as the weighted sum of the performance values ​​R1, R2, ..., whose weights are multiplied by a discount factor G < 1 for each future time step. In this way, the overall performance RET can be determined according to RET = R1 + R2*G + R3*G 2< + .... For the discount factor G, a value of 0.99, 0.9, 0.8, or 0.5, for example, can be used.

[0076] The transition model NN and the above discounting procedure together form a performance evaluator PEV, which determines a total performance RET of the machine M resulting from the application of the control signals A, A1,... and state signals S, S1,....

[0077] Alternatively or additionally, the performance evaluator PEV can also be implemented using a Q-learning method and trained to determine the total cumulative performance RET over a future period.

[0078] Furthermore, a total reproduction error D, cumulative over several time steps, is determined from the reproduction errors D0, D1, ... of the respective control trajectory. This total reproduction error D then serves as a measure of the deviation of this control trajectory from the predetermined control sequence SP. In the present embodiment, the total reproduction error D is calculated as the sum of the individual reproduction errors D0, D1, ... according to D = D1 + D2 + ....

[0079] The total performance RET and the total reproduction error D determined in each case are fed together into the objective function TF to be optimized.

[0080] The objective function TF weights the determined overall performance RET and the determined overall reproduction error D according to the same weight value W that was also fed into the control agent POL along with the respective state signal S. Through training that optimizes the objective function TF, the control agent POL can therefore be trained to output a control signal, given a state signal and a weight value, that optimizes the objective function TF according to the input weight value, at least on average.

[0081] In the present embodiment, the objective function TF determines a target value TV from a respective overall performance RET, a respective overall reproduction error D, and a respective weight value W, according to TV = W*RET - (1-W)*D. Since the aim is for the control trajectory to be as close as possible to the specified control sequence SP, the overall reproduction error D is included in the target value TV with a negative sign. If necessary, the overall performance RET and / or the overall reproduction error D can be normalized with a constant normalization factor before calculating the target value TV.

[0082] The determined target value TV is preferably assigned to the respective state signal S and / or control signal A at the beginning of the respective control trajectory. The respective target value TV thus evaluates the current capability of the control agent POL to initiate a control sequence that is both performance-optimizing and closely approximates the predefined control sequence SP, according to the weight value W, in relation to the respective state signal S.

[0083] To train the control agent POL, i.e., to optimize its processing parameters, the determined target values ​​TV are used – as in Figure 4Indicated by a dashed arrow, the data is returned to the control agent POL. The processing parameters of the control agent POL are set or configured so that the target values ​​TV are maximized, at least on average. For the actual implementation of the training, a variety of efficient standard methods, especially reinforcement learning, can be used.

[0084] After successful training of the control agent POL, it can, as in connection with Figure 1 The described method is used for the optimized control of machine M. The weighting of the optimization criteria, performance and proximity to the predefined control sequence SP, can be easily set and changed during operation. In this way, the control of machine M can be tailored to the specific operational requirements or phase, prioritizing either performance or reliability.

Claims

1. Computer-implemented method for controlling a machine (M) by means of a learning-based control agent (POL), wherein a) a performance evaluator (PEV) is provided and uses a control signal (A, A1) to determine a performance (RET) for controlling the machine (M) by means of the control signal (A, A1), wherein the performance (RET) relates to a power, an efficiency, a resource consumption, a yield, a pollutant emission, a product quality and / or other operating parameters of the machine (M), b) an action evaluator (VAE) is provided and uses the control signal (A, A1) to determine a deviation (D) from a predefined control sequence (SP), c) a multiplicity of weight values (W) for weighting the performance (RET) with respect to the deviation (D) are generated, d) a multiplicity of state signals (S, S1) and the weight values (W) are fed into the control agent (POL), wherein - a respectively resulting output signal (A, A1) from the control agent (POL) is fed as a control signal into the performance evaluator (PEV) and into the action evaluator (VAE), - a performance (RET) respectively determined by the performance evaluator (PEV) is weighted with respect to a deviation (D) respectively determined by the action evaluator (VAE) by a target function (TF) according to the respective weight value (W), and - the control agent (POL) is trained to use a state signal (S, S1) and a weight value (W) to output a control signal (A, A1) optimizing the target function (TF), and e) in order to control the machine (M) - an operating weight value (WO) and an operating state signal (SO) from the machine (M) are fed into the trained control agent (POL), and - a resulting output signal (AO) from the trained control agent (POL) is supplied to the machine (M), and wherein the weight value (W) is gradually changed when controlling the machine (M) in such a manner that the performance (RET) is increasingly given a higher weighting with respect to the deviation (D).

2. Method according to Claim 1, characterized in that a respective performance (RET) is determined by the performance evaluator (PEV) and / or a respective deviation (D) is determined by the action evaluator (VAE) on the basis of a respective state signal (S, S1).

3. Method according to one of the preceding claims, characterized in that a performance value (R) is respectively read in for a multiplicity of state signals (S) and control signals (A) and quantifies a performance resulting from application of a respective control signal (A) to a state of the machine (M) specified by a respective state signal (S), in that the performance evaluator (PEV) is trained to reproduce an associated performance value (R) on the basis of a state signal (S) and a control signal (A).

4. Method according to one of the preceding claims, characterized in that the performance evaluator (PEV) is trained to determine a performance (RET) accumulated over a future period of time by means of a Q-learning method and / or another Q-function-based reinforcement learning method.

5. Method according to one of the preceding claims, characterized in that a multiplicity of state signals (S) and control signals (A) are read in, in that the action evaluator (VAE) is trained to use a state signal (S) and a control signal (A) to reproduce the control signal (A) following information reduction, wherein a reproduction error (DR) is determined, and in that the deviation (D) is determined on the basis of the reproduction error (DR).

6. Method according to one of the preceding claims, characterized in that the deviation (D) is determined by the action evaluator (VAE) by means of a variational autoencoder, by means of an autoencoder, by means of generative adversarial networks and / or by a comparison, in particular a state-signal-dependent comparison, with predefined control signals.

7. Method according to one of the preceding claims, characterized in that the weight values (W) are generated in a randomized manner.

8. Method according to one of the preceding claims, characterized in that a gradient-based optimization method, a stochastic optimization method, particle swarm optimization and / or a genetic optimization method is / are used to train the control agent (POL), the performance evaluator (PEV) and / or the action evaluator (VAE).

9. Method according to one of the preceding claims, characterized in that the control agent (POL), the performance evaluator (PEV) and / or the action evaluator (VAE) comprise(s) an artificial neural network, a recurrent neural network, a convolutional neural network, a multilayer perceptron, a Bayesian neural network, an autoencoder, a variational autoencoder, a deep learning architecture, a support vector machine, a data-driven trainable regression model, a k-nearest neighbour classifier, a physical model and / or a decision tree.

10. Method according to one of the preceding claims, characterized in that the machine (M) is a robot, a motor, a manufacturing plant, a factory, an energy supply device, a gas turbine, a wind turbine, a steam turbine, a milling machine or another device or another installation.

11. Controller (CTL) for controlling a machine (M), configured to carry out a method according to one of the preceding claims.

12. Computer program product configured to carry out a method according to one of Claims 1 to 10.

13. Computer-readable storage medium having a computer program product according to Claim 12.