Controlling a machine using a learning-based control device

Abstract

The invention relates to a computer-implemented method for controlling a machine (M) by means of a trained learning-based control device (CTL). The control device (CTL) was trained using a target function which contains a weighting of two variables, specifically a performance of the machine (M) with respect to a deviation from a respective action selection rule determined from a plurality of action selection rules. During controlling, the previously trained control device (CTL) ascertains operating control signals (AO) based on operating state signals (SO) of the machine (M) provided to the control device, values to be used for the weighting (W), and information (c) about the action selection rule of the plurality of action selection rules to be used, and outputs said operating control signals. The control device (CTL) selects the action selection rule to be used from the plurality of action selection rules by calculating, for each of the plurality of action selection rules, a deviation variable relating to a deviation between control signals, specified by a user based on operating state signals (SO), and operating control signals (AO), ascertained by the control device (CTL) based on the same operating state signals (SO).

Description

[0001] Description

[0002] Controlling a machine with a learning-based control device

[0003] The invention relates to a computer-implemented method for controlling a machine using a learning-based control device. Operating status signals of the machine are provided to the previously trained control device, and operating control signals are then fed from the control device to the machine.

[0004] Data-driven machine learning methods, particularly reinforcement learning methods, are increasingly being used to control complex technical systems such as robots, engines, manufacturing plants, power generation facilities, gas turbines, wind turbines, steam turbines, milling machines, or other machines. Control agents, particularly artificial neural networks, are trained using a large amount of training data to generate a control signal optimized for a given state of the technical system with respect to a given target function. Control devices for controlling technical systems are therefore used in the form of trained control agents, also known as policies.

[0005] The objective function used during training can be used to evaluate, in particular, a performance of the technical system to be optimized, e.g., power, efficiency, resource consumption, yield, pollutant emissions, product quality, and / or other operating parameters of the technical system. Such an objective function is often also referred to as a reward function, cost function, or loss function. Using such a performance-optimizing control agent, the performance of a technical system controlled by it can, in many cases, be significantly increased. However, the productive use of such a control agent also entails risks because the output operating status signals are often untested and thus potentially unreliable, or their consequences for the state of the technical system are difficult to assess.This is especially true under operating conditions that are not adequately covered by the training data. Therefore, it is considered to orient the control system at least partially on predefined action selection rules. This would require rules that specify which operating control signal should be issued when a specific operating status signal is present. This could, for example, be a control sequence previously performed by a user. The object of the invention is to demonstrate a method for controlling using a learning-based control device that enables efficient and safe control of a machine.

[0006] This object is achieved by a method having the features of claim 1. Furthermore, the invention relates to a corresponding control device, a corresponding computer program, a corresponding computer-readable, preferably non-volatile, storage medium, and a corresponding transmission signal. Advantageous embodiments and further developments are the subject of subclaims.

[0007] The computer-implemented method according to the invention serves to control a machine by means of a trained, learning-based control device. The control device was previously trained using an objective function that includes a weighting of the machine's performance against a deviation from a specific action selection rule from a plurality of action selection rules. To control the machine, the previously trained control device determines operating control signals based on the machine's operating state signals provided to it, values ​​to be used for the weighting, and information on the action selection rule to be used from the plurality of action selection rules, and outputs these determined operating control signals. The action selection rule to be used is selected by the control device from the plurality of action selection rules.This is done by calculating a deviation value for each of the plurality of action selection rules relating to a deviation between control signals specified by a user based on operating state signals and operating control signals determined by the control device based on the same operating state signals.

[0008] The control unit knows the state of the machine insofar as it is described by the last received operating state signal. Based on this, as well as on the values ​​to be used for the weighting and on information about the action selection rule to be used, the control unit determines an action to be performed on or by the machine, described by the operating control signal output by the control unit. This determination of a suitable operating control signal is possible because the control unit learned during training how to determine operating control signals that optimize the objective function. Two variables are included in the objective function used during training: the machine's performance, and a deviation from a specific action selection rule. The latter is a specification of which operating control signal should be issued when a certain operating state signal is present.In this case, training is carried out with a plurality of action selection rules, so that the control device learns during the training to control based on one action selection rule from this plurality, whereby this learning takes place for each action selection rule from the plurality.

[0009] By using the two described variables in the objective function, the goals of achieving the greatest possible performance and the smallest possible deviation from a specific action selection rule are pursued. Since these two objectives are not equally important in every situation, they are weighted so that, depending on the weighting value used, one of the two objectives can be emphasized more than the other, or one of the two objectives can even play no role and only the other objective is taken into account. The control device also uses a weighting value during operation, i.e. when controlling the machine. The machine is controlled according to this value and is based on the two explained objectives. The weighting value used for this purpose can be changed over time.

[0010] During training, not only the deviation from a single action selection rule in the objective function was considered, but rather a plurality of action selection rules, or the respective deviation from them, were considered. This makes it possible to adapt the control strategy to the various action selection rules contained in the training data. For this purpose, the control unit knows which of the plurality of action selection rules should be used for control. In this case, the action selection rule to be used is not specified externally; rather, the control unit itself selects the action selection rule to be used from the plurality of action selection rules.

[0011] To select the action selection rule to be used, deviation values ​​are calculated, specifically a separate deviation value for each action selection rule of the plurality of action selection rules. The deviation value compares control signals specified by a user with operating control signals determined by the control device. These determined operating control signals are based on the same operating state signals as the user's control signals. These operating state signals of the control device are not output but are determined by the control device solely for the purpose of calculating the deviation values. This makes it possible to evaluate how similar each action selection rule is to the user's approach. In this way, an action selection rule that comes as close as possible to the user's preferences can be selected and then used for control.

[0012] When calculating the deviation value, the operating control signals can be determined by the control device based on a weighting value that weights the machine's performance to zero. This performance masking allows the user's preferences to be compared with the various action selection rules without having to accept the disruptive effects of focusing on increasing performance.

[0013] The deviation value can be calculated for a series of control signals specified by the user. To do this, the user can control the machine for a while, after which the series of user-specified control signals is available. A corresponding series of operating status signals can also be determined by the control device for each of the action selection rules. The one deviation value for each action selection rule can then be calculated from the series of user control signals and a series of operating control signals for each action selection rule.

[0014] In addition to the action selection rule to be used, the control device can also specify the weighting values ​​to be used to control the machine. To this end, after selecting the action selection rule to be used, a weighting can initially be specified that places a low weight on the machine's performance. This ensures that, at the beginning of control by the control device, control is initially carried out in a manner close to the user's preferences. This low weighting of performance can be maintained. However, preferably, the machine's performance is given a higher weighting as time increases after the selected action selection rule has been applied. This increase can occur according to a predefined temporal pattern.However, it is better if the extent or speed of the increase in the weighting of the machine's performance is calculated by the control device, in particular if it is made dependent on a comparison of the operating control signals output by the control device with the action selection rule to be used. This ensures that the free optimization of performance by the control device does not result in the action selection rule being followed too little. For training, a performance evaluator is preferably used, which determines the machine's performance based on an operating state signal and an operating control signal, as well as an action evaluator, which determines a deviation from the action selection rule based on the operating state signal and the operating control signal.

[0015] The controlled machine may be, for example, a robot, a motor, a manufacturing plant, a factory, a power plant, a gas turbine, a wind turbine, a steam turbine, a milling machine or any other device or facility.

[0016] The method according to the invention and / or one or more functions, features and / or steps of the method according to the invention and / or one of its embodiments can be carried out with computer support. It can be carried out or implemented, for example, using one or more computers, processors, application-specific integrated circuits (ASICs), digital signal processors (DSPs) and / or so-called "field programmable gate arrays" (FPGAs). It can also be carried out at least partially in a cloud and / or in an edge computing environment. One or more interacting computer programs are used for the computer-supported process. If multiple programs are used, they can be stored and executed together on one computer, or on different computers at different locations.Since this is functionally equivalent, “the computer program” and “the computer” are formulated in the singular.

[0017] The invention is explained in more detail below using an exemplary embodiment. In the following:

[0018] Figure 1 : a control device when controlling a machine,

[0019] Figure 2: a training of a control agent of the control device,

[0020] Figure 3: a flowchart.

[0021] Figure 1 illustrates a control device CTL controlling a machine M, e.g. a robot, a motor, a production plant, a factory, an energy supply facility, a gas turbine, a wind turbine, a steam turbine, a milling machine or another device or system. In particular, a component or subsystem of a machine or system can also be understood as a machine M. The machine M has sensors SK for the preferably continuous detection and / or measurement of system states or subsystem states of the machine M and, if applicable, also of its environment. The control device CTL is shown in Figure 1 external to the machine M and coupled to it. Alternatively, the control device CTL can also be fully or partially integrated into the machine M.

[0022] The control device CTL has one or more processors PROC for executing method steps of the control device CTL as well as one or more memories MEM coupled to the processor PROC for storing data, e.g. information to be processed by the control device CTL, information received by the control device CTL, information output by the control device CTL. Furthermore, the control device CTL has a learning-based control agent POL, which is trained or trainable using reinforcement learning methods. Such a control agent is often also referred to as a policy or, for short, as an agent. In the present exemplary embodiment, the control agent POL is implemented as an artificial neural network. Furthermore, the control device CTL has a learning-based transition model NN and a learning-based action evaluator VAE.The control agent POL, the transition model NN, and the action evaluator VAE, and thus the control device CTL, are trained in advance using predefined training data and thus configured for optimized control of the machine M. The transition model NN and the action evaluator VAE are required only for the purpose of training the control agent POL; the machine M is then controlled by the trained control agent POL.

[0023] The training of the control agent POL is aimed in particular at two control criteria. On the one hand, the machine M controlled by the control agent POL should achieve the highest possible performance, while on the other hand, at least under certain circumstances, the control should not deviate too much from a reference policy. The performance of the machine M can, for example, relate to power, efficiency, resource consumption, yield, pollutant emissions, product quality and / or other operating parameters of the machine M. The reference policy is a specific action selection rule, i.e. the specification of control signals that should be issued when certain state signals are present. A reference policy can originate from a past control of the machine M, a similar machine, or a simulation of the machine M while it was controlled by a reference control agent.A verified, validated, and / or rule-based control agent can preferably be used as the reference control agent. The reference policy is included in the training data, as explained in Figure 2.

[0024] For optimized control of the machine M by the trained control agent POL, operating state signals SO, i.e. state signals determined during ongoing operation of the machine M, are transmitted to the control device CTL. The operating state signals SO each specify a state, in particular an operating state of the machine M and, if applicable, its environment, and are preferably each represented by a numerical state value or vector or a time series of state values ​​or vectors. The operating state signals SO can represent measurement data, sensor data, environmental data, or other data arising during the operation of the machine M or influencing its operation. Examples include data on a temperature, a pressure, a setting, an actuator position, a valve position, a pollutant emission, a utilization, a resource consumption, and / or a performance of the machine M or its components.In a production plant, the operating status signals SO can also relate to product quality or another product property. The operating status signals SO can be measured at least partially by the sensor system SK and / or determined by simulation using a simulator of the machine M.

[0025] Furthermore, the control unit CTL uses a weight value W. This can be generated by the control unit CTL itself or entered by a user via the user interface U1. The weight value W weights two variables against each other: the performance of the machine M and a deviation of the control by the control agent POL from the reference policy.

[0026] The operating state signals SO transmitted to the control device CTL, together with the weight value W, are fed into the trained control agent POL as input signals. Based on a respective supplied operating state signal SO and weight value W, the trained control agent POL generates a respective optimized operating control signal AO. The latter specifies one or more control actions that can be performed on the machine M. The generated operating control signals AO are transmitted from the control agent POL or from the control device CTL to the machine M. The transmitted operating control signals AO control the machine M in an optimized manner by executing the control actions specified by the operating control signals AO on the machine M.Using the weight value W, the trained control agent POL can be adjusted during the ongoing operation of the machine M to weight proximity to the reference policy higher than the performance of the machine M when determining optimized operating control signals AO, and conversely to weight proximity to the reference policy lower than the performance of the machine M.

[0027] First, the control agent POL is trained using the training data TD from the database DB. This training data contains a large number of data points, where in the simplest case, each data point is a tuple consisting of a state signal, a control signal, a reward for executing the control action corresponding to the control signal based on the respective state signal—this is equal to the performance of the machine after executing the corresponding control action—and a subsequent state resulting from the application of the respective control action corresponding to the control signal. The same training data TD was previously used to train the transition model NN and the action evaluator VAE. The training data contains the reference policy, with each data point containing a state signal and an associated control signal.It is possible that the entire reference policy is more comprehensive than the part contained in the training data. This is because the training data may not cover the entire state / action space, so that there are pairs of state signals and control signals produced by the generator of the reference policy that are not part of the training data.

[0028] The described configuration of the training data as a four-tuple, where the two-tuple of state signal and corresponding control signal represents the reference policy, is not sufficient for many applications. This is because the reference policy corresponds to the control procedure of a specific user or a specific entity that has carried out the control corresponding to the training data, hereinafter referred to collectively as the controller. In reality, there will be a plurality of controllers that have controlled the machine in the past and thereby generated a data set of training data. These different controllers may have pursued different control strategies. Typically, a control behavior in the form a = b(s) is learned during training, where s is the state signal, b is the model to be learned, and a is the control signal.If different controllers generated the training data, different control signals a1 and a2 exist for the same state signal s. Therefore, it makes sense to consider these different data sets separately during control by the control agent POL. The usual training approach is thus extended to a = b(s, c), where c is a controller specification.

[0029] It is therefore assumed that there is not just one reference policy against which a deviation is evaluated, but several. This is achieved by the data points of the training data containing, in addition to the aforementioned four-tuple consisting of a state signal, a control signal, a reward, and a subsequent state, another variable: the controller information. This controller information indicates which controller generated the respective data point. The controller producing the training data could, for example, be a human user who manually entered control commands, a rule-based programmed control system, or even an agent trained using reinforcement learning.

[0030] A training process for the control agent POL is explained in more detail with reference to Figure 2. The training of the control agent POL is carried out in such a way that the relative weighting of the two optimization criteria—performance of the machine M and proximity to a reference policy—can be changed by the weight value W during the ongoing operation of the trained control agent POL. Furthermore, several reference policies are available. The behavior to be learned can therefore be described as a = b(s, W, c).

[0031] Figure 2 illustrates the training of the control agent POL using the trained transition model NN and the trained action evaluator VAE. To illustrate successive work steps, Figure 2 shows several instances of the control agent POL, the trained transition model NN, and the trained action evaluator VAE. The different instances can, in particular, correspond to different calls or evaluations of routines by means of which the control agent POL, the trained transition model NN, or the trained action evaluator VAE are implemented.

[0032] The trained transition model NN, which can be implemented in particular as an artificial neural network, was trained to predict, based on a respective state signal and a respective control signal, a subsequent state of the machine M resulting from the application of the corresponding control action, as well as a resulting performance value of the machine M, the reward for the application of the corresponding control action, as accurately as possible. The training of such a transition model NN is described, for example, in publications EP 3940596 A1 and EP 4235317 A1. Instead of a transition model NN, other models can also be used to evaluate the reward, e.g., a value function.

[0033] The trained action evaluator VAE, which can in particular be a variational autoencoder implemented as a feed-forward neural network, evaluates arbitrary pairs of a state signal and a control signal by determining a reproduction error D for the pair. The reproduction error D is a measure of how well a state signal-control signal pair is covered by the training data TD, or how frequently or how likely it occurs there. Since the training data TD contains the reference policy, as explained above, the reproduction error indicates how strongly the state-control signal pair deviates from the reference policy. The training of such an action evaluator VAE is described, for example, in the publications EP 3940596 A1 and EP 4235317 A1, where the reference policy is referred to as the predetermined control sequence.In addition to the approach described in the above-mentioned publication, the VAE action evaluator uses the controller specification during training. In operational use, the VAE action evaluator should be able to predict the deviation from each of the multiple reference policies.

[0034] As already mentioned above, the control agent POL should be trained to output a control signal A for a respective state signal S of the machine M, which is optimized, on the one hand, with regard to the resulting performance of the machine M and, on the other hand, with regard to a proximity to or deviation from a reference policy. Obviously, proximity to a reference policy can also be represented by a negatively weighted deviation from the respective reference policy, and vice versa. The optimization aims at higher performance and greater proximity to or smaller deviation from the reference policy. The weighting of the two optimization criteria, performance and proximity or deviation, can be adjusted using the weight value W.

[0035] The weight value W can be set between 0 and 1. With a weight value of W=1, the trained control agent POL should output a control signal A that exclusively optimizes performance, whereas with a weight value of W=0, a control signal A that exclusively minimizes deviations from a reference policy should be output. With weight values ​​between 0 and 1, the two optimization criteria should be weighted proportionally accordingly. To train the control agent POL for different weight values, a generator GEN of the control device CTL generates a plurality of weight values ​​W, preferably randomly, which lie in the interval from 0 to 1. The weight values ​​W generated by the generator GEN are fed, as can be seen in Figure 2, both into the control agent POL to be trained and into an objective function TF to be optimized through training.

[0036] To train the control agent POL, a large number of state signals S from the training data TD are fed into the control agent POL as input signals. At the same time, these state signals S are also fed into the trained transition model NN and the trained action evaluator VAE as input signals. Based on the respective state signal S, the trained transition model NN predicts the subsequent states of the machine M resulting from the application of a control signal A. Furthermore, the pair formed by the respective state signal S and the associated control signal A is evaluated by the trained action evaluator VAE. Furthermore, the control agent POL receives the controller information c corresponding to the state signal S from the training data TD.This is also made available to the action evaluator VAE so that it can predict how the deviation of the respective pair (S, A) from the training data of the respective controller is.

[0037] The control agent POL derives an output signal from the respective state signal S and outputs it as control signal A. The control signal A is then fed together with the respective state signal S into the trained transition model NN, which predicts a subsequent state and outputs a subsequent state signal S1 specifying this as well as an associated performance value R1. In addition, the control signal A is fed alongside the respective state signal S into the trained action evaluator VAE, which determines and outputs a reproduction error DO for the pair of state signal S and control signal A - related to the controller specification c. As explained above, the reproduction error can be viewed as a measure of a deviation of control signals from the control signals specified by the respective reference policy.

[0038] The subsequent state signal S1 is in turn fed to the control agent POL, which derives a next control signal A1 for this subsequent state. The control signal A1, together with the subsequent state signal S1, is fed into the trained transition model NN, which predicts a further subsequent state and outputs a subsequent state signal S2 specifying this, as well as a corresponding performance value R2. Furthermore, the control signal A1, together with the subsequent state signal S1, is fed into the trained action evaluator VAE, which determines and outputs a reproduction error D1 for the pair of state signal S1 and control signal A1, again based on the controller specification c.

[0039] The above steps can be repeated iteratively, determining performance values ​​and reproduction errors for subsequent states. The iteration can be terminated if a termination condition is met, e.g., if a predefined number of iterations is exceeded. In this way, a control trajectory spanning several time steps, progressing from subsequent state to subsequent state, and extrapolated into the future with associated performance values ​​R1, R2, ... and reproduction errors DO, D1, ... can be determined. Such an extrapolation is often referred to as a rollout or virtual rollout.

[0040] From the performance values ​​R1, R2, ... of a respective control trajectory, a cumulative total performance RET of this control trajectory over several time steps is determined. Such cumulative total performance is referred to as return in reinforcement learning. The total performance RET is preferably assigned to the respective state signal S with the controller specification c at the beginning of the respective control trajectory and thus evaluates the ability of the control agent POL to determine a control signal A for the respective state signal S with the controller specification c, which initiates a control sequence with high performance over several time steps. To determine the total performance RET, the performance values ​​R1, R2, ... determined for future time steps are discounted, i.e., assigned weights that decrease for each time step. As a concrete example, the total performance RET can be expressed as the weighted sum of the performance values ​​R1, R2, ...whose weights correspond to a discount factor of 1 that decreases with each journal into the future. In this way, the overall performance RET can be determined, for example, according to RET = R1 + R2*G + R3*G. 2 + R4*G 3 + .... For example, a value of 0.99, 0.9, 0.8, or 0.5 can be used for G. Alternatively, the overall performance can be calculated without discounting or using a discount factor of 1.

[0041] The transition model NN and the above discounting method together form the performance evaluator PEV, which, based on control signals A, A1, ... and state signals S, S1, ..., determines an overall performance RET of the machine M resulting from the application of the control signals. Alternatively or additionally, the performance evaluator PEV can also be implemented using a Q-learning method and trained to determine the overall performance RET accumulated over a future period. Furthermore, a total reproduction error D accumulated over several time steps is determined from the reproduction errors DO, D1, ... of the respective control trajectory. The latter serves in the further method as a measure of the deviation of this control trajectory from the respective reference policy. In the present embodiment, the total reproduction error D is determined as the sum of the individual reproduction errors DO, D1, ... according to D = D0 + D1 + ....

[0042] The respectively determined total performance RET and the respectively determined total reproduction error D are both fed into the objective function TF to be optimized. The objective function TF weights the total performance RET and the total reproduction error D with the respective weight value W, which was also fed into the control agent POL together with the respective state signal S. Thus, through training that optimizes the objective function TF, the control agent POL can be trained to output a control signal A, upon input of a state signal S and a weight value W, which optimizes the objective function TF according to the input weight value W, at least on average.

[0043] In the present exemplary embodiment, the objective function TF determines a target value TV of the objective function TF from the overall performance RET, the overall reproduction error D, and the weight value W according to TV = W*RET - (1-W)*D. Since the goal is to ensure that the control trajectory is as close as possible to the respective reference policy, the overall reproduction error D is included in the target value TV with a negative sign. If necessary, the overall performance RET and / or the overall reproduction error D can be provided with a constant normalization factor before calculating the target value TV.

[0044] The determined target value TV is assigned to the respective state signal S and the associated controller specification c at the beginning of the respective control trajectory. The respective target value TV thus evaluates the current ability of the control agent POL to initiate a control sequence for the respective state signal S with the controller specification c, according to the weight value W, that both optimizes performance and is close to this reference policy. To train the control agent POL, i.e., to optimize its processing parameters, the determined target values ​​TV are fed back to the control agent POL—as indicated by a dashed arrow in Figure 2. The processing parameters of the control agent POL are set or configured such that the target values ​​TV are maximized, at least on average.To carry out the training, a variety of efficient standard methods can be used, in particular stochastic gradient descent, population-based optimization methods, or other methods of supervised learning.

[0045] After successful training of the control agent POL, it can be used for optimized control of the machine M, as described in Figure 1. For this purpose, the trained control agent POL requires an operating status signal, a controller specification c, and a weight value W, and outputs an operating control signal based on these. The weighting of the optimization criteria performance and proximity to the reference policy can be adjusted and changed during operation by specifying a value for W. In this way, the control of the machine M can be adjusted to whether performance or reliability should be given greater weight.

[0046] The applicable reference policy can be explicitly specified by a user via the user interface U1 by specifying a controller c. Thus, if the user knows that they want to have similar control to controller number 3, they can specify c=3 and, for example, W=0.1. This means a strong focus on controller number 3, since a small value for W is set, with some leeway for the control unit CTL to optimize performance. The user's decision for a controller contained in the training data is even more important if the user is identical to one of these controllers.

[0047] However, if the user does not know which controller to use, the CALC c component of the control unit CTL can be used to determine a suitable value for c. This is done by observing the user's control decisions to determine his preferences and then automatically adjusting c accordingly. To do this, the user controls the machine M for a sufficiently long time. This results in a control sequence a , a2, ... , a H with H control steps, where the index indicates the time or the control steps.

[0048] For each control step, the distance between the respective control signal a i twhich was specified by the user, and a control signal AO, which the control device CTL would have output based on the respective status signal SO. To determine this fictitious control signal AO, the control agent POL is given the value W=0, i.e., the control should be based exclusively on the reference policy. Furthermore, such a fictitious control signal is calculated for each value of the controller specification c. It is thus located in addition to the control sequence a , a2, ... , a H of the user for each controller specification c a fictitious control sequence AO c l ,AO c 2 , ... ,AO c H Thus, for each control step, the distances d; - AO C, are calculated. For each value of the controller specification c, these distances from the H points in time are summarized to form a total distance, e.g. as the mean square deviation. The controller specification c for which the smallest total distance was calculated is then specified by the CALC c component as the value for c for the subsequent control of the machine M by the control device CTL and is used by the control agent POL. After completing the H control steps, the user does not need to carry out any further control; this is done autonomously by the control device CTL. However, it is possible for the user to intervene and carry out the control themselves if they see the need. Furthermore, it is possible for the control device CTL to ask the user, after determining the most suitable value for c, whether this value should be used.

[0049] A suitable setting of the weighting W during runtime is described below. For this purpose, the CALC W component of the control device CTL is used, which calculates the weight value W and makes it available to the control agent POL for controlling the machine M. If the control device CTL determines the value for c by observing the user, as described, and then uses it to autonomously control the machine M, it is sensible for the control device CTL to also specify a value for W. For this purpose, a small value for W can be used initially after c has been determined. This means that control is carried out that is strongly oriented towards the reference policy corresponding to the value of c. Over time, the value for W can be increased in order to give the control device CTL the opportunity to optimize the performance of the machine M.

[0050] For the determination of W, it is also advantageous to compare the distance of the control by the user to the controller corresponding to the selected c with the distance of the actual control by the control device CTL to the controller corresponding to the selected c. The first of the two quantities is the total distance described above. The second quantity can be calculated by comparing the control signals from the training data of the controller corresponding to the selected c with the control signals AO of the control device CTL in the same way as explained for the total distance. If both quantities are approximately the same, a large value for W can be used, since in this case the deviation from the reference policy corresponding to the selected c is acceptable for the user. With this acceptable deviation from the reference policy, the control device CTL can then optimize the performance of the machine.The second variable, i.e. the deviation of the actual control from the reference policy, can be determined continuously or repeatedly at certain time intervals in order to adjust the value for W if necessary.

[0051] In summary, Figure 3 shows a flowchart of the described procedure. In a training step TRAIN 1, which precedes the use of the control device CTL to control the machine, the transition model NN and the action evaluator VAE are first trained using training data. In a further upstream training step TRAIN 2, the control agent POL is then trained using the same training data and with the help of the trained transition model NN and the trained action evaluator VAE. After deployment, the user begins to control the machine in the CONTROL step, resulting in a series of user-generated control signals a. Based on this, the control device CTL determines the controller specification c in the CALCULATE c step. Furthermore, the weight value W is also calculated in the CALCULATE W step; alternatively, this step can be omitted and the value for W can be specified by the user.The two values ​​for c and W are then used in the regular control mode OPERATE, in which the control device with the three trained units—the transition model NN, the action evaluator VAE, and the control agent POL—takes over control of the machine. The control agent POL outputs control signals AO to the machine. It is advantageous to adjust the value for W continuously or occasionally, as symbolized by the arrow returning from OPERATE to CALCUATE W.

[0052] The invention has been described above using an exemplary embodiment. It is understood that numerous changes and modifications are possible without departing from the scope of the invention.

Claims

Patent claims 1. A computer-implemented method for controlling a machine (M) by means of a trained learning-based control device (CTL), wherein the training of the control device (CTL) is carried out using an objective function which includes a weighting (W) of a performance of the machine (M) against a deviation from a specific action selection rule from a plurality of action selection rules, in which the previously trained control device (CTL) for controlling the machine (M) determines and outputs operating control signals (AO) based on the operating state signals (SO) of the machine (M) provided to it, values to be used for the weighting (W), and information (c) on the action selection rule to be used from the plurality of action selection rules, the action selection rule to be used is selected by the control device (CTL) from the plurality of action selection rules,by calculating, for each of the plurality of action selection rules, a deviation value relating to a deviation between control signals specified by a user based on operating state signals (SO) and operating control signals (AO) determined by the control device (CTL) based on the same operating state signals (SO).

2. Method according to claim 1, wherein, in calculating the deviation value, the operating control signals (AO) are determined by the control device (CTL) based on a value for the weighting (W) which weights the performance of the machine (M) with zero.

3. A method according to claim 1 or 2, wherein the deviation quantity is calculated for a series of control signals specified by the user.

4. Method according to one of claims 1 to 3, in which the values for the weighting (W) to be used to control the machine (M) are specified by the control device (CTL) in that, after selection and use of the action selection rule to be used, a weighting (W) is first specified which gives the performance of the machine (M) a low weighting.

5. Method according to claim 4, wherein the values for the weighting (W) to be used to control the machine (M) are predetermined by the control device (CTL) in that the performance of the machine (M) is weighted more as time increases.

6. Method according to claim 5, in which the values for the weighting (W) to be used to control the machine (M) are predetermined by the control device (CTL), in that the increase in the weighting of the performance of the machine (M) depends on a comparison of the operating control signals (AO) output by the control device (CTL) with the action selection rule to be used.

7. Method according to one of claims 1 to 6, in which a performance evaluator (PEV) is used for training, which determines a performance of the machine (M) on the basis of an operating state signal (SO) and an operating control signal (AO), and an action evaluator (VAE) is used, which determines a deviation from the action selection rule on the basis of the operating state signal (SO) and the operating control signal (AO).

8. Method according to one of the preceding claims, characterized in that the machine (M) is a robot, a motor, a manufacturing plant, a factory, a power plant, a gas turbine, a wind turbine, a steam turbine, a milling machine or another device or installation.

9. Control device (CTL) for controlling a machine (M), arranged to carry out a method according to one of claims 1 to 8.

10. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium (MEM) comprising a computer program according to claim 10.

12. A transmission signal transmitting the computer program according to claim 10.

Images