Control device for controlling a technical system and method for configuring a control device

Abstract

According to the present invention, the operating signal (BS) of the technical system (TS) is fed into a first machine learning module (NN1), which is trained to reproduce the behavior signal of the technical system that occurs specifically when no control action is currently applied, and outputs the reproduced behavior signal (VSR1) as a first output signal. The first output signal (VSR1) is fed into a second machine learning module (NN2), which is trained to reproduce the behavior signal of the technical system obtained from the control action signal (AS) and outputs the reproduced behavior signal (VSR2) as a second output signal. Furthermore, the operating signal (BS) of the technical system is fed into a third machine learning module (NN3), and the third output signal (AS) of the third machine learning module (NN3) is fed into the trained second machine learning module (NN2). The control action performance (Q) is determined based on the second output signal (VSR2). Thus, the third machine learning module (NN3) is trained to optimize the control action performance (Q). The control device (CTL) is configured to control the technical system by training the third machine learning module (NN3).

Description

[0001] Machine learning methods are increasingly being used to control complex technological systems, such as gas turbines, wind turbines, internal combustion engines, robots, processing facilities, or power grids. This learning approach allows for the training of machine learning models of control devices based on training data. These models determine specific control actions that induce desired or optimized behavior in the technological system, thereby optimizing its performance, starting from the system's current operating signals. Such machine learning models used to control technological systems are often referred to as policies or control models. Numerous known training methods, such as reinforcement learning, can be used to train such policies.

[0002] However, in control optimization within industrial environments, convergence problems and / or issues regarding the reproducibility of the learning process arise with many known training methods. This can be attributed, for example, to: only a small portion of the state space of the technical system being considered; the sensor data of the technical system being noisy; and / or control actions typically taking effect with time delays, where different control actions often result in different time delays. These symptoms frequently occur in complex real-world systems and can significantly impact learning outcomes.

[0003] The objective of this invention is to describe a control device for a control technology system that allows for more efficient training, and a method for configuring the control device.

[0004] This task is accomplished by a method implemented by a computer for configuring a control device for a technical system, by a control device for controlling a technical system, by a computer program product, and by a computer-readable storage medium.

[0005] To configure a control device for the technical system, the operating signals of the technical system are fed into a first machine learning module. This first machine learning module is trained to reproduce, based on the operating signals, the behavioral signals of the technical system that occur specifically when no control action is currently applied, and outputs the reproduced behavioral signals as a first output signal. The first output signal is fed into a second machine learning module, which is trained to reproduce, based on control action signals, the behavioral signals derived therefrom, and outputs the reproduced behavioral signals as a second output signal. Furthermore, the operating signals of the technical system are fed into a third machine learning module, and the third output signal of the third machine learning module is fed into the trained second machine learning module. Control action performance is determined based on the second output signal. Thus, the third machine learning module is trained to optimize control action performance based on the operating signals of the technical system. Finally, the control device is configured to control the technical system using the third output signal of the third machine learning module.

[0006] To perform the method according to the invention, a control device, a computer program product, and preferably a non-volatile computer-readable storage medium are provided.

[0007] The method according to the invention and the control device according to the invention can be executed or implemented, for example, by means of one or more computers, processors, application-specific integrated circuits (ASICs), digital signal processors (DSPs), and / or so-called "field-programmable gate arrays" (FPGAs).

[0008] This invention enables significantly more efficient configuration or training of control devices. Since the trained second machine learning module is used when training the third machine learning module, it is generally no longer necessary to implicitly learn or represent important components of the system's behavior when training the third machine learning module. This leads to significantly improved convergence behavior and / or better reproducibility of training results in many cases. Furthermore, the training often becomes more stable and / or robust relative to changes in the training data. Additionally, less training data, computation time, and / or computational resources are required in many cases.

[0009] Advantageous embodiments and improvements of the present invention are described in the dependent claims.

[0010] According to an advantageous embodiment of the invention, a third machine learning module can be trained based on a first output signal. Thus, the third machine learning module can often be trained particularly effectively because specific information about the system behavior in the absence of currently applied control actions is available to the third machine learning module.

[0011] According to a particularly advantageous embodiment of the invention, the performance of the control action can be determined for each corresponding time point based on a single time step of the behavioral signal. Often, it is unnecessary to expend effort to determine or estimate the future effects on performance. Thus, dynamic effects operating on different time scales can also be considered efficiently. Furthermore, the time steps can be of varying lengths depending on the control action and / or behavioral signal, and the future effects of the control action can also be mapped.

[0012] The first and / or second portions of the operating signal of the technical system can advantageously be selected specifically based on whether or not the first and / or second portions include control actions. Thus, the first portion of the operating signal that does not include control actions can be specifically used to train a first machine learning module, and / or the second portion of the operating signal that includes control actions can be specifically used to train a second machine learning module. The first and / or second machine learning modules can be trained particularly effectively through specific selections of training data oriented towards the corresponding training objectives.

[0013] According to another advantageous embodiment of the invention, a behavior signal nominal value can be read in and a second output signal can be compared with the behavior signal nominal value. The control action performance can then be determined based on the comparison result. In particular, the deviation between the second output signal and the behavior signal nominal value can be determined, for example, in the form of the absolute value or squared difference. The control action performance can then be determined based on the deviation, where a larger deviation generally results in lower control action performance.

[0014] The behavioral signal ratings can also be fed into a third machine learning module. This third machine learning module can then be trained to optimize control performance based on the behavioral signal ratings.

[0015] According to another advantageous embodiment of the invention, the performance of the control action can be determined based on the first output signal. Here, in particular, the deviation between the first and second output signals, for example in the form of the absolute value or square of the difference, can be determined. Alternatively or additionally, the deviation of the sum of the first and second output signals from the nominal value of the behavior signal can be determined. The performance of the control action can then be determined based on the deviation thus determined. In this case, the system behavior when the control action is applied can be evaluated, in particular, based on the deviation: how well the system behavior is distinguished from that when the control action is not applied. It has been shown that this distinction can significantly improve the determination of the performance of the control action in many cases.

[0016] According to an advantageous improvement of the invention, a first machine learning module and / or a second machine learning module can be trained to individually reproduce multiple behavioral signals of different processes operating in a technical system. Thus, the performance of control actions can be determined based on the reproduced behavioral signals. In this regard, the first and / or second machine learning modules may in particular include a set of machine learning models or sub-models that model specific processes operating in the technical system in a process-specific manner. In many cases, such separate training proves more efficient than combined training because the individual dynamics based on separate models typically have simpler response behavior than combined system dynamics.

[0017] As long as the present invention allows the determination of control action performance at corresponding points in time based on individual, adaptable time steps of behavioral signals, this is particularly relevant when training a third machine learning module, especially addressing the less frequent synchronization problems that typically arise between processes with different operating speeds. In many cases, it has been shown that for different process-specific machine learning models, the performance of control actions can be evaluated accurately and robustly in a single step.

[0018] Furthermore, specific behavioral signal ratings can be read into the corresponding behavioral signals. Then, the control performance can be determined by comparing the reproduced behavioral signal with the specific behavioral signal rating.

[0019] In particular, the third machine learning module can be trained to optimize control action performance based on specific behavioral signal ratings.

[0020] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. Here, schematic diagrams are provided respectively: Figure 1 This illustrates a gas turbine along with a control device according to the invention. Figure 2 The control device according to the invention is shown in the first training phase. Figure 3 The control device is shown in the second training phase, and Figure 4 The control device is shown in the third training phase.

[0021] Figure 1 An example is illustrated of a gas turbine along with a control unit CTL as a technical system TS. Alternatively or additionally, the technical system TS may also include a wind turbine, an internal combustion engine, a processing facility, a chemical, metallurgical or pharmaceutical manufacturing process, a robot, a motor vehicle, an energy delivery network, a 3D printer or other machine, other equipment or other facilities.

[0022] The gas turbine (TS) is coupled to a control unit (CTL), which can be implemented as part of the gas turbine (TS) or wholly or partially external to the gas turbine (TS). For clarity, in Figure 1 In the diagram, the control unit CTL is shown outside the technical system TS.

[0023] The control device CTL is used to control the technical system TS and is trained for this purpose using machine learning methods. In this context, the control technical system TS should also be understood as regulating the technical system TS and outputting and using control-related data or signals that contribute to the control of the technical system TS.

[0024] Such control-related data or signals may include, in particular, control action signals, predictive data, monitoring signals, status data and / or classification data, which may be used to optimize the operation of the technical system TS, monitor or maintain it and / or to identify wear or damage.

[0025] The gas turbine TS has a sensor S that continuously measures one or more operating parameters of the technical system TS and outputs the operating parameters as measured values. The measured values ​​of the sensor S, as well as the operating parameters detected by the technical system TS in other ways when necessary, are transmitted from the technical system TS to the control unit CTL as operating signals BS.

[0026] Operating signals (BS) can include, in particular, physical, chemical, regulatory, effect-related, and / or structurally determined operating parameters, characteristic data, performance data, effect data, status signals, behavioral signals, system data, preset values, control data, control action signals, sensor data, measured values, environmental data, monitoring data, predictive data, analytical data, and / or other data that occurs during the operation of the technical system (TS) and / or describes the operating status or control actions of the technical system (TS). This could be, for example, data regarding the temperature, pressure, emissions, vibration, oscillation status, or resource consumption of the technical system (TS). Especially in the case of gas turbines, operating signals (BS) can relate to turbine power, rotational speed, vibration frequency, vibration amplitude, combustion dynamics, combustion alternating pressure amplitude, or nitrogen oxide concentration.

[0027] The trained control unit (CTL) determines control actions based on the operating signal (BS), which optimize the performance of the technical system (TS). The performance to be optimized may specifically involve power, revenue, speed, operating time, precision, error rate, resource requirements, efficiency, hazardous substance production, stability, wear, service life, and / or other target parameters of the technical system (TS).

[0028] The control unit CTL transmits the corresponding control action signal AS to the technical system TS to facilitate the control actions that determine the optimal performance. Through these control actions, for example in the case of a gas turbine, gas supply, gas distribution, or air supply can be set.

[0029] Figure 2 The diagram illustrates a learning-based control device CTL according to the invention during the first training phase. The control device CTL should be configured to control the technology system TS. Where the same or corresponding reference numerals are used in the figures, these reference numerals denote the same or corresponding entities.

[0030] In this embodiment, the control device CTL is coupled to the technology system TS. The control device CTL includes one or more processors PROC for executing the method according to the invention and one or more memories MEM for storing method data.

[0031] The control unit CTL receives operating signals BS from the technical system TS as training data. The operating signals specifically include timing information, i.e., a time series of values ​​for the operating parameters of the technical system TS. In this embodiment, the operating signals BS include a status signal SS specifying the state of the technical system TS over time, a control action signal AS specifying or causing control actions of the technical system TS, and a behavior signal VS specifying the system behavior of the technical system TS. The latter may, for example, specify changes in combustion alternation pressure amplitude, emissions, speed, or temperature of the gas turbine. The behavior signal VS, in particular, can detect the status signals of the technical system TS related to its performance.

[0032] The operating signal BS may also be received, at least in part, from a technical system similar to the technical system TS, from a database of operating signals stored in a technical system TS or similar to it, and / or from an analog of the technical system TS or similar to it.

[0033] The control device CTL further includes a first machine learning module NN1, a second machine learning module NN2, and a third machine learning module NN3. The corresponding machine learning modules NN1, NN2, or NN3 can in this case be designed as artificial neural networks or as a collection of neural subnetworks. The first machine learning module NN1 can in particular be designed as a submodule of the third machine learning module NN3.

[0034] Preferably, the machine learning modules NN1, NN2, and / or NN3 can use or implement supervised learning methods, reinforcement learning methods, recurrent neural networks, convolutional neural networks, Bayesian neural networks, autoencoders, deep learning architectures, support vector machines, data-driven trainable regression models, k-nearest neighbor classifiers, physical models, decision trees, and / or random forests. Numerous efficient implementations are available for the illustrated variations and their training.

[0035] In this context, training should generally be understood as optimizing the mapping from the input signal to the output signal. This mapping has been optimized so far during the training phase according to pre-given, learned, and / or to-be-learned criteria. In this case, the criteria could be, for example, the prediction error in the case of a prediction model, the classification error in the case of a classification model, or the outcome of the control action (Erfolg) in the case of a control model. Through training, for example, the network structure of the neurons in the neural network and / or the weights of the connections between neurons can be set or optimized to satisfy the pre-given criteria as well as possible. Therefore, training can be interpreted as an optimization problem. For such optimization problems in the field of machine learning, a large number of efficient optimization methods are available. In particular, gradient descent, particle swarm optimization, and / or genetic optimization methods can be used.

[0036] In passing Figure 2 In the first training phase, a first machine learning module NN1 is trained. This first machine learning module should be trained to predict or reproduce the behavior of the technical system TS in the absence of current control actions, based on the operating signals of the technical system TS.

[0037] To improve training results, the training data BS is filtered through a filter F1 coupled to the first machine learning module NN1, so as to obtain training data preferably in the absence of control action or without the action of control. For this purpose, the running signal BS is fed into the filter F1. The filter F1 includes a control action detector ASD for detecting control actions in the running signal BS based on the control action signal AS contained therein.

[0038] Based on the detection of control actions by the control action detector ASD, a first portion of the operating signal BS is selected by filter F1 and extracted from the operating signal BS. Here, it is preferable to select the first portion of the operating signal BS that does not contain a control action and / or does not have the effect of a control action. Thus, the first portion of the operating signal BS can be extracted, for example, from a time window following the detected current control action, wherein the time window is selected such that the control action cannot yet affect the system behavior.

[0039] The first filtered portion of the running signal BS includes the first portion SS1 of the state signal SS and the first portion VS1 of the behavior signal VS. The first portions SS1 and VS1 are output through filter F1 and used to train the first machine learning module NN1.

[0040] The first part SS1 of the state signal SS is fed as an input signal into the first machine learning module NN1 for training. Through training, the first machine learning module aims to reproduce, as accurately as possible, the behavioral signals of the technical system TS when no control action is applied, based on the operating signals of the technical system TS. That is, the output signal VSR1 of the first machine learning module NN1 (hereinafter referred to as the first output signal) should correspond as closely as possible to the actual behavioral signals of the technical system TS. To this end, the deviation D1 between the first output signal VSR1 and the corresponding first part VS1 of the behavioral signal VS is determined. In this case, the deviation D1 represents the reproduction or prediction error of the first machine learning module NN1. The deviation D1 can be specifically expressed as D1 = (VS1 - VSR1). 2 Alternatively, D1 = │VS1-VSR1│ is calculated as the difference, especially the square or absolute value of the vector difference.

[0041] like Figure 2 As indicated by the dashed arrow, the bias D1 is fed back to the first machine learning module NN1. Based on the fed-back bias D1, the first machine learning module NN1 is trained to minimize the bias D1 and thus minimize the reproduction error. As shown above, a variety of optimization methods, such as gradient descent, particle swarm optimization, or genetic optimization, are available to minimize the bias D1. In this way, the first machine learning module NN1 is trained using a supervised learning method. The trained first machine learning module NN1 uses the first output signal VSR1 to reproduce the behavioral signal of the technical system TS, which would occur if no control action were currently applied.

[0042] The first machine learning module NN1 is trained particularly effectively with respect to the training objective by using filtered running signals SS1 and VS1. It should also be noted that the first machine learning module NN1 can also be trained outside of the control unit CTL.

[0043] The training method described above can be used in a particularly advantageous manner to individually reproduce multiple behavioral signals of different processes operating in the technical system TS. For this purpose, the first machine learning module NN1 may include multiple process-specific neural subnetworks, which are trained individually or uniquely using process-specific behavioral signals, as described above. In many cases, such separate training proves more efficient than combined training because the individual dynamics on their own often exhibit simpler and / or more uniform response behavior.

[0044] Figure 3 The control device CTL in the second training phase is explained. In the second training phase, the second machine learning module NN2 should be trained to predict or reproduce the behavior of the technical system TS induced by corresponding control actions based on the operating signals BS of the technical system TS and, in particular, based on the control action signals AS contained therein.

[0045] To train the second machine learning module NN2, the control device CTL receives the operating signal BS of the technology system TS as training data. As mentioned above, the operating signal BS specifically includes the timing of the state signal SS, the control action signal AS, and the behavior signal VS. The trained first machine learning module NN1 is further used to train the second machine learning module NN2. In this embodiment, the training of the first machine learning module NN1 has already been completed when training the second machine learning module NN2.

[0046] To improve training results, the training data BS is filtered through a filter F2 coupled to the second machine learning module NN2 in order to preferably obtain training data that contains control actions or the effects of control actions.

[0047] For this purpose, the running data BS is fed into filter F2. Filter F2 includes a control action detector ASD for specifically detecting control actions in the running signal BS based on the control action signal AS contained therein. Based on the detection of control actions by the control action detector ASD, a second portion of the running signal BS is selected by filter F2 and extracted from the running signal BS. Here, it is preferable to select the second portion of the running signal BS that includes control actions and / or the effects of control actions. For example, the second portion of the running signal BS can be extracted from a time window surrounding the separately detected control actions and / or from a time window in which the effects of the corresponding control actions can be expected. The filtered second portion of the running signal BS specifically includes a second portion VS2 of the behavior signal VS and a second portion AS2 enriched with control action signals. The second portions AS2 and VS2 of the running signal BS are output by filter F2 and used to train a second machine learning module NN2.

[0048] For training, the second part AS2 of the running signal BS is fed as an input signal into the second machine learning module NN2. Furthermore, the running signal BS is fed into a first machine learning module NN1, which has already been trained, from which the behavior signal VSR1 is derived and output as the first output signal. As described above, the behavior signal VSR1 reproduces the behavior signal that the technology system TS would exhibit if no control action were currently applied. The behavior signal VSR1 is also fed into the second machine learning module NN2 as another input signal.

[0049] Through training, the second machine learning module NN2 strives to reproduce, as accurately as possible, the behavioral signals induced by control actions in the technical system TS, based on the operational signals containing control actions (AS2 in this case) and the behavioral signals that occur when no control actions are currently applied (VSR1 in this case). In other words, the output signal VSR2 of the second machine learning module NN2 (hereinafter referred to as the second output signal) should be as consistent as possible with the actual behavioral signals of the technical system TS under the influence of control actions.

[0050] During training, the deviation D2 between the second output signal VSR2 and the corresponding second part VS2 of the behavior signal VS is determined. In this case, the deviation D2 represents the reproduction or prediction error of the second machine learning module NN2. The deviation D2 can be expressed, for example, as D2 = (VS2 - VSR2). 2 Alternatively, D2 = │VS2-VSR2│ is calculated as the difference, especially the square or absolute value of the vector difference.

[0051] like Figure 3 As indicated by the dashed arrow, the bias D2 is fed back to the second machine learning module NN2. Based on the fed-back bias D2, the second machine learning module NN2 is trained to minimize the bias D2 and thus minimize the reproduction error. As mentioned above, a large number of known optimization methods, especially supervised learning methods, can be used to minimize the bias D2.

[0052] The trained second machine learning module NN2 uses the second output signal VSR2 to reproduce the behavioral signals of the technology system TS generated by the current application control action.

[0053] The second machine learning module NN2 is trained particularly effectively with respect to the training objective by using filtered running signals AS2 and VS2. Furthermore, in many cases, training results can be significantly improved by feeding the behavior signal VSR1 into the second machine learning module NN2, as specific information about the difference between the system behavior induced by controlled actions and the system behavior without controlled actions becomes available to the second machine learning module NN2. It should also be noted that the second machine learning module NN2 can also be trained outside of the control device CTL.

[0054] The training method described above can be used in a particularly advantageous manner to individually reproduce multiple behavioral signals of different processes operating in the technical system TS. For this purpose, the second machine learning module NN2 can include multiple process-specific neural subnetworks, like the first machine learning module NN1, which are trained individually or uniquely using process-specific behavioral signals, as described above.

[0055] Figure 4 The control device CTL in the third training phase is explained. In this phase, the third machine learning module NN3 is trained to generate control action signals for optimizing the performance of the technical system TS based on its operating signals. Here, optimization should be understood as an approximation of the optimal value. By training the third machine learning module NN3, the control device CTL is configured to control the technical system TS.

[0056] To train the third machine learning module NN3, the control device CTL receives the operating signal BS of the technology system TS as training data. For this training, specifically using the first machine learning module NN1 and the second machine learning module NN2, which have been trained as described above. In this embodiment, the training of machine learning modules NN1 and NN2 has already been completed during the training of the third machine learning module NN3.

[0057] In addition to the components described above, the control device CTL includes a performance evaluator EV, which is coupled to machine learning modules NN1, NN2, and NN3. Furthermore, the first machine learning module NN1 is coupled to machine learning modules NN2 and NN3, and the second machine learning module NN2 is coupled to the third machine learning module NN3.

[0058] The performance evaluator (EV) is used to determine the performance of the behavior of the technical system TS triggered by a corresponding control action. For this purpose, a reward function Q is evaluated. The reward function Q determines and quantifies the reward, referred to here as the performance of the current system behavior. Such a reward function is often also called a cost function, loss function, objective function, reward function, or value function. The reward function Q can, for example, be implemented as a function of one or more nominal values ​​OB of the operating state, control action, and system behavior.

[0059] If multiple behavioral signals are evaluated through machine learning modules NN1, NN2, NN3 and / or through performance evaluator EV, multiple behavioral signal ratings OB can be pre-defined specifically for the corresponding behavioral signals.

[0060] To train the third machine learning module NN3, the running signal BS is fed as an input signal into the trained machine learning modules NN1 and NN2, as well as into the third machine learning module NN3.

[0061] Based on the operating signal BS, the trained first machine learning module NN1 reproduces the behavior signal VSR1 that would occur in the technical system TS if no control action is currently applied. The reproduced behavior signal VSR1 is fed from the first machine learning module NN1 into the second machine learning module NN2, the third machine learning module NN3, and the performance evaluator EV. In addition, one or more behavior signal ratings OB are fed into the third machine learning module NN3 and the performance evaluator EV.

[0062] The output signal AS (hereinafter referred to as the third output signal), obtained by the third machine learning module NN3 from the running signal BS, the reproduced behavior signal VSR1, and the rated values ​​OB of the one or more behavior signals, is also fed as an input signal into the trained second machine learning module NN2. Based on the third output signal AS, the reproduced behavior signal VSR1, and the running signal BS, the trained second machine learning module NN2 reproduces the control action-induced behavior signal VSR2 of the technology system TS, which is fed into the performance evaluator EV by the trained second machine learning module NN2.

[0063] The performance evaluator EV quantifies the current performance of the technical system TS based on the reproduced behavior signal VSR2, taking into account the reproduced first behavior signal VSR1 and the rated values ​​OB of the one or more behavior signals. In this case, the performance evaluator EV specifically determines a first deviation between the control-induced behavior signal VSR2 and the rated values ​​OB of the one or more behavior signals. In this case, a decrease in control action performance, typically observed as the deviation increases, is determined. Furthermore, a second deviation between the control-induced behavior signal VSR2 and the behavior signal VSR1 is determined. Based on the second deviation, the performance evaluator EV can evaluate how the system behavior is distinguished between the application of the control action and the absence of the control action. It has been demonstrated that this distinction can significantly improve performance evaluation in many cases.

[0064] The control performance determined by the reward function Q is as follows: Figure 4 As indicated by the dashed arrow, the input is redirected back to the third machine learning module, NN3. NN3 is then trained to maximize the control action performance based on the redirected control action performance. As mentioned several times above, numerous known optimization methods can be used to maximize control action performance.

[0065] As long as the second machine learning module NN2, in particular, anticipates the control action signal as its input signal to a certain extent, the third machine learning module NN3 is implicitly trained to output such a control action signal, denoted here as AS. By optimizing the control action performance, the third machine learning module NN3 is thus trained to output the optimized control action signal AS.

[0066] By using the reproduced behavior signal VSR1 in addition to the running signal BS to train the third machine learning module NN3, the third machine learning module NN3 can be trained particularly effectively because specific information about the behavior of the system without controlled actions is available to the third machine learning module NN3.

[0067] A particular advantage of this invention can be seen in that, in many cases, it is sufficient to evaluate only a single, adaptable time step (Zeitschritt) of the behavioral signal for each corresponding time point via the performance evaluator EV. Often, it is not costly to determine or estimate future rewards. Thus, effects operating on different time scales (Zeitskalen) can also be considered efficiently.

[0068] Furthermore, the pre-given dataset of the running signals can be used multiple times to train the third machine learning module NN3 by pre-given varying behavioral signal ratings OB for the behavioral signal VSR2. This enables the learning of different, rating-specific control action signals from the same running signal, thereby achieving better coverage of the control action space.

[0069] By training the third machine learning module NN3, the control device CTL is configured to control the technology system TS in a performance-optimized manner using the control action signal AS of the trained third machine learning module NN3.

Claims

1. A computer-implemented method for configuring a control unit (CTL) for a technical system (TS), wherein... a) The operating signal (BS) of the technical system is fed into a first machine learning module (NN1), which is trained to reproduce the behavior signal of the technical system in a specific case where no control action is currently applied, based on the operating signal (BS) of the technical system, and outputs the reproduced behavior signal (VSR1) as a first output signal. b) The first output signal (VSR1) is fed into a second machine learning module (NN2), which is trained to reproduce the behavior signal obtained from the technical system based on the control action signal (AS) and output the reproduced behavior signal (VSR2) as a second output signal. c) The operating signal (BS) of the technical system is fed into the third machine learning module (NN3). d) The third output signal (AS) of the third machine learning module (NN3) is fed into the trained second machine learning module (NN2). e) Determine the control action performance (Q) based on the second output signal (VSR2). f) The third machine learning module (NN3) is trained to optimize the control action performance (Q) based on the operating signals (BS) of the technical system. g) The control device (CTL) is configured according to the third machine learning module (NN3) to control the technical system by means of the third output signal (AS) of the third machine learning module (NN3).

2. The method according to claim 1, characterized in that, The third machine learning module (NN3) is trained based on the first output signal (VSR1).

3. The method according to claim 1 or 2, characterized in that, The control action performance (Q) is determined based on a single time step of the behavior signal at the corresponding time point.

4. The method according to claim 1 or 2, characterized in that, The first part (SS1, VS1) and / or the second part (AS2, VS2) of the operating signal (BS) of the technical system are specifically selected based on whether the first part and / or the second part includes a control action, and The first part (SS1, VS1) of the operating signal (BS) excluding control actions is specifically used to train the first machine learning module (NN1) and / or the second part (AS2, VS2) of the operating signal (BS) including control actions is specifically used to train the second machine learning module (NN2).

5. The method according to claim 1 or 2, characterized in that, Read in the behavior signal rating (OB). The second output signal (VSR2) is compared with the behavior signal rating (OB), and The control action performance (Q) is determined based on the comparison results.

6. The method according to claim 5, characterized in that, The behavioral signal rating (OB) is fed into the third machine learning module (NN3), and The third machine learning module (NN3) is trained to optimize the control action performance (Q) based on the behavioral signal rating (OB).

7. The method according to claim 1 or 2, characterized in that, The control action performance (Q) is determined based on the first output signal (VSR1).

8. The method according to claim 7, characterized in that, The deviation between the first output signal (VSR1) and the second output signal (VSR2) is determined, and the control action performance (Q) is determined based on the deviation.

9. The method according to claim 1 or 2, characterized in that, The first machine learning module (NN1) and / or the second machine learning module (NN2) are trained to individually reproduce multiple behavioral signals of different processes operating in the technical system, and The control action performance (Q) is determined based on the reproduced behavioral signal.

10. The method according to claim 9, characterized in that, For the corresponding behavior signal (VS), a specific behavior signal rating (OB) is read in, and The control action performance (Q) is determined by comparing the reproduced behavior signal with the specific behavior signal rating.

11. The method according to claim 10, characterized in that, The third machine learning module (NN3) is trained to optimize the control action performance (Q) based on the specific behavioral signal rating (OB).

12. A control device (CTL) for a control technology system (TS), said control device being configured to perform the method according to any one of claims 11.

13. A computer program product configured to perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program product according to claim 13.

Images