Method for configuring a control agent for a technical system and control device
By training the control agent using a dynamic model and a variational autoencoder, and optimizing the action dataset using reproduction errors, the problems of large training data requirements and prediction error accumulation are solved, thereby improving the training efficiency and performance of the control system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SIEMENS AG
- Filing Date
- 2021-07-12
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies require a large amount of training data when training control agents and are easily affected by the accumulation of prediction errors, resulting in unfavorable control behavior in uncovered state space regions.
By training the control agent using a dynamic model and a variational autoencoder, and optimizing the action dataset using a performance function, the reproduction error is used as a parameter to reduce the impact of performance, thereby reducing the training data requirement and the cumulative effect of prediction error.
This approach enables successful training of the control agent with limited training data, reducing the negative impact of uncovered state space regions and improving the training efficiency and performance of the control system.
Smart Images
Figure CN115867920B_ABST
Abstract
Description
Background Technology
[0001] Data-driven machine learning methods are increasingly being used to control complex technological systems, such as gas turbines, wind turbines, steam turbines, engines, robots, chemical reactors, milling machines, manufacturing systems, refrigeration systems, or heating systems. In this context, artificial neural networks, in particular, are trained using reinforcement learning methods to generate optimized, state-specific control actions for controlling the technological system in response to its corresponding states. This control agent trained to control the technological system is often referred to as a policy or simply an agent.
[0002] To successfully train a control agent, a large amount of operational data from the technical system is typically required as training data. This training data should, as comprehensively as possible, cover the operational states and other operating conditions of the technical system.
[0003] In many cases, this type of training data exists in the form of databases that store operational data recorded at the technical system. This stored training data is often referred to as batch training data or offline training data. Experience shows that the success of training usually depends on the extent to which this batch training data covers the possible operating conditions of the technical system. Correspondingly, it can be expected that a control agent trained using batch training data will perform poorly in operating conditions where only a small amount of batch training data exists for it. To improve control behavior in state space regions with limited training data coverage, the publications "Off-Policy Deep Reinforcement Learning without Exploration" (pp. 2052-2062) by S. Fujimoto, D. Meger, and D. Precup at the 2019 International Conference on Machine Learning, and "Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction" (pp. 11761-11771) by A. Kumar, J. Fu, G. Tucker, and S. Levine in Advances in Neural Information Processing Systems, proposed limiting selectable control actions through suitable constraints and training a so-called Q-function based on these constraints. However, training the Q-function typically requires a large amount of data. Summary of the Invention
[0004] The objective of this invention is to describe a method for configuring a control agent for a technical system and a control device for controlling the technical system, the method and the control device allowing for more efficient training and / or requiring less training data.
[0005] The task is accomplished by a method having the features of claim 1, a control device having the features of claim 9, a computer program product having the features of claim 10, and a computer-readable storage medium having the features of claim 11.
[0006] To configure a control agent for a technical system, pre-given training data is read in. This training dataset includes a state dataset specifying the state of the technical system, an action dataset specifying a control action, and performance values obtained by the technical system due to the application of that control action. Based on this training data, a data-driven dynamic model is trained to reproduce the obtained performance values based on the state dataset and the action dataset. Furthermore, based on this training data, an action evaluation process is trained to reproduce the action dataset after information reduction based on the state dataset and the action dataset, wherein a reproduction error is determined. Training data is then fed to the trained dynamic model, the trained action evaluation process, and the control agent. Here, the performance values output by the trained dynamic model are fed into a pre-given performance function. Furthermore, the reproduction error output by the trained action evaluation process is fed into this performance function as a parameter affecting performance reduction. Additionally, the control agent is trained to output an action dataset optimized for this performance function based on the state dataset. Through training, the control agent is configured to optimally control the technical system.
[0007] To perform the method according to the present invention, a corresponding control device, a computer program product, and a computer-readable storage medium, preferably a non-volatile storage medium, are provided.
[0008] The method and control device according to the invention can be implemented or realized, for example, by means of one or more computers, processors, application-specific integrated circuits (ASICs), digital processors (DSPs), and / or so-called "field programmable gate arrays" (FPGAs).
[0009] The advantages of this invention are particularly evident in that it typically requires far less training data for successful training compared to training the Q-function. Furthermore, the method according to this invention is in many cases less susceptible to the accumulation of prediction errors compared to training the Q-function.
[0010] Furthermore, it can be observed that the reproduction error is generally smaller in states and control action regions well covered by the training data than in poorly covered regions. Therefore, the reproduction error can generally be understood as a measure of the degree to which the corresponding control action is covered by the training data in the corresponding state. As long as the reproduction error is used as a parameter affecting performance degradation when training the control agent, the negative impact of poorly covered regions on training success can be effectively reduced.
[0011] Advantageous embodiments and extensions of the invention are described in the dependent claims.
[0012] According to an advantageous embodiment of the invention, the action evaluation process can be performed by means of a variational autoencoder, an autoencoder, and / or a generative adversarial network. By using an autoencoder, or especially a variational autoencoder, a reduced and / or less redundant representation of the information in the fed state dataset and action dataset can be determined. Furthermore, a measure of the accuracy at which these action datasets can be reconstructed from the reduced parameters can be determined in a simple manner. This measure can advantageously be used as the reproduction error.
[0013] Furthermore, to train the control agent, the state dataset can be fed into the control agent, and the state dataset and the action dataset output by the control agent can be fed into a trained dynamic model and a trained action evaluation process, respectively. In this way, the action dataset output by the control agent to be trained can be directly evaluated through the action evaluation process.
[0014] According to another advantageous embodiment of the invention, the corresponding training dataset may include a subsequent state dataset, which specifies the subsequent states of the technical system obtained due to the application of control actions. Then, based on this training data, the dynamic model can be trained to reproduce the resulting subsequent state dataset based on the state dataset and the action dataset. Next, during the training of the control agent, the subsequent state dataset output by the trained dynamic model can be fed to the trained dynamic model, the trained action evaluation process, and / or the control agent, respectively. Then, the performance values and / or reproduction errors determined for the fed subsequent state dataset can be fed into a performance function. In this way, the states and control actions can be progressively extrapolated or predicted to positions, thus determining a control trajectory comprising multiple time steps. This extrapolation is often referred to as roll-out or virtual roll-out. Then, for the control trajectory, the accumulated performance over multiple time steps can be calculated using the performance function and preferably assigned to the control action at the beginning of the trajectory. In reinforcement learning, this accumulated performance is often referred to as reward. To calculate this return, the performance values determined for future time steps can be discounted, i.e., these performance values can be assigned weights that decrease with each time step.
[0015] To train the control agent, gradient-based optimization methods, stochastic optimization methods, particle swarm optimization, and / or genetic optimization methods can preferably be used. Several efficient implementations of the aforementioned methods are available. Attached Figure Description
[0016] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. Hereinafter, schematic diagrams are provided:
[0017] Figure 1 A control device according to the present invention is shown in a control technology system;
[0018] Figure 2 The training of a data-based dynamic model for a technical system is shown;
[0019] Figure 3 Training of the variational autoencoder is shown; and
[0020] Figure 4 The training of the control agent for the control device is shown. Detailed Implementation
[0021] If the same or corresponding reference numerals are used in different figures, these reference numerals denote the same or corresponding entities, which in particular can be realized or designed as described in conjunction with the relevant figures.
[0022] Figure 1The invention clarifies the control device CTL according to the present invention when controlling a technical system TS, such as a gas turbine, wind turbine, steam turbine, chemical reactor, milling machine, manufacturing system, factory, robot, motor vehicle, engine, refrigeration system, heating system, power supply equipment, or other machine, equipment, or system. In particular, components or subsystems of a composite technical system can also be understood as the technical system TS. The technical system TS has a sensing device SK for preferably continuously detecting and / or measuring the system device or subsystem status of the technical system TS. For the present embodiment, it should be assumed that the control of the gas turbine is the technical system TS.
[0023] exist Figure 1 In this embodiment, the control unit CTL is shown outside the technical system TS and coupled to the technical system. Alternatively, the control unit CTL may be fully or partially integrated into the technical system TS.
[0024] The control device CTL has: one or more processors PROC for executing method steps of the control device CTL; and one or more memories MEM coupled to the processors PROC for storing data to be processed by the control device CTL.
[0025] The control device CTL also features a learning-based control agent (POL) that can be trained using reinforcement learning methods. This control agent is often referred to as a policy or simply an agent. In the current embodiment, the control agent POL is implemented as an artificial neural network.
[0026] The control agent POL and the control device CTL are pre-trained in a data-driven manner based on pre-given training data and configured for optimized control of the technical system TS. Training data for training the control agent POL is obtained from a database DB, where this training data is stored as multiple training datasets TD. The stored training datasets TD are pre-recorded at the technical system TS or a similar technical system and / or generated in a simulated or data-driven manner. According to the invention, the control agent POL is trained based on this offline training data to determine optimized control actions for corresponding pre-given states of the technical system TS. The training process is described in more detail below.
[0027] After training is complete, the trained control agent (POL) can be used for optimized control of the technical system (TS). For this purpose, the state datasets (S) of the technical system (TS) are transmitted to the control unit (CTL). These state datasets (S) specify states, particularly the operating states of the technical system (TS), and are preferably represented by digital state vectors. These state datasets (S) may include measurement data, sensor data, environmental data, or other data that occurs or affects the operation of the technical system (TS). For example, data regarding the temperature, pressure, settings, actuator positions, valve positions, emissions, utilization rates, resource consumption, and / or power of the technical system (TS) or its components. In the case of a production system, these state datasets may also be related to product quality or other product attributes. These state datasets (S) can be measured, at least in part, by the sensors (SK) of the technical system (TS) or determined through simulation using a simulator of the technical system.
[0028] The state dataset S, to be transmitted to the control unit CTL, is fed as input data to the trained control agent POL. Based on the transmitted state dataset S, the trained control agent POL generates optimized control actions in the form of a corresponding action dataset A. In this case, the corresponding action dataset A specifies the control actions or corresponding control signals that can be performed at the technical system TS. The generated action dataset A is transmitted to the technical system TS by the control agent POL or the control unit CTL. The technical system TS is controlled in an optimized manner using the transmitted action dataset A, in a way that the technical system TS executes the control actions specified by these action datasets A.
[0029] Figure 2 The training of a dynamic model neural network (NN) for a technical system (TS) is explained. The dynamic model NN can be trained in a data-driven manner and is designed to model state transitions when control actions are applied to a pre-given state of the technical system TS. This dynamic model is often also referred to as a transition model or system model. The dynamic model NN can be implemented in a control unit (CTL) or can be implemented entirely or partially outside of the control unit. In the current embodiment, the dynamic model NN is implemented as an artificial neural network.
[0030] The dynamic model NN should be trained on the training dataset TD contained in the database DB to predict, with the highest possible accuracy, the subsequent state of the technical system TS resulting from the application of the control action, and the performance value derived therefrom, based on the corresponding state and the corresponding control action.
[0031] The dynamic model NN is trained using a training dataset TD stored in a database DB. In this case, the corresponding training dataset TD includes a state dataset S, an action dataset A, a subsequent state dataset S′, and performance values R. As mentioned above, the state datasets S specify the states of the technical system TS, and the action datasets A specify the control actions that can be performed at the technical system TS. Correspondingly, the subsequent states of the technical system TS are specified by the corresponding subsequent state dataset S′, resulting from applying the corresponding control actions to the corresponding states. Finally, the respective performance values R quantify the performance of performing the corresponding control actions in the corresponding states. In this case, the performance values R may specifically relate to the currently obtained power, currently obtained emissions, currently obtained resource consumption, currently obtained product quality, and / or other operating parameters obtained due to performing the current control actions. In the context of machine learning, these performance values are also represented by the terms "reward," "bonus," or—complementarily—"cost" or "loss."
[0032] To train the dynamic model NN, a state dataset S and an action dataset A are fed to the dynamic model NN as input data. The dynamic model NN should be trained so that the output data of the dynamic model reproduces the corresponding subsequent states and the corresponding performance values as accurately as possible. Preferably, this training is performed using a supervised machine learning method.
[0033] In this context, training should generally be understood as the optimization of the mapping from the input data of the machine learning model (here, the neural network), S and A, to its output data. This mapping is optimized during the training phase according to pre-given, learned, and / or to-be-learned criteria. Specifically, the success of control actions can be used as a criterion in a control model, or prediction error in a prediction model. Through this training, for example, the cross-linking structure of the neurons in the neural network and / or the weights of the connections between these neurons can be set or optimized to satisfy the pre-given criteria as well as possible. Therefore, this training can be understood as an optimization problem. For such optimization problems in the field of machine learning, various efficient optimization methods are available. Here, optimization should always be understood as an approximation of the optimal value.
[0034] In particular, it can train artificial neural networks, recurrent neural networks, convolutional neural networks, perceptrons, Bayesian neural networks, autoencoders, variational autoencoders, deep learning architectures, support vector machines, data-driven regression models, k-nearest neighbor classifiers, physical models, or decision trees.
[0035] In the case of a dynamic model neural network (NN), the state dataset S and action dataset A from the training data are fed into the dynamic model—as mentioned above. For the corresponding input dataset (S, A), the dynamic model NN outputs an output dataset OS′ as the subsequent state dataset and outputs the output data and OR as the performance value. The goal of training the dynamic model NN is to make the output dataset OS′ match the actual subsequent state dataset S′ as well as the output dataset OR match the actual performance value R as well as possible. To this end, the bias DP between the output dataset (OS′, OR) and the corresponding dataset (S′, R) contained in the training data is determined. In this case, the bias DP represents the prediction error of the dynamic model NN. This bias DP can be determined, for example, by calculating the Euclidean distance between the respective vectors, e.g., according to DP = (OS′ - S′). 2 +(OR-R) 2 .
[0036] like Figure 2 As shown by the dashed arrow, the bias DP is returned to the dynamic model NN. Based on the returned bias DP, the dynamic model NN is trained to minimize the bias DP and thereby the prediction error. Several efficient optimization methods are available to minimize the bias DP, such as gradient-based optimization methods, stochastic gradient descent, particle swarm optimization, and / or genetic optimization methods. By minimizing the bias DP, the dynamic model NN is trained to predict the subsequent states and performance values as well as possible for a given state and a given control action.
[0037] Figure 3 The training of a variational autoencoder (VAE) according to the present invention is explained, which acts as a data-driven action evaluation process. The VAE can be implemented in a control unit (CTL) or can be implemented entirely or partially outside the control unit. The VAE is used to evaluate the reproducibility of corresponding state-specific control actions. As explained in more detail below, this reproducibility is demonstrated to be a measure of the similarity between the corresponding state-specific control action and control actions present in the training data, or is demonstrated to be a measure of the probability in the training data.
[0038] In the current embodiment, the variational autoencoder (VAE) is implemented as a Bayesian neural network and includes an input layer IN, a hidden layer H, and an output layer OUT. Besides the hidden layer H, the VAE may have other hidden layers. The autoencoder is characterized by a significantly smaller hidden layer H, i.e., fewer neurons than the input layer IN or the output layer OUT.
[0039] The variational autoencoder (VAE) should be trained on training data read from the database DB to reproduce the fed action dataset A as accurately as possible based on the fed state dataset S and action dataset A—after information reduction due to the smaller hidden layer H. Here, the prediction error DR should also be determined.
[0040] For this purpose, the state dataset S and action dataset A from the training data are fed as input data into the input layer IN and processed through layers IN, H, and OUT. Finally, the processed data is output as output data OA by the output layer OUT, which aims to reproduce the fed action dataset A as accurately as possible according to the training objective. Preferably, the state dataset S is also fed into the hidden layer H.
[0041] If the input data, here S and A, must pass through a smaller hidden layer H to some extent and should be largely reconstructed from the smaller or reduced amount of data existing there according to the training objective, then a reduced representation of these input data is obtained in the hidden layer H. The variational autoencoder (VAE) learns efficient encoding or compression of these input data in this way. By also feeding the state dataset S directly into the hidden layer H, it is possible to achieve, in particular, that the action dataset A is encoded, reduced, or represented more efficiently in the hidden layer H.
[0042] In the hidden layer H, training is used to realize, in particular, the so-called latent parameter space or latent representation of the action dataset A or the control action specified therein. The data existing in the hidden layer H corresponds to an abstract and information-reduced representation of the state-specific control action contained in these training data.
[0043] To achieve the aforementioned training objective, an optimization method is implemented to set the processing parameters of the variational autoencoder (VAE) such that the reproduction error DR is minimized. In this case, the distance between the output data OA and the fed action dataset A can be specifically determined as the reproduction error or reproduction uncertainty DR, according to DR = (OA - A). 2 .
[0044] To train the variational autoencoder (VAE) or to optimize its processing parameters, the calculated distance DR—such as… Figure 3 The dashed arrow indicates a return to the variational autoencoder (VAE). Various efficient standard methods can be employed to execute this training.
[0045] After successful training, the trained variational autoencoder (VAE) can be used to evaluate any pair of state datasets S and corresponding action datasets A: how well the corresponding action dataset A can be reconstructed from the reduced representation based on the information in the hidden layer H. It can be expected that the reproduction error DR of state-control action pairs that frequently appear in these training data will be lower than that of state-control action pairs that rarely appear in these training data. That is, the reproduction error DR of the trained variational autoencoder (VAE) can be used as a measure of the extent to which the corresponding state-control action pair is covered by these training data, or the frequency or probability of that state-control action pair appearing there.
[0046] Figure 4 This paper elucidates the training of the control agent POL of the control device CTL using a trained dynamic model NN and a trained variational autoencoder VAE. To illustrate the sequential operational steps, in... Figure 4 The diagram illustrates multiple entities of the control agent POL, the trained dynamic model NN, and the trained variational autoencoder (VAE). These different entities can correspond to different calls or evaluations of routines, which are used to implement the control agent POL, the trained dynamic model NN, or the trained action evaluation process (VAE).
[0047] As mentioned above, the control agent POL should be trained to determine performance-optimized control actions in the form of an action dataset A, given a pre-defined state dataset S. This training is performed using training data stored in a database DB.
[0048] For this training, a large amount of state dataset S from the database DB is fed into the control agent POL as input data. In this case, the corresponding state dataset S specifies a given state of the technical system TS. Based on the given state, the subsequent state of the technical system TS due to the application of control actions is predicted by a trained dynamic model NN. Furthermore, the predicted subsequent state and the given state are evaluated by a trained variational autoencoder VAE.
[0049] For these purposes, the state dataset S is also fed as input data to the trained dynamic model NN and the trained variational autoencoder VAE.
[0050] Based on the corresponding state dataset S, output data A0 is derived by controlling the agent POL and output as the action dataset. The action dataset A0, together with the corresponding state dataset S, is fed into a trained dynamic model NN, which predicts subsequent states and outputs a subsequent state dataset S1 specifying the subsequent state, along with the associated performance value R1.
[0051] Furthermore, the action dataset A0, together with the corresponding state dataset S, is fed into a trained variational autoencoder (VAE), which determines and outputs the reproduction error D0 of the action dataset A0.
[0052] The subsequent state dataset S1 is then fed into the control agent POL, which uses it to derive the action dataset A1 for that subsequent state. Action dataset A1, along with the corresponding subsequent state dataset S1, is fed into a trained dynamic model NN, which uses this model to predict the next subsequent state and outputs a subsequent state dataset S2 specifying that next subsequent state, along with the associated performance value R2. Furthermore, action dataset A1, along with the subsequent state dataset S1, is fed into a trained variational autoencoder (VAE), which uses this model to determine and output the reproduction error D1 of action dataset A1.
[0053] The above method steps can be repeated iteratively, wherein the performance value and reproduction error of the next subsequent state are determined. The iteration can end when an interruption condition occurs, such as after a predetermined number of iterations. In this way, a control trajectory comprising multiple time steps, progressing from one subsequent state to another, can be determined, with associated performance values R1, R2, ... and reproduction errors D0, D1, ...
[0054] To determine the overall performance of the corresponding control trajectory, the determined performance values R1, R2, ... and the determined reproduction errors D0, D1, ... are fed into the performance function P.
[0055] The performance function P determines the expected overall performance RET for the corresponding control trajectory, which is the accumulated performance over multiple time steps; in reinforcement learning, this overall performance is often also referred to as the reward. The accumulated performance RET is then assigned to the control action at the beginning of the trajectory, here A0, and the current capability of the control agent POL is evaluated to determine the control action, here A0, with favorable optimization performance for the corresponding state dataset S. The accumulated performance RET is preferably calculated by the performance function P as the performance discounted over future time steps of the control trajectory.
[0056] For this purpose, a weighted sum RD of performance values R1, R2, ... is preferably calculated using a performance function P, where the weights of these performance values are multiplied by a discount factor of W < 1 for each future time step. In this way, it can be expressed as RB = R1 + R2 * W + R3 * W 2 The weighted sum is calculated using +... For the discount factor W, a value of 0.99 or 0.9 can be used, for example.
[0057] Furthermore, the determined reproduction errors D1, D1, ... are included in the calculation of the overall performance RET as a performance-reducing parameter through the performance function P. In reinforcement learning, this performance-reducing parameter is often also referred to as a penalty. In the current embodiment, the individual reproduction errors D0, D1, ... are summed to form the total reproduction error D = D0 + D1 + ... . Thus, the overall performance RET is calculated according to RET = RD - D.
[0058] To train the control agent POL, that is, to optimize the processing parameters of the control agent, the overall performance RET is used—such as… Figure 4 The dashed arrow indicates a return to the control agent POL. Here, the processing parameters of the control agent POL are set or configured to maximize the overall performance RET. Various efficient standard methods, especially reinforcement learning, can be employed for the specific execution of this training.
[0059] Because a high total reproduction error (D) reduces the overall trajectory performance (RET), the adverse effects of control actions with high reproduction errors on the training are effectively mitigated. As mentioned above, such control performance may be less covered by this training data and could negatively impact training success.
[0060] It has been demonstrated that the training success of the control agent POL can be significantly improved by following the method of the present invention, especially in the case of offline training. Furthermore, significantly less training data is typically required compared to training the Q-function.
[0061] After successfully training the control agent POL, the control agent can be combined as follows: Figure 1 As described, it is used for optimized control of the technical system TS.
Claims
1. A computer-implemented method for configuring a control agent (POL) for a technology system (TS), wherein... a) Read in pre-given training data (TD), wherein the corresponding training dataset includes a state dataset (S) specifying the state of the technical system, an action dataset (A) specifying the control action, and the performance values (R) of the technical system obtained by applying the control action. b) A data-based dynamic model (NN) is trained on the training data (TD) to reproduce the obtained performance values (R) based on the state dataset (S) and the action dataset (A). c) The Action Evaluation Process (VAE) is trained based on the training data (TD) to reproduce the action dataset (A) after information reduction based on the state dataset (S) and the action dataset (A), wherein a reproduction error (DR, D0, D1) is determined. The Action Evaluation Process (VAE) is performed by means of an autoencoder, wherein the reproduction error (DR, D0, D1) is understood as a measure of the degree to which the corresponding control action in the corresponding state is covered by the training data. d) Feed training data (TD) to the trained dynamic model (NN), the trained action evaluation process (VAE), and the control agent (POL), wherein The performance values (R1, R2) output by the trained dynamic model (NN) are fed into a pre-given performance function (P). - The reproduction errors (D0, D1) output by the trained Action Evaluation Process (VAE) are fed into the performance function (P) as parameters affecting performance degradation, and - The control agent (POL) is trained to output an action dataset (A) that optimizes the performance function (P) based on the state dataset (S). To train the control agent (POL), the state dataset (S) is fed into the control agent (POL), and the state dataset (S) and the action datasets (A0, A1) output by the control agent (POL) are fed into the trained dynamic model (NN) and the trained action evaluation process (VAE), respectively. e) The control agent (POL) is configured through the training for optimized control of the technology system (TS). f) The technology system (TS) is optimized by means of a configured control agent (POL), wherein the state dataset (S) of the technology system (TS) is measured by the sensing device of the technology system (TS), and the measured state dataset (S) is sent as input data to the configured control agent (POL). The configured control agent (POL) generates an optimized performance control action in the form of a corresponding action dataset (A) based on the corresponding sent state dataset (S), and the technology system (TS) executes the optimized performance control action.
2. The method according to claim 1, characterized in that, The action evaluation process is performed using variational autoencoders (VAEs) and / or generative adversarial networks.
3. The method according to claim 1, characterized in that, The corresponding training dataset includes a subsequent state dataset (S'), which specifies the subsequent states of the technical system (TS) obtained due to the application of the control actions. The dynamic model (NN) is trained based on the training data to reproduce the resulting subsequent state dataset (S') based on the state dataset (S) and the action dataset (A). During the training of the control agent (POL), the subsequent state datasets (S1, S2) output by the trained dynamic model (NN), the trained action evaluation process (VAE), and / or the control agent (POL) are respectively fed to the trained dynamic model (NN), and... The performance value (R2) and / or reproduction error (D1) determined for the subsequent state datasets (S1, S2) are fed into the performance function (P).
4. The method according to any one of claims 1-3, characterized in that, To train the control agent (POL), gradient-based optimization methods, stochastic optimization methods, particle swarm optimization, and / or genetic optimization methods are used.
5. The method according to any one of claims 1-3, characterized in that, The performance function (P) is used to calculate the weighted sum or weighted difference of the performance values (R1, R2) and the reproduction errors (D0, D1).
6. The method according to any one of claims 1-3, characterized in that, The dynamic model (NN), the control agent (POL), and / or the action evaluation process (VAE) include artificial neural networks, recurrent neural networks, convolutional neural networks, multilayer perceptrons, Bayesian neural networks, autoencoders, deep learning architectures, support vector machines, data-driven trainable regression models, k-nearest neighbor classifiers, physical models, and / or decision trees.
7. The method according to any one of claims 1-3, characterized in that, The technology system (TS) is a gas turbine, wind turbine, steam turbine, chemical reactor, milling machine, manufacturing system, plant, robot, engine, refrigeration system, heating system or other machine, other equipment or other system.
8. A control device (CTL) for a control technology system (TS), said control device being configured to implement the method according to any one of claims 1-7.
9. A computer program product configured to perform the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program product according to claim 9.