Method and system for training reinforcement learning agent of industrial process system

Abstract

The embodiment of the invention relates to a method and system for training a reinforcement learning agent of an industrial process system. A method for training a reinforcement learning (RL) proxy of an industrial process system is provided. The method includes training an RL agent with plant history data of the industrial process system, and retraining the RL agent using the plant history data of the industrial process system and a low fidelity simulator. Retraining the RL agent includes analyzing plant history data to identify white points as regions of unexplored process states and dynamic behaviors during training of the RL agent, and retraining the RL agent by preferential exploration with information obtained from white dots and by simulating simulation data provided by the industrial process system with a low fidelity simulator.

Description

TECHNICAL FIELD

[0001] The present invention relates to the field of reinforcement learning agents for industrial process systems. Embodiments of the present application relate to a method for training a reinforcement learning agent for an industrial process system and a system for training a reinforcement learning agent for an industrial process system. BACKGROUND

[0002] Reinforcement learning based control is a promising new approach for industrial process control. Reinforcement learning agents can be used for industrial process systems. In order to be used in an industrial process system, a reinforcement learning agent must be trained for the specific industrial process system. Training of reinforcement learning agents for industrial process systems is challenging for a number of reasons.

[0003] Due to the high complexity of the samples, a large amount of experience, i.e. information about the process behavior, is required during training to reach a certain level of performance. Furthermore, due to safety and economic considerations, reinforcement learning agents cannot be trained on the actual industrial process system. One solution is to train the reinforcement learning agent using a high-fidelity process simulator. However, the development of a high-fidelity process simulator requires a large amount of time, money and computational cost. Furthermore, due to the inherent mismatch between the model on which the training is performed and the actual process, and further due to changes in the dynamics of the industrial process system, the performance of the reinforcement learning agent can not be optimal. Moreover, to prevent costly malfunctions and accidents, the agent can only explore the safe region of the industrial process system during training and deployment.

[0004] Therefore, there is a need for more efficient methods and systems for training reinforcement learning agents for industrial process systems.

[0005] The present invention solves these problems by providing a method for training a reinforcement learning agent for an industrial process system and a system for training a reinforcement learning agent for an industrial process system according to the independent claims. SUMMARY

[0006] Reference will now be made in detail to various embodiments of the present disclosure, one or more examples of which are illustrated in the figures. Generally, only the differences from the various embodiments are described. Each example is provided by way of explanation of the present disclosure and is not meant as a restriction of the present disclosure. Features of one embodiment can be used in other embodiments or in combination with other embodiments to produce yet other embodiments. This description is intended to include such modifications and variations.

[0007] Reinforcement learning (RL) based control stands out compared to traditional control methods such as proportional-integral-derivative (PID) control, model predictive control (MPC), and advanced process control (APC) because it can handle complex systems with nonlinear behavior. It is able to learn from interactions with the process and optimize different objectives, which makes it a dynamic solution. Moreover, RL control excels in adapting to evolving operating conditions and disturbances while significantly reducing the manual modeling and tuning work associated with traditional industrial control strategies.

[0008] Applying reinforcement learning to industrial process control, e.g. providing a reinforcement learning agent, comes with challenges. Sample efficiency is one of the main challenges for RL agents in industrial process systems, especially during training and maintenance of the RL agent. RL algorithms traditionally require a large number of interactions with the process and / or environment to learn an effective policy. It is therefore an object of the present invention to reduce the number of samples required to train a RL agent for an industrial process and / or to more efficiently provide samples for training a RL agent, especially for achieving superior performance.

[0009] In many industrial environments, it can not be feasible and / or safe to train a RL agent from scratch in the actual operating environment. To achieve the desired accuracy of the RL agent, a high-fidelity simulator can be used during the training phase. A drawback of high-fidelity simulators is that they are expensive and executing these simulators can result in training times of months. Moreover, many industrial systems have high-dimensional state and action spaces. Traditional RL algorithms can struggle to effectively navigate in such complex environments. It is therefore an object of the present invention to provide a method that is able to handle continuous and high-dimensional spaces.

[0010] Furthermore, because a RL agent must not only obtain an optimal policy, safety first, it must also adhere to strict safety constraints while exhibiting resilience to changes in the environment. It is a further object of the present invention to establish mechanisms that adapt in real-time to evolving conditions over time and ensure their robustness.

[0011] The present invention solves the above problems and objects by providing an overall and systematic approach for training a reinforcement learning agent for an industrial process system.

[0012] The method according to the present application provides an accelerated, safe and resilient training and maintenance framework that forms a pipeline for RL agent development and lifecycle management. The RL agent is trained using a combination of historical closed loop data and low-fidelity simulators. The RL agent can use multiple low-fidelity simulators. The RL agent can be further adapted to changing processes. Beneficially, sample efficient training is achieved using historical data which reduces the amount of interaction with the simulator or actual process. This further results in the agent training in safe regions in the state space as it only explores regions that have been experienced historically. Furthermore, a safety verifier and safety enforcer can be provided to beneficially ensure safety of the process during training and deployment. Advantageously, the low-fidelity simulator with uncertainty considerations is robust without a high-fidelity simulator, improves optimality of learning a policy from historical closed loop data, and can be used to learn unexplored regions. Adaptive capability is introduced through a monitor and fine-tune module that continuously monitors the performance of the agent and adjusts the policy of the agent to achieve optimal performance.

[0013] The method according to the present application for training a RL agent of an industrial process system beneficially improves the trade-off between balancing sample efficiency, exploration, safety, realism, and iterative feedback, making RL agent training and maintenance of industrial process systems more effective. The method and system according to the present application provide a structured and efficient approach for training a RL agent of a complex and dynamic industrial process system. Furthermore, the method and system according to the present application provide a facilitation for introducing RL agents into the industrial process control market. Beneficially allows for simplifying the training process, ensuring safe exploration, and being able to adapt to dynamic conditions.

[0014] In view of the above, a method for training a reinforcement learning agent of an industrial process system and a system for training a reinforcement learning agent of an industrial process system are provided. Further aspects, advantages and features are apparent from the dependent claims, the description and the figures.

[0015] According to an aspect, a method for training a reinforcement learning (RL) agent of an industrial process system is provided. The method comprises training the RL agent with plant historical data of the industrial process system, and retraining the RL agent using the plant historical data of the industrial process system and a low-fidelity simulator. The retraining of the RL agent comprises analyzing the plant historical data to identify white points as regions of process states and dynamic behaviors that are unexplored during training of the RL agent, and retraining the RL agent with information obtained from the white points and by prioritized exploration with simulated data provided by simulating the industrial process system with the low-fidelity simulator.

[0016] The method is for training a reinforcement learning (RL) agent for an industrial process system. The industrial process system can be a refining system, a mining system, a pulp and paper making system, a cement production system, a power generation system, or any other suitable type of industrial process system. In particular, the industrial process system can be any type of industrial process system that can benefit from an RL agent, in particular from an RL agent that controls the operation of the industrial process system.

[0017] The method includes training the RL agent with plant historical data. The method for training the RL agent can begin by training the RL agent with plant historical data, also referred to herein as closed loop historical data. The plant historical data can be collected on a plant, i.e. an industrial process system. The plant historical data can be collected over an extended period of time during operation of the plant. The plant historical data can be collected over an operation time of the plant of up to 1 week, 2 weeks, 3 weeks, 4 weeks, 2 months, 6 months, 1 year, 2 years, 5 years, or up to 10 years or even longer. The plant historical data can be collected on the same industrial process system for which the RL agent is trained. The plant historical data can be collected from an industrial process system that is similar to the industrial process system for which the RL agent is trained. For example, the same industrial process system can be replicated multiple times. The RL agent of the newly replicated industrial process system can be trained with plant historical data of the industrial process system that has been commissioned and is operating.

[0018] Training the RL agent with plant historical data can be an initial training phase of the RL agent. Training the RL agent with plant historical data advantageously provides the RL agent with valuable insights about the industrial process system. Advantageously, this allows the RL agent to make more informed decisions, making more correct decisions early in the training, and speeding up the learning process.

[0019] Retraining the RL agent using plant historical data of the industrial process system and a low-fidelity simulator. Retraining the RL agent can be completed after the initial training phase of the RL agent with plant historical data. Advantageously, retraining the RL agent allows to improve the performance of the RL agent.

[0020] Analyzing the plant historical data to identify white spots. White spots can be identified through a detailed analysis of the plant historical data. White spots are regions of process states and dynamic behavior that are unexplored during the training of the RL agent, in particular during the training of the RL agent with plant historical data. White spots can correspond to regions of process states and dynamic behavior that have not been experienced in the plant historical data and / or that have not been explored during the training of the RL agent with plant historical data. White spots can correspond to regions of process states and dynamic behavior that the industrial process system from which the plant historical data is taken has only experienced with limited capability or not at all.

[0021] An RL agent trained with plant historical data can have limited or no information about white spots at all. Beneficially, identifying white spots allows for the collection of information about white spots. White spots can serve as a focal point for RL exploration, in particular to collect information about process states and dynamic behavior of white spots. Knowledge about white spots, i.e. the existence of white spots, can be used to retrain the RL agent.

[0022] The knowledge about white spots (preferential exploration areas) obtained from the analysis of historical data, by interacting with a low-fidelity simulator, can be used to plan further training. Such training expands the scope of the agent to cope with unexpected conditions that are not present in regular historical data but can occur in the future. Since the low-fidelity simulator is computationally less expensive, the RL agent can perform such exploration and experimentation in a less resource-intensive environment.

[0023] The RL agent is retrained with the information obtained from white spots and with simulated data provided by simulating the industrial process system with a low-fidelity simulator by preferential exploration. The information obtained from white spots and the information from the low-fidelity simulator are combined in a synergistic way to retrain the RL agent. The low-fidelity simulator can provide simulated process environments, in particular simulated process environments that can be different from the process environments recorded in the plant historical data.

[0024] Throughout this disclosure, a process environment is understood to describe process-related properties of an industrial process system. A process environment can describe an operational state of an industrial process system. For example, a process environment can include information about the type of industrial process, the type and quantity of input products, the type and quantity of output products, manufacturing steps, distribution steps, storage of input products, storage of output products, and / or process behavior. An industrial process is a procedure involving chemical, physical, electrical, or mechanical steps to help manufacture one or more items, typically performed on a large scale. Simulated process environments can be similar and / or different from the process environments recorded in the plant historical data. The information about white spots can then be used to specifically explore white spots, i.e. process states and dynamic behavior that have not been explored before. The low-fidelity simulator can be fine-tuned to further improve white spot exploration. The low-fidelity simulator can be fine-tuned such that a higher quality of simulation is provided for process states in the vicinity of white spots. At the same time, the quality of simulation can be reduced for well-known process states, e.g. for process states that have already been explored with the plant historical data. The total processing power required for the low-fidelity simulator can thus remain the same while improving white spot exploration.

[0025] The low-fidelity simulator can be continually improved and / or adjusted to the industrial process system for which the RL agent is trained. The low-fidelity simulator can be initiated using plant historical data. The plant historical data can be used to fine-tune parameters of the low-fidelity simulator to ensure closeness to the actual industrial process system. The low-fidelity simulator can be a first low-fidelity simulator. In some embodiments, retraining of the RL agent can be accomplished using one or more second low-fidelity simulators. Retraining of the RL agent by prioritized exploration can further leverage information obtained from the one or more second low-fidelity simulators. The one or more second low-fidelity simulators can be similar to the first low-fidelity simulator. The one or more second low-fidelity simulators can be different from the first low-fidelity simulator. The one or more second low-fidelity simulators can be fine-tuned to provide high-quality simulation data for process states including and / or adjacent to the white point. The one or more second low-fidelity simulators can provide simulation data for a subsystem of the industrial process system.

[0026] Domain knowledge facilitated by the low-fidelity simulator is used to ensure feasibility and effectiveness of RL white point exploration. Knowledge gained from the white point is further used to determine the optimal number of agents required and fine-tune their objectives and reward structures to adapt to the complexity of the specific system. Using historical data and information about the white point determined during training reduces the amount of search space for high-dimensional systems, which can make the training process focused and more rapid.

[0027] The white point can be identified through a detailed analysis of plant historical data. The white point can be identified using variable space exploration. Variable space exploration is a data mining process that uses rules and / or algorithms to analyze the distribution of variable data in historical data. The white point can be identified by a lower bound and an upper bound of a variable. In some embodiments, analyzing plant historical data to identify the white point includes retrieving a lower bound of a process state variable and / or an upper bound of a process state variable, and identifying the white point by variable space exploration using the lower bound and / or the upper bound.

[0028] In industrial process systems, safety is important to avoid accidents that can lead to shutdowns and even harm to personnel. A safety verifier and / or a safety guarantor can be provided. Beneficially, the safety verifier and / or the safety guarantor can guarantee safety of the industrial process system throughout the training and deployment phase of the reinforcement learning agent of the industrial process system.

[0029] A safety verifier can be provided. The safety verifier can be based on safety relevant variables. The safety relevant variables can be related to manipulated variables of the industrial process system. The safety verifier can predict safety variables based on current values of the manipulated variables. The safety verifier can be constructed using plant historical data and / or information obtained during low-fidelity simulation runs. This can be done by inferring dynamics of the safety relevant variables from plant historical data and / or from the exploration policy during the low-fidelity simulation runs. Beneficially, this approach allows to disentangle the relationship between safety variables and manipulated variables.

[0030] The manipulated variables can be variables of the industrial process system. The manipulated variables can be active variables, which can be actively manipulated to manipulate operation of the industrial process system. The manipulated variables can be passive variables, which depend on other variables of the industrial process system, e.g. on active variables and / or other passive variables of the industrial system.

[0031] The safety verifier can monitor some or all of the manipulated variables of the industrial process system. The manipulated variables can be related by the safety verifier to safety relevant variables. One or more manipulated variables can be related by the safety verifier to one safety relevant variable. The same manipulated variable can be related to one or more safety relevant variables. The safety verifier can determine safety relevant variables based on the manipulated variables.

[0032] Safety constraints can be provided to the safety verifier. The safety constraints can comprise safety parameters associated with safety variables. The safety constraints, in particular the safety parameters, can be predetermined. The safety constraints, in particular the safety parameters, can depend on the industrial process system for which the reinforcement learning agent is trained. The safety parameters can define a safety range for the associated safety variable. The safety variable can be safe within the safety range, and / or can be unsafe outside the safety range. The safety parameters can define an upper bound and / or a lower bound, below which the associated safety variable is safe, above which the associated safety variable is safe.

[0033] The safety verifier can be configured to determine safe or unsafe operation of the industrial process system based on the manipulated variables and the safety constraints, in particular based on the safety variables and the safety parameters. The safety verifier can monitor the manipulated variables and determine the associated safety variables. The safety verifier can compare the safety parameters and the associated safety variables. When a safety variable is within a safety range defined by an associated safety parameter, the safety variable can be labeled as safe. When a safety variable is outside the safety range, the safety variable can be labeled as unsafe. If no safety variable is labeled as unsafe, in particular if all safety variables are labeled as safe, the industrial process system can be in a safe operating state. If one or more safety variables are labeled as unsafe, the industrial process system can be in an unsafe operating state. The safety verifier can adjust the manipulated variables to ensure compliance with the safety constraints. The safety verifier can monitor and adjust the manipulated variables before a corresponding safety variable is labeled as unsafe. The safety verifier can adjust the manipulated variables after a corresponding safety variable is labeled as unsafe, in particular adjust the manipulated variables such that the corresponding safety variable is labeled as safe. Beneficially, the safety verifier ensures that the RL agent complies with the safety constraints both during training and agent deployment.

[0034] According to embodiments which can be combined with other embodiments described herein, the method comprises inferring dynamics of safety-relevant variables from plant historical data and / or by prioritized exploration; and utilizing the dynamics of the safety-relevant variables to build a safety verifier configured to predict safety variables based on values of manipulated variables. Furthermore, the predicted safety variables can be compared to predetermined safety constraints, and the values of the manipulated variables can be adjusted to ensure that the safety variables comply with the safety constraints.

[0035] A safety guarantor can be provided. The safety guarantor can be provided in addition to the safety verifier. The safety guarantor can be provided as a failsafe for the safety verifier. The safety guarantor can take effect in case the safety verifier fails. For example, the safety verifier can experience a failure due to insufficient learning of the relationship between the manipulated variables and the safety variables. The safety guarantor can take effect when one or more safety variables are labeled as unsafe. For example, the safety guarantor can take effect if more than a predetermined number of safety variables are labeled as unsafe to avoid a failure of the safety verifier due to an overload. The safety guarantor can maneuver the industrial process system into a predefined safe state. That is, the safety guarantor can bring the industrial process system into a predefined safe state. Beneficially, the safety guarantor can ensure that costly downtime is avoided. The safety guarantor together with the safety verifier ensures safety of the process. According to embodiments, in case the safety verifier fails due to insufficient learning and / or the safety variables not complying with the safety constraints, the safety guarantor maneuvers the industrial process system into a predefined safe state.

[0036] RL agents for industrial process systems can be fine-tuned to improve the performance of the RL agent. RL agents trained with plant historical data and on a low-fidelity simulator can be fine-tuned using a high-fidelity simulator and / or by interacting with an actual industrial process system, i.e., the industrial process system for which the RL agent was trained. Fine-tuning by interacting with an actual process beneficially allows learning of a policy that bridges the gap between simulation and reality. The RL agent can be deployed to an industrial process system before fine-tuning the RL agent by interacting with the actual industrial process system. Fine-tuning the RL agent can occur after the trained RL agent is deployed to the industrial process system.

[0037] RL agents trained with plant historical data and on a low-fidelity simulator already have knowledge of the industrial process system. Advantageously, this knowledge can be leveraged to simplify fine-tuning. Fine-tuning can be computationally lean and safer. Compared to training the RL agent directly using a high-fidelity simulator, the interaction of the RL agent with the computationally expensive high-fidelity simulator can be reduced. Furthermore, the RL agent that already knows the industrial process system can ensure improved safety when interacting with the actual industrial process system. Safety can be further improved when a safety verifier and / or safety guarantor is provided. In some embodiments, a machine learning based proxy model created from high-fidelity simulation data and plant historical data can be used to speed up fine-tuning. This can help overcome CPU time limitations.

[0038] According to embodiments, which can be combined with other embodiments described herein, the method comprises fine-tuning the RL agent by using a high-fidelity simulator, by interaction of the RL agent with the industrial process system, by using plant historical data, or by a combination thereof.

[0039] After training the RL agent using plant historical data and a low-fidelity simulator, and possibly fine-tuning, the RL agent can be deployed. The RL agent can be deployed to the industrial process system for which the RL agent has been trained. The RL agent can be deployed to more than one industrial process system. The RL agent can be trained for a first industrial process system. The RL agent can be deployed to the first industrial process system and / or one or more second industrial process systems. The one or more second industrial process systems can be similar and / or identical to the first industrial process system, i.e., the one or more second industrial process systems can be a replica of the first industrial process system. For example, the one or more second industrial process systems can be of the same type as the first industrial process system. The first industrial process system and the one or more second industrial process systems can be installed at the same manufacturing plant or location. The first industrial process system and the one or more second industrial process systems can be installed at different manufacturing plants or locations. According to embodiments, which can be combined with other embodiments described herein, the method comprises deploying the RL agent to an industrial process system.

[0040] After deploying the RL agent to the industrial process system, the RL agent can be continuously monitored. The performance and behavior of the RL agent deployed to the industrial process system can be continuously monitored by collecting historical data, by collecting rewards, and by continuously analyzing the interactions of the RL agent. The RL agent policy can be refined by adjusting the policy parameters, i.e. fine-tuning the RL agent, and / or exploring new actions. After monitoring the RL agent for a predetermined period of time, the policy parameters can be refined. The modified policy parameters can be obtained based on the monitored performance and / or behavior of the RL agent. The modified policy parameters can be directly applied to the RL agent deployed in the industrial process system, i.e. rolled out to the industrial process system. Preferably, the obtained modified policy parameters are validated offline using the collected historical data to assess the impact of the modified policy parameters on the performance. In case the modified policy parameters pass the validation, the modified policy parameters can be applied to the RL agent deployed in the industrial process system, in particular the modified policy parameters can be systematically rolled out to the industrial process system. This iterative feedback loop can be repeated periodically to ensure continuous enhancement of the RL agent performance.

[0041] According to embodiments which can be combined with other embodiments described herein, the method comprises fine-tuning the RL agent policy by iteratively performing the following steps: monitoring the performance and behavior of the RL agent by collecting historical data and rewards, analyzing the performance and behavior of the RL agent, fine-tuning the RL agent policy by adjusting the policy parameters and exploring new actions to obtain an updated RL agent policy, validating the updated RL agent policy offline using the collected historical data to assess its impact on the performance due to the policy changes, and rolling out the updated RL agent policy systematically to the industrial process upon determining that the updated RL agent policy passes the validation.

[0042] According to an aspect, a system for training a reinforcement learning (RL) agent of an industrial process system is provided. The system comprises a data storage medium configured for storing plant historical data of the industrial process system and a low-fidelity simulator of the industrial process system.

[0043] A system for training a reinforcement learning (RL) agent of an industrial process system can be configured to perform a method for training a RL agent as described herein. Components of the system, e.g. a low-fidelity simulator, can be components described with respect to the method for training a RL agent as described herein. A data storage medium can store plant history data of the industrial process system for which the RL agent is trained. The system can comprise one or more second low-fidelity simulators. According to embodiments which can be combined with other embodiments described herein, the system comprises a computer-implemented medium storing instructions which, when executed, cause the system for training a RL agent to perform a method of training a RL agent according to the invention. The system can comprise a training module configured to train a RL agent according to the method of the invention. The training module can use the system for training a reinforcement learning (RL) agent to perform steps of a method of training a RL agent as described herein. The training module can have a controller, or the training module can be a controller. The controller can be configured to use the system for training a reinforcement learning (RL) agent to perform steps of a method of training a RL agent as described herein.

[0044] In some embodiments, the system comprises a safety verifier configured to predict a safety variable based on a value of a manipulated variable. The system can further comprise a safety guarantor configured to manipulate the industrial process system to a predefined safety state in case the safety verifier fails. The safety verifier and / or the safety guarantor can be configured as described herein. According to embodiments, the system comprises a high-fidelity simulator of the industrial process system.

[0045] Those skilled in the art will realize additional features and advantages upon reading the following detailed description, and upon viewing the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order that the foregoing aspects and features of the present disclosure can be understood in detail, a more particular description will be rendered by reference to example embodiments. The accompanying drawings will be referenced, and will be described in the following detailed description:

[0047] Figure 1 is a schematic diagram of a method for training a reinforcement learning agent of an industrial process system.

[0048] Figure 2 is a schematic diagram of a system for training a reinforcement learning agent of an industrial process system.

[0049] Figure 3 is a schematic diagram of a system for training a reinforcement learning agent of an industrial process system. DETAILED DESCRIPTION

[0050] Reference will now be made in detail to the various embodiments, one or more examples of which are illustrated in each of the figures. Each example is provided by way of explanation of the embodiments and is not meant as a restriction of the embodiments. For example, features illustrated or described as part of one embodiment can be used with any other embodiment or in combination with any other embodiment to produce yet another embodiment. It is intended that the present disclosure include such modifications and variations.

[0051] Figure 1 is a schematic illustration of a method 100 for training a reinforcement learning (RL) agent for an industrial process system.

[0052] At step 110, the RL agent is trained with plant historical data of the industrial process system.

[0053] At step 120, the RL agent is retrained using plant historical data and a low-fidelity simulator of the industrial process system. The retraining of the RL agent is completed in steps 130 and 140.

[0054] At step 130, the plant historical data is analyzed to identify white points as regions of process states and dynamic behaviors that are unexplored during training of the RL agent. The analysis of the plant historical data to identify white points can include retrieving lower bounds of process state variables and / or upper bounds of process state variables, and identifying white points by variable space exploration using the lower bounds and / or the upper bounds.

[0055] At step 140, the RL agent is retrained by prioritized exploration with information obtained from the white points and with simulated data provided by simulating the industrial process system with the low-fidelity simulator.

[0056] Optionally, a safety verifier can be constructed to train the RL agent. Dynamics of safety-relevant variables can be inferred from the plant historical data and / or by prioritized exploration. The dynamics of the safety-relevant variables can be utilized to construct a safety verifier configured to predict safety variables based on values of manipulated variables. Further, the predicted safety variables can be compared with predetermined safety constraints, and values of the manipulated variables can be adjusted to ensure that the safety variables comply with the safety constraints. In case the safety verifier fails due to insufficient learning and / or the safety variables do not comply with the safety constraints, the safety guarantor can maneuver the industrial process system to a predefined safe state.

[0057] The RL agent can be fine-tuned by using a high-fidelity simulator, by interacting the RL agent with the industrial process system, by using plant historical data, or by a combination thereof. Further, the RL agent policy can be fine-tuned by iteratively performing the following steps: monitoring the performance and behavior of the RL agent by collecting historical data and rewards, analyzing the performance and behavior of the RL agent, fine-tuning the RL agent policy by adjusting policy parameters and exploring new actions to obtain an updated RL agent policy, validating the updated RL agent policy offline using the collected historical data to assess its impact on performance due to policy changes, and systematically rolling out the updated RL agent policy to the industrial process upon determining that the updated RL agent policy passes validation.

[0058] Figure 2 is a schematic diagram of a system 200 for training a reinforcement learning (RL) agent 270. Figure 2 Further shown is a flow diagram of training a RL agent according to embodiments described herein. A data storage medium 210 is provided, which is configured for storing plant historical data. The RL agent can be trained 220 with plant historical data of the industrial process system. Further, the plant historical data can be analyzed 250 to identify white points.

[0059] The system 200 comprises a low-fidelity simulator 240. The low-fidelity simulator can be constructed using plant historical data. The plant historical data can be used to fine-tune parameters of the low-fidelity simulator 240 to ensure closeness to the actual industrial process system. It can be determined 230 whether the low-fidelity simulator 240 needs to be tuned before plant historical data is provided to the low-fidelity simulator 240. If tuning is required, the low-fidelity simulator 240 can be fine-tuned, in particular using plant historical data. The plant historical data can be provided to the low-fidelity simulator.

[0060] The RL agent is retrained by prioritized exploration 260 with information obtained from the white points and with simulated data provided by the low-fidelity simulator 240. The low-fidelity simulator 240 can provide simulated data based on plant historical data. The low-fidelity simulator 240 can simulate the industrial process, in particular by using plant historical data received from the data storage medium 210.

[0061] Figure 3is a schematic diagram of a system 200 for training an RL agent 270. The system 200 further comprises a safety verifier 280 and a safety guarantor 285. The safety verifier 280 can be configured to determine whether an industrial process system is operating safely based on manipulated variables of the industrial process system and safety constraints of the industrial process system. In addition to the safety verifier 280, a safety guarantor 285 can be provided. The safety guarantor 285 can be provided as a failsafe for the safety verifier. The safety guarantor can come into effect in case of a failure of the safety verifier.

[0062] The RL agent 270 trained with plant history data and on a low-fidelity simulator can be fine-tuned using a high-fidelity simulator 310 and / or by interacting 320 with the actual industrial process system, i.e. the industrial process system for which the RL agent is trained. The RL agent 270 can be deployed 300 to the industrial process system. The deployed RL agent 300 can be fine-tuned using the high-fidelity simulator 310 and / or by interacting 320 with the industrial process system.

[0063] The deployed RL agent 300 can further be fine-tuned by fine-tuning the RL agent policy. The RL agent policy can be fine-tuned by iteratively monitoring 330 the performance and behavior of the RL agent, analyzing 340 the performance and behavior of the RL agent, fine-tuning 350 the RL agent policy by adjusting policy parameters and exploring new actions, offline validating 360 the updated RL agent policy using collected history data to assess its impact on performance due to policy changes, and systematically promoting 370 the updated RL agent policy to the industrial process upon determining that the updated RL agent policy passes validation. If it is determined that the updated RL agent policy does not pass validation, the process of fine-tuning the RL agent policy can return to the step of monitoring 330 the performance and behavior of the RL agent. This can be done as often as needed, e.g. until the updated RL agent policy passes validation. The RL agent policy can be fine-tuned continuously and / or after predetermined time intervals during deployment of the RL agent.

[0064] Within the following description of the drawings, same reference numbers represent same or similar components. In general, only the differences with respect to the various embodiments are described. The description of parts or aspects in one embodiment can also apply to corresponding parts or aspects in another embodiment, unless otherwise specified.

[0065] While the foregoing is directed to embodiments, other and further embodiments can be devised without departing from the basic scope, and the scope thereof is determined by the claims that follow.

Claims

1. A method for training a reinforcement learning (RL) agent for an industrial process system, the method comprising: The RL agent is trained using historical factory data from the industrial process system. as well as The RL agent is retrained using historical factory data from the industrial process system and a low-fidelity simulator. This retraining of the RL agent includes: Analyze the factory's historical data to identify white spots as regions of process states and dynamic behaviors that were not explored during the training of the RL agent; as well as The RL agent is retrained by prioritizing exploration using information obtained from the white points and simulation data provided by simulating the industrial process system using the low-fidelity simulator.

2. The method of claim 1, wherein analyzing the factory historical data to identify white spots comprises: Retrieve at least one boundary selected from the group consisting of lower bounds and upper bounds for process state variables; as well as The white point is identified by exploring the variable space using at least one of the boundaries selected from the group consisting of the lower and upper boundaries.

3. The method according to any one of claims 1 to 2, comprising: The dynamics of safety-related variables can be inferred from at least one piece of factory historical data or through the aforementioned priority exploration; as well as A security verifier is constructed using the dynamics of the security-related variables, and the security verifier is configured to predict security variables based on the values ​​of the manipulated variables.

4. The method according to claim 3, comprising: Compare the predicted security variables with the predetermined security constraints; as well as Adjust the value of the manipulated variable to ensure that the safety variable conforms to the safety constraints.

5. The method of claim 3, wherein in the event that the safety verifier fails due to at least one event selected from the group consisting of insufficient learning and the safety variable not conforming to the safety constraints, the industrial process system is manipulated to a predefined safety state by the safety guarantor.

6. The method of claim 4, wherein in the event that the safety verifier fails due to insufficient learning and / or the safety variable does not conform to the safety constraints, the industrial process system is manipulated to a predefined safety state by the safety guarantor.

7. The method according to any one of claims 1 to 2, comprising: The RL agent can be fine-tuned by using a high-fidelity simulator, by interacting with the industrial process system through the RL agent, by using plant history data, or by a combination thereof.

8. The method according to claim 3, comprising: The RL agent can be fine-tuned by using a high-fidelity simulator, by interacting with the industrial process system through the RL agent, by using plant history data, or by a combination thereof.

9. The method according to claim 4, comprising: The RL agent can be fine-tuned by using a high-fidelity simulator, by interacting with the industrial process system through the RL agent, by using plant history data, or by a combination thereof.

10. The method of claim 5, comprising: The RL agent can be fine-tuned by using a high-fidelity simulator, by interacting with the industrial process system through the RL agent, by using plant history data, or by a combination thereof.

11. The method of claim 6, comprising: The RL agent can be fine-tuned by using a high-fidelity simulator, by interacting with the industrial process system through the RL agent, by using plant history data, or by a combination thereof.

12. The method according to any one of claims 1 to 2, comprising: Deploy the RL agent to the industrial process system.

13. The method of claim 12, further comprising fine-tuning the RL agent strategy by iteratively performing the following steps: The performance and behavior of the RL agent are monitored by collecting historical data and rewards; Analyze the performance and behavior of the RL agent; The RL agent policy is fine-tuned by adjusting policy parameters and exploring new actions to obtain an updated RL agent policy; The updated RL agent policy is validated offline using the collected historical data to assess its impact on performance due to the policy change. as well as After confirming that the updated RL agent strategy has passed the verification, the updated RL agent strategy is systematically promoted to the industrial process.

14. A system for training reinforcement learning (RL) agents for industrial process systems, the system comprising: A data storage medium is configured to store historical factory data of the industrial process system. as well as The low-fidelity simulator of the industrial process system.

15. The system of claim 14, further comprising a security verifier configured to predict a security variable based on the value of the manipulated variable.

16. The system of claim 15, further comprising a safety guarantor configured to manipulate the industrial process system to a predefined safety state in the event of a failure of the safety guarantor.

17. The system according to any one of claims 14 to 16, comprising a high-fidelity simulator of the industrial process system.

18. The system according to any one of claims 14 to 16, comprising a training module configured to train an RL agent according to any one of claims 1 to 2.

19. The system of claim 17, further comprising a training module configured to train an RL agent according to any one of claims 1 to 2.

Images