Controlling industrial processes using virtualized instances of control software

Abstract

Embodiments of the present disclosure relate to controlling an industrial process using a virtualized instance of control software, comprising the steps of providing process data of the industrial process as input to an active instance of the control software, wherein the process data comprises a set of process variables of the industrial process and / or plant variables of an industrial plant executing the process; updating, based on the input, an internal state of the active instance of the control software, characterized by a set of internal variables of the active instance; determining, based on the updated internal state, an output signal by the active instance and outputting the output signal to at least one device configured to alter a state and / or behavior of the industrial process; and providing a prioritized subset of the internal variables characterizing the state of the active instance to at least one other instance of the control software configured to take over control of the industrial process in case of a failure and / or malfunction of the active instance.

Description

TECHNICAL FIELD

[0001] The present invention relates to a software-driven control of an industrial process that remains reliable even if an instance of the control software fails or malfunctions. BACKGROUND

[0002] In an industrial automation system, a control application maintains an internal state from input signals (e.g. sensor readings from a chemical process) and a system clock. In each computation cycle (e.g. every 100 ms), the control program processes its current input signals and its current internal state (i.e. values of internal variables) to compute output signals. For non-trivial control applications, the internal state can be the result of a long approximation process and will not simply depend on the current input signals. A typical example is the use of a PID controller that continuously applies corrections to the control function. The PID comprises a computation of an integral value that depends on all past input signal values since initialization and the ongoing clock. In case of a reset of the control program (e.g. a restart of the control device due to a power failure), it is crucial to maintain such an internal state to enable a seamless continuation of the process control.

[0003] While this problem has been solved in simple scenarios with dedicated hardware controllers, it is challenging in containerized and clustered environments without such hardware. In a setup with hardware controllers, redundant controllers are either synchronized via a dedicated high-bandwidth low-latency communication channel or the state is persisted to non-volatile storage (e.g. in each execution cycle). In a containerized clustered environment, a full synchronization between multiple control application instances is challenging due to shared network resources and clock synchronization. Moreover, a tight coupling of a single control application instance is less flexible and hinders the integration of failover mechanisms with container orchestration solutions. Therefore, a setup with more than two redundant control application instances is difficult to realize due to the required communication overhead. SUMMARY

[0004] It is therefore an object of the present invention to facilitate the deployment of instances of control software in a containerized clustered environment while still meeting given requirements on reliability and / or availability.

[0005] This object is achieved by a method for controlling an industrial process according to the first independent claim and by a method for configuring an industrial control system according to the second independent claim. Further advantageous embodiments are detailed in the respective dependent claims.

[0006] The invention provides a method for controlling an industrial process. In the course of the method, an active instance of control software is provided with process data of the industrial process as input. The process data comprises a set of process variables of the industrial process and / or plant variables of an industrial plant executing the process. For example, the process data can comprise any type of sensor readings from the industrial plant and / or any type of operating states of equipment in the industrial plant, such as on / off switch states of the plant, open / close states of valves, and / or set points of low-level controllers in the plant. For example, such low-level controllers can control equipment in the plant to maintain a temperature, a pressure, a fill level, or some other process parameter at a desired set point value.

[0007] Based on the input provided to it, the active instance of control software updates its internal state. This internal state is characterized by a set of internal variables. That is, starting from an initialization execution environment with a fresh instance of the control software, the internal state of the active instance can be reproduced given these internal variables. In particular, the internal variables can comprise calculation results or other values obtained on the basis of the input. As mentioned before, the internal state can also comprise, for example, an integral value maintained within a PID control loop.

[0008] Based on the updated internal state, the active instance determines an output signal and outputs this output signal to at least one piece of equipment configured to alter the state and / or behavior of the industrial process. For example, the output signal can be delivered to one or more actuators that directly act on the physical behavior of the industrial process. However, for example, the output signal can also comprise a set point to be delivered to a low-level controller.

[0009] A prioritized subset of the internal variables characterizing the state of the active instance is provided to at least one other instance of the control software, which is configured to take over control of the industrial process in case of a failure and / or malfunction of the active instance and / or in a maintenance scenario. The other instance can comprise the same control software as the active instance, but can also comprise a different control software. For example, software diversity between the active instance and the other instance can be intentionally employed to prevent a single problem from simultaneously bringing down all available instances. In a maintenance scenario, the primary instance can be intentionally "failed" because it is forcefully killed due to a normal maintenance task (of the cluster hardware). The technicians of the data center, who are not control experts, need to trust that the system will continue to run and repair itself after the hardware node is removed. Another example of a maintenance scenario is an operating system upgrade on a node of the cluster.

[0010] As mentioned above, for many industrial processes, the switchover of control from the active instance to other instances of the control software during failover can be very time critical. For some processes, the switchover needs to be completed within 100 milliseconds or less. For example, many chemical process recipes specify very strict ranges in which process parameters are to be maintained. If any of these process parameters goes outside its range, the recipe is violated and the entire batch being processed can no longer be sold as a product conforming to the given specifications. Other adverse consequences can arise if the product solidifies, decomposes, ignites or explodes or the pressure or temperature is too high, causing damage to the plant. In particular, while a sudden increase in temperature is slowed down by the heat capacity in the system, pressure can increase very rapidly.

[0011] If the other instance is to seamlessly take over control of the process in a "hot failover", it needs to constantly update the state of the active instance. However, in a containerized environment or other virtualized environment, the communication bandwidth for these updates is limited. A straightforward solution might be to require the constant replication of the complete state of the active instance to the other instance and to provide the required bandwidth anyway. However, the present invention proposes to replicate only a prioritized subset of the internal variables. Then, the output of the other failover instance can deviate from the output that the active instance has already produced to some extent. Thus, some accuracy of the control behavior can be sacrificed. But in return for this sacrifice, the time required for the failover is greatly reduced. In particular, if the industrial process can develop very rapidly, it is more important to act earlier than to act perfectly. This situation is somewhat analogous to lossy compression of images. For example, using JPEG2000, a color image of 640x480 pixels can be compressed to 1 KB to 2 KB. The compressed version of this small size can be transmitted very quickly over a network, but still contains all the information of a scene that requires immediate action.

[0012] Moreover, the requirements of the instance of the control software on the execution environment are also relaxed, greatly facilitating deployment in a containerized environment or other virtualized environment. In comparison to the deployment of a hardware controller, virtualized deployment provides higher flexibility and lower operating costs, because the hardware is used in a potentially shared manner. But this shared use limits the communication bandwidth and processing resources. By transmitting only a prioritized subset of the internal variables, time can be saved, so that the virtualized environment becomes fast enough to accomplish the task at hand. Even if some dedicated hardware such as a processor or its cores is assigned to the instance of the control software, the mentioned flexibility is advantageous, because these assignments can be dynamically adjusted. For example, in case of maintenance or interruption, resources can be seamlessly reassigned.

[0013] The determination of the prioritized subset can be determined in an automated fashion. For example, a computer analysis can be performed on the source code and / or the object code of the control software used as the active instance. The analysis can be performed as a preparatory step before starting the control software. But depending on the available computing power, the analysis can also be performed temporarily while the control software is running. The analysis can comprise a static analysis, but alternatively or in combination, it can also comprise executing the object code in a sandbox and monitoring its behavior. The analysis derives which internal variables are more important than others for the control task being performed. Herein, one potential significance can be derived from the influence of a particular internal variable or set of variables on driving decisions.

[0014] In particular, the code analysis can comprise evaluating the names of the accessed variables, the dependencies between the variables (e.g. one variable can just be the result of a simple calculation based on other more important variables) and / or the tags of the accessed variables with a "reserved" level. According to IEC 61131-3 and related standards, it is common practice in the development of control applications to tag internal variables that have been considered important when designing the control software. Thus, the effort already put into coding the control software can be reused for determining the most important internal variables.

[0015] However, even if the control software is a "black box" without meaningful variable names or variable tags, a more in-depth automated analysis can reveal the most important internal variables.

[0016] For example, the frequency of accessing and / or using a variable in a calculation can be evaluated. If the value of a variable is needed more frequently, it can be more important than a variable that is only accessed once, or even not at all after being written. For example, a variable can have been replaced by another variable during the development process, but the "orphaned" old variable is still in the calculation.

[0017] Furthermore, the static dependencies between variables can be evaluated. In particular, the number of variables that depend on any particular variable can be evaluated. A variable that many other variables depend on is likely to be more important.

[0018] Still further, the quantitative dependency of the output of the control software on one or more particular variables can be evaluated. This quantitative dependency is somewhat similar to the "partial derivative" of the output of the control software with respect to the variable in question. If certain variables take up a lot of space, but if they are ignored, the output of the control application changes little, then it can be beneficial to exclude them.

[0019] In addition to the tagging of the internal variables that enter the control software, the tagging of the process or plant variables that certain internal variables depend on can also be used to evaluate the importance of these internal variables.

[0020] In an advantageous embodiment, the prioritized subset of internal variables comprises:

[0021] • at least one internal variable indicative of and / or dependent on at least one pressure in the industrial plant in which the process is executed; and / or

[0022] • the current state of all sequential function charts (SFCs) used in the active instance.

[0023] In particular, as discussed previously, pressures are among the quantities that can change very fast during chemical processes and during processes requiring vacuum. For example, if a small leak exists in an ultra-high vacuum chamber, the pressure drops from 10 -9 millibar to 10 -6 millibar, which is a 1000-fold increase in pressure.

[0024] SFCs describe, for example, the sequential steps of a process and actions to be triggered on transitions within and between steps. To make the switching control seamless, internal states of the SFCs can be required. For example, these internal states can include information about the current active step and timers specific to the SFC.

[0025] In a particularly advantageous embodiment, different prioritized subsets of internal variables are provided to different other instances of the control software, which are configured to take over control of the industrial process at different time periods from the active instance failing and / or malfunctioning.

[0026] For example, there can be a “hot failover” instance of the control software, which is configured to take over control immediately when the active instance becomes unavailable. To provide a higher level of redundancy, there can be another “warm failover” instance that is updated with less information from the active instance during normal operation and a “cold failover” instance that is updated with even less information. This reduces the communication bandwidth consumption during normal operation. Furthermore, the “warm failover” instance and the “cold failover” instance can run with lower priority and even be in a hibernation state during normal operation, thereby reducing the overhead of processing resources for increased redundancy.

[0027] If the active instance becomes unavailable and the “hot failover” instance takes over control, then the “warm failover” instance can receive greater state updates and more processing resources so that in the event that this “hot failover” instance fails next, the “warm failover” instance can immediately take over control from the “hot failover” instance.

[0028] The "cold failover" instance can even be "cold" in the sense that it is only started in case of a failure of the active instance, in order to restore the degree of redundancy to the desired level even if the active instance is no longer available.

[0029] In another particularly advantageous embodiment, at least one other instance of the control software is configured to control the industrial process in a degraded mode of operation compared to the normal mode of operation performed by the active instance. In this way, less resource overhead has to be provided for the other instance. For example, the degraded operation can mean that the industrial process is running slower, and thus has less throughput. The smaller throughput incurs a cost. However, if this happens, say, once a month, for a few minutes, this cost can be lower than the cost of having to provide the virtualization environment with processing power to cope with a full copy of the active instance 24 / 7.

[0030] The invention also provides a method for configuring an industrial control system to control an industrial process according to the method described above. The method is performed by at least one failover manager or other management entity.

[0031] The method first maps a given set of requirements on the behavior of the control system in case of a failure and / or malfunction of the active instance of the control software to a configuration of:

[0032] • one or more other instances of the control software, which are to take over the control of the industrial process after the failure and / or malfunction; and

[0033] • a respective prioritized subset of internal variables of the active instance to be provided to the other instances.

[0034] The at least one or more instances are created in respective virtualized execution environments on at least one virtualization platform according to the configuration thus determined. The transfer of the respective prioritized subset of internal variables from the active instance to the respective other instances of the control software is orchestrated and / or performed by the failover manager or other management entity. The active instance can also be part of the configuration to be created. That is, the complete control system, including the active instance and all other backup instances, can be created by the failover manager and / or other management entity. However, the method can also be performed without central coordination, i.e. in a peer-to-peer or other decentralized manner without a failover manager.

[0035] If a management entity is used, the plant operator only has to specify the requirements without having to know any details of the virtualization platform, its capabilities and current usage state. All these details are taken care of by the failover manager or other management entity. In particular, any changes to the virtualization environment can be seamlessly accommodated. For example, the entire virtualization environment can be moved to another hardware somewhere else in the local area with different network latencies. In this case, the failover manager can automatically recalculate the configuration of instances that need to be deployed in order to meet the given requirements.

[0036] In particular, the mapping to the configuration can be based at least in part on one or more of the following:

[0037] • a required level of reliability of the control of the process;

[0038] • a requirement for a longest time period after a failure and / or malfunction during which at least one other instance has to take over the control of the industrial process;

[0039] • a time required for creating and / or initializing at least one other instance on the virtualization platform;

[0040] • an availability of resources for creating other instances on the virtualization platform;

[0041] • an available communication bandwidth between the active instance and at least one other instance; and

[0042] • a state and / or phase of the industrial process.

[0043] That is, the failover manager and / or other management entity can act as a "broker" that combines the requirements from the plant operator's domain with the given technical situation on the virtualization platform's side. The plant operator does not even have to know any details about the virtualization platform that is being used. Thus, in the example where the virtualization environment is moved to different hardware or is otherwise changed, no gas input from the plant operator is required to determine an updated configuration.

[0044] By making the mapping dependent on the state of the industrial process, the failover manager and / or other management entity can take into account that the requirements for failover can be very different at different phases of the industrial process. For example, in a phase of a chemical process where there is an explosive intermediate product, maximum redundancy with numerous other failover instances can be required. But in a last phase where the finished product that is no longer explosive is just packaged, only one failover instance can be required.

[0045] In particular, orchestrating the transfer of the prioritized subset of internal variables to the other instances can comprise establishing at least one communication channel between the active instance and the at least one other instance. For example, in a containerized environment, the control software instances in different containers initially do not know any information about each other. But the superior failover manager and / or other management entity can connect them.

[0046] In a particularly advantageous embodiment, the communication channel comprises a publish / subscribe one-to-many communication and / or a named pipe. If more than one other instance of the control software is involved, the one-to-many communication saves communication bandwidth. Moreover, the publish / subscribe nature of the communication requires changes only on the side of the receiving other instances if a new such instance is added or if the role of such an instance changes (e.g. from "warm failover" to "hot failover").

[0047] In another advantageous embodiment, performing the transfer of the prioritized subset of internal variables to the other instances can comprise buffering the prioritized subset in the failover manager and / or other management entity. For example, a fixed-size ring buffer can hold a rolling history of the latest states of the active instance. If the time requirements for failover are not too high, it can not be necessary to have the other failover instances running all the time. Instead, the failover manager and / or other management entity can start another instance when the active instance fails and a failover has to be performed. Then, the newly started other instance can be provided with the state from the ring buffer.

[0048] The methods can be fully or partially computer-implemented. Therefore, the present application also provides one or more computer programs with machine-readable instructions which, when executed on one or more computers, cause the one or more computers to perform the methods described above. In particular, the virtualization platform and the one or more hardware controllers can be regarded as computers.

[0049] The present application also provides one or more non-transitory storage media and / or download products with one or more computer programs. A download product is a product which can be sold in an online store to be implemented immediately by download. The present application also provides one or more computers with one or more computer programs and / or with one or more non-transitory machine-readable storage media and / or download products.

[0050] In particular, the present application provides at least the following advantages:

[0051] • Automatic selection and externalization of application state based on criticality. Criticality is determined based on existing annotations (e.g. retention) or automatically derived using static code analysis and analysis of variable change behavior during runtime.

[0052] • Specific failover instance deployments are derived by the failover manager (a software agent) based on high level availability requirements (e.g. as per service level agreements (SLAs)) and orchestrates these deployments. Given a desired application availability level, the failover manager computes how many redundant standby instances should be active at any given time and how they are distributed across hardware nodes or (e.g. availability zones). These availability SLAs can be provided as a percentage of uptime (e.g. 99.999%) or as maximum tolerable downtime minutes / seconds per year / month / week.

[0053] • The failover manager automatically hatches application failover instances and assigns failover roles. The failover agent as part of the application deployment monitors for release rate violations and autonomously decides to take over the active executive role.

[0054] • In one embodiment, the migration manager instance itself also subscribes to state updates as per its criticality (e.g. all instances subscribe to cold preservation, few instances subscribe to preservation, only one instance subscribes to normal variables) and maintains a temporary copy of the preservation state to hatch new application instances on demand. This locally maintained state copy is used to hatch new application instances that can take over execution in the absence of a failover application.

[0055] • The mechanisms described in this invention can be used for migration use cases where the application needs to be migrated across hosts. In this case, a failover instance is hatched and synchronized before the active instance is shut down to force a failover. Depending on the migration time requirements, a full or partial transfer can be done.

[0056] The invention enables customers to run control applications in a container orchestration environment with high availability requirements. To do this, no changes are required to the way the control application is designed. Even legacy / brownfield applications can be migrated to a container orchestration environment without changing the control logic, in case the preservation markers have already been used within the control logic and tested. All the required failover mechanisms are implemented from outside the application. Depending on the expressed failover requirements, the failover semantics of the application in the new environment are well defined and similar to the expected behavior of a hardware controller warm / cold reboot.

[0057] Key ideas include:

[0058] • The idea of utilizing control code preservation level annotations and their transfer priority based on the criticality of the failover mechanism in the container orchestration setup to implement a flexible and highly configurable failover mechanism for control applications.

[0059] • The mechanism of externalizing state based on preservation level annotation semantics to meet user defined high availability and redundancy / failover requirements.

[0060] • Mechanism to translate high availability and redundancy / failover requirements of advanced user specifications into concrete failover instance deployments that are robust against different types and severity of failure scenarios.

[0061] • Monitoring-based failover mechanism that enables failover instances to autonomously or jointly detect failed application instances in the failover cluster to take over control application execution.

[0062] In the following, exemplary embodiments are described without being intended to limit the scope of the invention.

[0063] For the actual delivery of externalized states, an access layer, network layer or application layer publish / subscribe mechanism or application layer subscription is used. For this, for example, the OPC UA mechanism can be used, but the invention is not limited to this, as it is also applicable to other delivery mechanisms. In this context, the concept of efficiency refers to the potential use of a publish / subscribe mechanism to reduce network load (in case of states that are replicated to more than one failover instance) and to transmit only changed states. Herein, incremental delivery mechanisms such as the keyframe / incremental frame mechanism and the subscription semantics of OPC UA can be used.

[0064] Deadbands for individual variables can be defined to further reduce the rate of state changes that need to be transmitted. This can be achieved by introducing additional annotations that the process engineer can add to the variable definition or that are automatically derived from deadbands configured for input signals and automatically applied to variables that depend on these signals.

[0065] The invention makes use of the fact that control application engineers manually annotate internal variables when designing control applications. This is usually done by marking the variables with specific keywords such as "retentive" in case of IEC 61131-3 application code. The semantics of this annotation is that the value of the variable is persisted (e.g. every processing cycle) to non-volatile storage to continue to exist after a restart of the application or the complete hardware controller. In some commercial control platforms, additional annotation levels such as "cold retentive" are used to distinguish different degrees of persistence requirements and provide the engineer with even finer-grained control over the persistence overhead imposed by the application on the control platform. The invention is not specific to these retentive levels but also applicable to other retentive levels and definitions. It should be noted that control application engineers reuse existing well-tested control logic libraries as part of their own application. For these libraries, the retentive annotation either already exists and has been tested against real-time reconfiguration scenarios or only needs to be exported once to be compatible with the envisioned retentive level-based state transfer mechanism.

[0066] In addition to utilizing the previously described existing manual annotations, the present invention also contemplates using static code analysis and runtime monitoring of the changing behavior of variables to automatically derive state annotations. In this way, variables that can be considered relevant to the failover level can be based on the dependencies of other variables and the control application logic's dependence on the variable itself. Variables with many static dependencies mean that their state is more important for seamless failover than other variables that have no dependencies. This avoids the possibility of the variable being affected in the case of a lost transmission. Variables that are considered critical can be deemed relevant and transferred along with manually annotated variables (e.g., preserved variables). Likewise, the dynamic changing behavior of variables can lead to a decision to include the variable in state transfer. Because the overhead of a transfer using an incremental state transfer mechanism is small, additional variables that change slowly over time can be considered for transfer. In contrast, variables that change quickly over time can also be considered for transfer depending on the number of dependencies on them. The failover agent local to the application instance monitors the variables to dynamically determine and update the importance of the variables for different failover levels. However, because this dynamic analysis can draw incorrect conclusions due to non-obvious dependencies and actual criticality, the mechanism can be used without automatically making changes but with the engineer or operator needing to confirm any patterns of monitored state variables.

[0067] For scenarios where the control application engineer would annotate too many variables as preserved variables, the above mechanism can also help to automatically reduce the number of transferred variables to a smaller subset, reducing the state replication effort.

[0068] The criticality rating of variables is used to unlock the ability to adapt state replication to the redundancy requirements of the control application. This makes the present invention directly applicable to brownfield settings and applications in that any redesign of the control code in an expensive way is avoided, so that a well-tested and understood persistent setup of a highly available control application can be used to run the same application in a containerized environment without changes. The additional dynamic criticality rating based on code and change analysis helps to fine-tune replication and identify relevant state that the control engineer did not annotate correctly.

[0069] In the case of failover, special attention is needed to the relevant state of the sequence function chart (SFC). The SFC describes, for example, the sequential steps of a process and the actions to be triggered on the transitions within and between the steps. Step transitions can be triggered based on timers and Boolean conditions defined on variables and process signals.

[0070] To support seamless continuation of the control program after failover, the current state must also be transferred to all SFCs of the control program. This includes SFC-specific timers and information about the current active step. Therefore, changes to steps and SFC timers are transferred by default using reserved variables.

[0071] This semantics can differ from the "normal" reserved behavior of the application, and the engineer can typically explicitly choose whether to keep the SFC state or move to the initial state during a CPU restart.

[0072] During runtime, the failover manager interacts with the container orchestration environment to detect that an update of the application deployment is required due to a host becoming unavailable. It therefore requests the deployment of failover control application instances. The requirements (e.g. triple redundancy) influence the deployment of the instances, e.g. to different hosts, clusters or availability zones. Each instance is equipped with a sidecar container (a so-called "failover agent") that receives the configuration and the assigned role from the failover manager. Depending on the criticality, the application can have different types of failover instances. A hot failover instance subscribes to all externalized states (cold reserve, reserve, others) of the active application instance as well as to the input signals of the application. Depending on the timing needs, a failover instance can maintain a dormant mode (not actively executing control code) or an active mode (constantly executing control code based on recent state updates). By default, all types of failover instances are started as dormant instances. The next section describes an alternative reaction mode where the cold reserve instance is only started upon occurrence of a failure.

[0073] As an optimization, an actively executing hot failover instance can monitor the difference between local computation results and remote computation results it receives via subscription. As soon as both are synchronized, the failover instance can gradually reduce its subscriptions to a minimum to avoid unnecessary state transfers. At least, it subscribes to the input of the application and the output signals of the main application instance to monitor its availability and synchronization state. In case of a difference, the subscriptions are established again to achieve synchronization. From this description, it is clear that an actively executing hot failover instance trades a small amount of transferred state for more required computational resources spent on executing the control application.

[0074] The failover instances do not write output signals, but monitor the output of the active instance. The hot failover instance can immediately take over the role of the active instance and also needs to constantly process all state updates or actively perform the application itself (see previous paragraph on sleep and active mode). The warm failover instance only subscribes to the retention level state updates, while the cold failover instance only subscribes to the cold retention state updates (if applicable for the control platform). Thus, the warm and cold failover instances can receive a smaller number of state updates and also can take over execution in a non-seamless way. However, the well-defined retention semantics lead to failover behavior that in the worst case resembles the behavior of a traditionally deployed control application in case of a hardware controller restart. Thus, the failover can not be seamless, but the behavior is well-defined, as expected by the control engineer and considered acceptable for the particular industrial process.

[0075] It should be noted that the overall approach of replicating the state to potentially many failover instances can imply a significant network load. Thus, it can be necessary to limit the number of failover instances and roles depending on network and host capabilities. However, for state-of-the-art computing cluster setups, this is not seen as a problem. Here, network bandwidth is traded for the ability to implement fast failover and the ability to flexibly adapt to fast heterogeneous and fast changing environments. A potential optimization can be to retain multiple failover instances, but only replicate the state to a fixed number of instances. In case of a failure, then the instance with the active synchronization takes over and the state replication is extended to additional instances.

[0076] The actual decision to take over execution of the active application instance is made by each individual failover agent itself based on their role and the active monitoring of the received state updates and optionally the monitored output signal updates of the active instance. The subscription is configured to provide regular heartbeat messages in case no state update occurs. This way, a failed application can be detected in a fast way based on a preconfigured number of missed heartbeats. The failover instances can also monitor each other and use a consensus mechanism to determine a leader to define a deterministic failover sequence in case more than one instance is available per role. The failover manager monitors the availability of the application instances via the container orchestration framework, which is responsible for automatically hatching new instances in case of deployment changes, e.g. in case of instance failures or container orchestration events.

[0077] In case of an application instance failure, the failover manager ensures that the loss of the output signal does not lead to any process interruption. To this end, the failover manager can temporarily send a keep-alive message to the enforcer upon start-up and configuration of the failover instance. However, in case of a non-seamless failover due to a too long start-up time, it can be necessary to reset the enforcer while starting the application anew (e.g. in case of a cold failover instance only or in case of a start-up is required). In this case, the enforcer can first return to the safe operating condition while starting the application similar to a controller reset. In this case, the SFC can also be reset to implement a safe restart procedure.

[0078] In certain setups, it is more desirable to implement the monitoring mechanism and the triggering of the failover as part of the failover manager or orchestration framework. This can be advantageous in case of a high availability setup of many standby nodes and the failover manager itself.

[0079] In one variant, the migration manager itself subscribes to the state updates and monitors the availability of the primary application instance. It caches the received states in a ring buffer that always contains the complete set of internal variables of the last execution cycle. As soon as the complete set of the next cycle is received, the old state is overwritten to limit the amount of state to be managed and maintained. Furthermore, in a cluster setup, the management cluster is also redundant. Therefore, multiple migration manager instances subscribe to the state updates. However, this can happen in a way that the redundancy needs are dynamically adapted. One instance can subscribe to the complete set of internal variables (all retained level variables and automatically retrieved variables), while a second instance serves as a backup that only maintains the retained variables and configuration updates. This way, the second instance can only start a cold failover instance anew.

[0080] When the migration manager detects a failure of the primary application instance, it immediately starts (switches on) a new failover instance and injects the last known state into the instance. To make the failover time as short as possible, the application instance container image is adapted to enable a fast start-up and the container image is pre-distributed to the potential execution nodes and decompressed, including all its dependencies, to enable a fast deployment and start-up. To this end, the previously proposed container orchestration environment extension is used. Therefore, the failover instance can be started up quickly, which mainly depends on the time it takes to transfer and inject the state into the failover instance. This failover mode is expected to be applicable for applications with a more relaxed cycle time (e.g. > 100 ms). Depending on the size of the state and the network latency, a faster failover can also be feasible. This reactive failover implementation can also be used to complement the previously described proactive failover mechanism.

[0081] The described mechanism can also be used for application migration use cases. In this case, a failover instance is hatched and execution takeover is triggered by simply triggering the shutdown / deletion of the active instance via the container orchestration environment. Here, the failover manager can wait for the failover instance to signal the first completion (according to the desired failover level) state transfer occurrence and issue the deletion of the related execution unit (e.g. the pod running the control application and its failover proxy sidecar). Depending on the migration time requirements, a full or partial state transfer can be done. BRIEF DESCRIPTION OF DRAWINGS

[0082] In the following, the application is described using the attached drawings, which are not intended to limit the scope of the application. The drawings show:

[0083] Figure 1 is an exemplary embodiment of a method 100 for controlling an industrial process 1 ;

[0084] Figure 2 is an exemplary embodiment of a method 200 for configuring an industrial control system to control a process 1 according to the method 100;

[0085] Figure 3 is a first exemplary configuration 7 generated by the method 200;

[0086] Figure 4 is a second exemplary configuration 7 generated by the method 200. DETAILED DESCRIPTION

[0087] Figure 1 is a schematic flow chart of an exemplary embodiment of a method 100 for controlling an industrial process 1. In step 110, an active instance 2 of a control software is provided with process data 1a of a process 1 as input. In step 120, based on the input, the active instance 2 updates its internal state 2a. In step 130, based on the updated internal state 130, an output signal 2b is generated. In step 140, the output signal 2b is output to at least one piece of equipment 1b configured to alter the state and / or behavior of the industrial process 1. As discussed earlier, examples of equipment 1b include actuators that directly act on the process 1 and low-level controllers that keep certain process parameters at setpoint values.

[0088] Figure 1Two exemplary ways of obtaining a prioritized subset 2a*, 2a**, 2a*** of the internal variables characterizing the state 2a of the activity instance are shown. In step 150, the prioritized subset 2a*, 2a**, 2a*** is determined by an analysis of the source code and / or the object code of the control software used as the activity instance 2. In step 160, the prioritized subset 2a*, 2a**, 2a*** is determined by an analysis of the marking of the internal variables and / or the marking of the procedures or plant variables with a reservation level on which these internal variables depend.

[0089] Irrespective of how the prioritized subset 2a*, 2a**, 2a*** is determined, in step 170, the prioritized subset 2a*, 2a**, 2a*** is provided to at least one other instance 3, 4, 5 of the control software. These instances can contain the same software as used in the activity instance, but here it is possible to deliberately introduce software diversity, for example, in order to further increase the reliability. According to block 171, different prioritized subsets 2a*, 2a**, 2a*** of the internal variables can be provided to different other instances 3, 4, 5 of the control software.

[0090] Figure 2 is a schematic flow chart of an exemplary embodiment of a method 200 for controlling an industrial process 1 according to the method 100 described above.

[0091] In step 210, a given set of requirements for the behavior of the control system that the system should exhibit in the event of a failure and / or malfunction of the activity instance 2 is mapped to a configuration 7. The configuration 7 comprises one or more other instances 3, 4, 5 of the control software that are to take over the control of the industrial process 1 after the failure and / or malfunction; and a corresponding prioritized subset 2a*, 2a**, 2a*** of the internal variables of the activity instance 2, which is provided to the other instances 3, 4, 5.

[0092] In step 220, the one or more other instances 3, 4, 5 are created in a corresponding virtualized execution environment on at least one virtualized platform according to the configuration 7 thus determined. In Figure 2 In the example shown, the activity instance 2 is also created in step 220.

[0093] In step 230, a transfer of the respective prioritized subsets 2a*, 2a**, 2a*** of internal variables from the active instance 2 to respective other instances 3, 4, 5 of the control software is orchestrated and / or performed. According to block 231, the orchestration can comprise establishing at least one communication channel between the active instance 2 and at least one other instance 3, 4, 5. According to block 232, performing the transfer of the prioritized subsets 2a*, 2a**, 2a*** of internal variables to the other instances 3, 4, 5 can comprise caching the prioritized subsets in the failover manager and / or other management entity 10, 11.

[0094] Figure 3 A first exemplary embodiment of a configuration 7, which can be a result of the method 200, is shown. In the shown example, the configuration 7 comprises: Figure 3 In the shown example, a management cluster 11 serves as a management entity for the instances 2, 3, 4, 5 of the control software and for the failover behavior. The management cluster 11 comprises a container orchestration module 11a and a failover manager 11b. The container orchestration module 11a determines the configuration 7. The failover manager 11b initiates and configures the instances 2, 3, 4, 5 according to the configuration 7 and confirms to the container orchestration module 11a that the configuration 7 has been implemented.

[0095] The active instance 2 for normal operation of the industrial process 1 is deployed in a container 12, which comprises a virtualized execution environment 12a of the active instance 2 and a failover proxy sidecar 12b. The failover proxy sidecar 12b extracts from the internal state 2a of the active instance 2:

[0096] • a first prioritized subset 2a* of internal variables marked as “cold reserve”,

[0097] • a second prioritized subset 2a** of internal variables marked as “reserve”, and

[0098] • a third prioritized subset 2a*** of internal variables marked as “other”.

[0099] All different subsets 2a*, 2a**, 2a*** of internal variables are published in a publish / subscribe system on separate channels that can be subscribed to by other instances 3, 4, 5 of the control software.

[0100] In the shown example, the configuration 7 further comprises: Figure 3 In the shown example, the configuration 7 further comprises:

[0101] • a hot failover instance 3 of the control software deployed in a container 13,

[0102] • a warm failover instance 4 of the control software deployed in a container 14, and

[0103] • a cold failover instance 5 of the control software deployed in container 15.

[0104] Like container 12, each container 13, 14, 15 comprises its own virtualized execution environment 13a, 14a, 15a for the respective instance 3, 4, 5, and its own failover proxy sidecar 13b, 14b, 15b.

[0105] The failover proxy sidecar 13b subscribes to all subsets 2a*, 2a**, 2a*** of internal variables and uses these subsets to update the internal state 3a of instance 3.

[0106] The failover proxy sidecar 14b only subscribes to the "cold reserve" subset 2a* and the "reserve" subset 2a** of internal variables and uses these subsets to update the internal state 4a of instance 4. It does not subscribe to the "other" subset 2a*** of internal variables.

[0107] The failover proxy sidecar 15b only subscribes to the "cold reserve" subset 2a* and uses this subset to update the internal state 5a of instance 5.

[0108] The failover roles of all failover proxy sidecars 12b, 13b, 14b and 15b are configured by a failover manager 1 lb in the management cluster 11.

[0109] Figure 4 A second exemplary embodiment of a configuration 7, which can be the result of the method 200, is shown. Here, two redundancy management entities 10, 11 are used. Each management entity 10, 11 comprises a container orchestration module 10a, 11a and a migration manager 10b, 1 lb.

[0110] Like in the first exemplary embodiment of the configuration 6, Figure 3 The active instance 2 of the control software is deployed in a first container 12 with a virtualized execution environment 12a and a failover proxy sidecar 12b. From the internal state 2a of the active instance 2, the failover proxy 12b again extracts the "cold reserve" subset 2a*, the "reserve" subset 2a* and the "other" subset 2a** of internal variables. All these subsets 2a*, 2a** and 2a*** are again published on different channels in the publish / subscribe system.

[0111] The migration manager 10b in the management entity 10 subscribes only to the "cold-preserved" subset 2a* and caches the records of this subset 2a* in a ring buffer 10c of fixed size. The migration manager 1 lb in the management entity 11 subscribes to the "cold-preserved" subset 2a* and the "preserved" subset 2a**. The received records are cached in a ring buffer 11c. When the ring buffer 10c, 11c is full and a next record enters, the respective oldest record in the ring buffer 10c, 11c is discarded to make room for the new record.

[0112] In Figure 4 In the case shown, the container orchestration module 11a in the management entity 11 instructs the migration manager 1 lb to failover from the active instance 2 to the new instance 3 to be created, so that the active instance 2 can be migrated to other hardware (or any other maintenance work that can be performed on it). The migration manager 1 lb causes the instance 3 to start in a container 13 with a virtualized execution environment 13a and a failover proxy sidecar 13b. The failover proxy sidecar 13b fetches the records of the "cold-preserved" subset 2a* and the "preserved" subset 2a** from the ring buffer 11c of the migration manager 1 lb and uses these records to update the internal state 3a of the newly created instance 3, so that it can take over the work of the active instance 2. Then, the migration manager 1 lb confirms the successful failover to the container orchestration module 11a.

[0113] List of reference signs

[0114] 1 industrial process

[0115] 1a process data of the process 1

[0116] 1b device, acting on the process 1

[0117] 2 active instance of the control software

[0118] 2a internal state of the active instance 2

[0119] 2a* "cold-preserved" subset of internal variables of the active instance 2

[0120] 2a** "preserved" subset of internal variables of the active instance 2

[0121] 2a*** "other" subset of internal variables of the active instance 2

[0122] 2b output signal

[0123] 3 hot failover instance of the control software

[0124] 3a internal state of the hot failover instance 3

[0125] 4 Warm failover instance of the control software

[0126] 4a Internal state of warm failover instance 4

[0127] 5 Cold failover instance of the control software

[0128] 5a Internal state of cold failover instance 5

[0129] 6 Behavioral requirements of the control system

[0130] 7 Configuration of instances 2, 3, 4, 5 and subsets 2a*, 2a**, 2a***

[0131] 10, 11 Management entity

[0132] 10a, 11a Container orchestration module in management entity 10, 11

[0133] 10b, 11b Migration manager / failover manager in management entity 10, 11

[0134] 10c, 11c State ring buffer in management entity 10, 11

[0135] 12 Containers of active instance 2

[0136] 12a Virtualized execution environment in container 12

[0137] 12b Failover proxy sidecar in container 12

[0138] 13 Containers of hot failover instance 3

[0139] 13a Virtualized execution environment in container 13

[0140] 13b Failover proxy sidecar in container 13

[0141] 14 Containers of warm failover instance 4

[0142] 14a Virtualized execution environment in container 14

[0143] 14b Failover proxy sidecar in container 14

[0144] 15 Containers of cold failover instance 5

[0145] 15a Virtualized execution environment in container 15

[0146] 15b Failover proxy sidecar in container 15

[0147] 100 Method for controlling an industrial process 1

[0148] 110 providing process data 1a to active instance 2

[0149] 120 updating internal state of active instance 2

[0150] 130 determining output signal 2b

[0151] 140 outputting output signal 2b to device 1b

[0152] 150 code parsing of control software of active instance 2

[0153] 160 analysis of labels of variables

[0154] 170 providing subsets 2a*, 2a**, 2a*** to other instances 3, 4, 5

[0155] 171 using different subsets 2a*, 2a**, 2a*** for different instances 3, 4, 5

[0156] 200 method for configuring an industrial control system

[0157] 210 mapping requirements 6 to configuration 7

[0158] 220 creating other instances 3, 4, 5

[0159] 230 orchestrating / performing transfer of subsets 2a*, 2a**, 2a***

[0160] 231 establishing a communication channel

[0161] 232 caching subsets 2a*, 2a**, 2a***

Claims

1. A method (100) for controlling an industrial process (1), comprising the steps of: • providing (110) process data (1a) of the industrial process (1) as input to an active instance (2) of a control software, wherein the process data (1a) comprises a set of process variables of the industrial process (1) and / or plant variables of an industrial plant executing such process (1); • updating (120) an internal state (2a) of the active instance (2) of the control software based on such input, characterized by a set of internal variables of such active instance (2); • determining (130) an output signal (2b) by the active instance (2) based on such updated internal state (2a), and outputting (140) such output signal (2b) to at least one device (1b) configured to alter a state and / or behavior of the industrial process (1); and • providing (170) a prioritized subset (2a , 2a , 2a ) of the internal variables characterizing the state (2a) of the active instance (2) to at least one other instance (3, 4, 5) of the control software, the at least one other instance (3, 4, 5) of the control software being configured to take over the control of the industrial process (1) in case of a failure and / or malfunction of the active instance (2), wherein the prioritized subset (2a , 2a , 2a ) of the internal variables comprises internal variables that are more important for the performed control task than other internal variables.

2. The method (100) of claim 1, further comprising: determining (150) the prioritized subset (2a) of the internal variables by analyzing source code and / or object code of the control software used as the active instance (2) ,2a ,2a ).

3. The method (100) of claim 2, wherein the code analysis of the control software comprises an evaluation of one or more of: • names of accessed variables; • tags of accessed variables with a level of retention; • frequency of variables accessed and / or used in computations; • static dependencies between variables; • number of variables co-dependent on any one particular variable; and • quantitative dependency of the output of the control software on one or more particular variables.

4. The method (100) according to any one of claims 1 to 3, further comprising: determining (160) the prioritized subset of the internal variables (2a ,2a ,2a ).

5. The method (100) according to any one of claims 1 to 3, wherein the prioritized subset (2a ,2a ,2a ) of the internal variables comprises: • at least one internal variable indicative of and / or dependent on at least one pressure in the industrial plant executing the process (1); and / or • current state of all sequential function charts, SFCs, used in the active instance.

6. The method (100) according to any one of claims 1 to 3, wherein different subsets (2a ,2a ,2a ) of the internal variables of different priority are provided (171) to different other instances (3, 4, 5) of the control software configured to take over control of the industrial process (1) at different time periods from the active instance (2) where the malfunction and / or failure occurs and / or in maintenance scenarios.

7. The method (100) of any one of claims 1 to 3, wherein at least one other instance (3, 4, 5) of control software is configured to control the industrial process (1) in a degraded mode of operation compared to a normal mode of operation performed by the active instance (2).

8. A method (200) for configuring an industrial control system to control an industrial process (1) according to the method (100) of any one of claims 1 to 7, comprising the following steps performed by at least one failover manager and / or other management entity (10, 11): • mapping (210) a given set of requirements (6) for behavior of the control system in case of a failure and / or malfunction of an active instance (2) of control software to a configuration (7) of: o one or more other instances (3, 4, 5) of control software to take over control of the industrial process (1) after the failure and / or malfunction has occurred; and o a respective prioritized subset (2a , 2a , 2a ) of internal variables of the active instance (2) is to be provided to these other instances (3, 4, 5), wherein the respective prioritized subset (2a , 2a , 2a ) of internal variables comprises internal variables that are more important for the control task performed than other internal variables. • creating (220) the at least one or more other instances (3, 4, 5) in a respective virtualized execution environment on at least one virtualization platform according to the configuration (7) thus determined; and • scheduling and / or executing (230) said respective other instances (3, 4, 5) of the control software from said active instance (2) to said respective other instances (3, 4, 5) of the control software. ,2a ,2a ) from said active instance (2) to said respective other instances (3, 4, 5) of the control software.

9. The method (200) of claim 8, wherein the mapping is based at least in part on one or more of: • requirements for a level of reliability of the control of the process (1); • requirements for a level of availability of the control of the process (1); and • requirements for a level of performance of the control of the process (1). • a requirement for a longest time period after which the malfunction and / or failure occurs, during which time period at least one other instance (3, 4, 5) has to take over the control of the industrial process (1); • a time required for creating and / or initializing at least one other instance (3, 4, 5) on the virtualization platform; • an availability of resources for creating other instances (3, 4, 5) on the virtualization platform; • an available communication bandwidth between the active instance (2) and at least one other instance (3, 4, 5); and • a state and / or stage of the industrial process (1).

10. The method (200) of any of claims 8-9, wherein orchestrating the transfer of the prioritized subset of internal variables to the other instance comprises: At least one communication channel is established (231) between the active instance (2) and at least one other instance (3, 4, 5).

11. The method (200) of claim 10, wherein the communication channel comprises: Publish / subscribe one-to-many communication and / or named pipes.

12. The method according to any of claims 8 to 9, wherein the performing of the transfer of the prioritized subset (2a ,2a ,2a ) of internal variables to other instances (3, 4, 5) comprises: a cache (232) of a prioritized subset (2a , 2a , 2a ).

13. One or more computer programs comprising machine-readable instructions which, when executed on one or more computers, cause the one or more computers to perform the method (100, 200) according to any one of claims 1 to 12.

14. One or more non-transitory storage media or download products having the one or more computer programs according to claim 13.

15. One or more computers having the one or more computer programs according to claim 13 and / or having the one or more non-transitory storage media or download products according to claim 14.

Images