Runtime components for backup and high availability

Abstract

To provide a duplex configuration for synchronizing application states.SOLUTION: A primary controller actively monitors and controls a plant / process, and when the primary controller fails, a secondary controller takes over. Input data is received at each input of the primary and secondary controllers. The controller determines which of the received input data is associated with an external event or should be treated as the external event, synchronizes execution of events associated with the same input data, and realizes synchronization of application states between the primary controller and the secondary controller.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 571,788, filed March 29, 2024, and U.S. Patent Application No. 18 / 910,656, filed October 9, 2024, the entire disclosures of which are incorporated herein by reference. [Background technology]

[0002] The need for distributed control topologies has led to the development of programming language standards, such as IEC 61499, an international standard published by the International Electrotechnical Commission (IEC) specifically for distributed (event-based) industrial applications. Generally, IEC 61499 defines a generic architecture that enables application-centric design, where one or more applications, defined by a network of interconnected function blocks, are created for the entire system and then distributed to available devices. All devices in the system are described within a device model, and the system topology is reflected by a system model. IEC 61499 specifies function-block-based distributed control applications for industrial process measurement and control systems. According to this standard, control application execution is event-driven, and events represent changes in the system's state or condition, unlike traditional scan-based distributed control systems. The IEC 61499 standard also specifies a set of software components and applications that implementers of the standard must implement or develop.

[0003] An Integrated Development Environment (IDE) is an engineering and configuration software application or component that can be used to design and develop function blocks, develop control applications, and maintain and manage devices that are assigned to and execute downloaded control applications (or at least portions thereof). A Runtime is a software component that can host and execute applications / control loops downloaded to components according to the execution model specified in the 61499 standard.

[0004] In the IEC 61499 architecture model, distributed applications are constructed similarly to designing a circuit board with integrated circuits by interconnecting reusable function block type components with appropriate event and data connections. Using IEC 61499 compliant software tools, these function blocks can be distributed and deployed over a network into runtime components of IEC 61499 compliant physical devices (controllers). In this way, distributed control and automation systems can be constructed from libraries of reusable IEC 61499 compliant components. Summary of the Invention

[0005] Aspects of the present disclosure provide application state synchronization between a primary (e.g., active) and a secondary (e.g., backup) controller arranged in a duplex configuration by synchronizing the execution of events associated with the same input data. Methods embodying aspects of the present disclosure are efficient and streamlined software implementations that can be used in various deployment models for industrial automation systems, such as bare metal or embedded deployment as an application on an operating system, or deployment as a container.

[0006] In one aspect, an active / primary controller actively monitors and controls a plant / process, and a backup / secondary controller takes over if the active / primary controller fails. The method includes receiving input data at respective inputs of the active / primary controller and the backup / secondary controller. The input data includes one or more of data from sensors associated with the plant / process, data received from an operator interface, and data received from a peer controller. The method further includes classifying which of the received input data is associated with or should be treated as an external event, and achieving application state synchronization between the active / primary controller and the backup / secondary controller by synchronizing execution of external events associated with the same input data.

[0007] In one aspect, a method for achieving application state synchronization includes receiving an event by a primary controller. The primary controller actively monitors and controls at least one process of a plant in response to the received event. The method also includes determining whether the event and associated input data received by the primary controller include an asynchronous external event, queuing the external event for execution by the primary controller, forwarding the external event to a secondary controller in a duplex configuration with the primary controller, and queuing the external event for synchronous execution by the secondary controller. The secondary controller actively monitors and controls at least one process of the plant in response to a failure of the primary controller. The method further includes synchronizing execution of the external event by the primary and secondary controllers in response to the external event and associated input data received by the primary and secondary controllers to achieve application state synchronization between the primary and secondary controllers. According to one or more embodiments, execution of the external event schedules a series of function blocks and executes their algorithms, representing the execution of a control loop based on a user-deployed control application.

[0008] In another aspect, a system for achieving application state synchronization includes a control resource associated with a primary controller and a corresponding control resource associated with a secondary controller in a dual configuration with the primary controller. The system also includes one or more storage memories coupled to the control resource for storing processor-executable instructions. When the instructions are executed, the control resource performs two operations: receiving an event by the primary controller; determining whether the event received by the primary controller and its associated input data comprises an asynchronous external event; queuing the external event for execution by the primary controller; and queuing the external event for synchronous execution by the secondary controller. In response to the external event and associated input data received by the primary and secondary controllers, the instructions cause the control resource to synchronize execution of the external event by the primary and secondary controllers to achieve application state synchronization between the primary and secondary controllers. According to one or more embodiments, execution of the external event results in the scheduling of a series of function blocks and the execution of their algorithms, which represents the execution of a control loop based on a user-deployed control application.

[0009] In yet another aspect, a process control system for achieving application state synchronization includes a dual primary controller and a secondary controller. The system also includes one or more storage memories coupled to the primary controller and the secondary controller, the storage memories storing processor-executable instructions. When executed, the instructions cause the controllers to: receive an event by the primary controller; determine whether the event received by the primary controller and associated input data includes an asynchronous external event; queue the external event for execution by the primary controller; and queue the external event for synchronous execution by the secondary controller. In response to the input data associated with the external event received by the primary controller and the secondary controller, the instructions include causing the controllers to synchronize execution of the external event by the primary controller and the secondary controller to achieve application state synchronization between the primary controller and the secondary controller. According to one or more embodiments, execution of the external event schedules a series of function blocks and executes their algorithms, representing the execution of a control loop based on a user-deployed control application.

[0010] Other objects and features of the present invention will be in part apparent and in part pointed out hereinafter. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 illustrates a process control system according to one embodiment.

[0012] [Figure 2] FIG. 2 illustrates an industrial automation system according to one embodiment.

[0013] [Figure 3] FIG. 3 illustrates a high availability backup method according to one embodiment.

[0014] [Figure 4A] FIG. 4A illustrates an example of an IEC 61499 function block according to one embodiment. [Figure 4B] FIG. 4B illustrates an example of an IEC 61499 function block network according to one embodiment.

[0015] [Figure 5] FIG. 5 illustrates event synchronization between a primary controller and a secondary controller according to one embodiment.

[0016] [Figure 6] FIG. 6 illustrates an example of a simplex configuration of event execution according to one embodiment.

[0017] [Figure 7] FIG. 7 illustrates an example of a dual configuration of event execution according to one embodiment.

[0018] [Figure 8A] 8A and 8B, and 8C and 8D each illustrate an example of synchronization initiation for a duplex configuration according to one embodiment. [Figure 8B] 8A and 8B, and 8C and 8D each illustrate an example of synchronization initiation for a duplex configuration according to one embodiment. [Figure 8C] 8A and 8B, and 8C and 8D each illustrate an example of synchronization initiation for a duplex configuration according to one embodiment. [Figure 8D] 8A and 8B, and 8C and 8D each illustrate an example of synchronization initiation for a duplex configuration according to one embodiment.

[0019] [Figure 9] FIG. 9 illustrates an example of tracking and monitoring event synchronization in a dual configuration according to one embodiment.

[0020] [Figure 10A]FIG. 10A illustrates an example of queue processing on a secondary controller during a failover in a duplex configuration according to one embodiment.

[0021] [Figure 10B] FIG. 10B illustrates an example of a change in the order of events during a failover in a dual configuration according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] Corresponding reference characters indicate corresponding parts throughout the drawings.

[0023] Features and other details of the concepts, systems, and techniques sought to be protected herein are more particularly described below. It will be understood that the particular embodiments described herein are provided by way of example and not as limitations on the disclosure and concepts described herein. Features of the subject matter described herein can be employed in various embodiments without departing from the scope of the concepts sought to be protected.

[0024] As mentioned above, in the IEC 61499 standard, the execution of a control application is event-driven, with an event representing a change in the state or condition of the system. According to IEC 61499, an "event" is a momentary occurrence that is significant to the execution schedule of a function block / algorithm. Aspects of the present disclosure provide a runtime solution in which external events represent asynchronous stimuli / changes to the execution of control logic. External events, such as a change in an input / output (IO) value, an update of an operator setting from a human-machine interface (HMI), or a data change from a subscribed peer controller, trigger the execution of function blocks within the controller. In one embodiment, even events generated by an internal timer object are treated as external events. In this respect, external events are a subset of events defined in IEC 61499. Synchronicity relates to the execution of the control logic (i.e., the scheduling and execution of a series of function block (FB) components). This is in contrast to traditional scan-based execution (read inputs - execute logic - write outputs). For example, in IEC 61131, "events" are evaluated synchronously with the execution of logic and are always evaluated at the "read input" step before the logic is executed.

[0025] According to aspects of the present disclosure, an external event represents an asynchronous stimulus / change to the device's environment (e.g., expiration of a timer, receipt of a network packet from an HMI), and therefore causes a function block (i.e., control logic) to be asynchronously queued to a resource for later execution. A runtime subsystem (e.g., OPCUA server, MODBUS client) queues an external event when it detects receipt of an OPCUA client request or receipt of a MODBUS response from the network via the network interface controller and OS networking stack.

[0026] This disclosure describes a method for achieving application state synchronization between a primary (e.g., active) and secondary (e.g., backup) controller in a dual configuration by synchronizing the execution of external events associated with the same input data, which represent a new state or condition of the system. The order of environmental changes / excitations (and therefore the order in which external events are queued and executed on the primary device) and the associated input values ​​detected on the primary are considered the only truth (until a failure occurs on the primary). If the excitation order, i.e., input values, of the primary and secondary controllers differ, the primary's belief always prevails. When identical applications on two controllers receive the same external event with the same input data, the states of their function blocks after execution will be identical. Synchronizing the associated input data and external events from the primary controller to the secondary controller ensures identical control application states at the end of execution.

[0027] In one embodiment, synchronous execution is based on the order in which asynchronous external events are queued, not temporal. For example, an external event on a secondary controller may execute several seconds (e.g., 2 seconds) after the same event executes on the primary controller, but aspects of the present disclosure ensure synchronization of the final state after all events have executed. According to one or more embodiments, execution of an external event leads to the scheduling of a series of function blocks and the execution of their algorithms, which represents the execution of a control loop based on a user-deployed control application.

[0028] Referring now to the drawings, aspects of the present disclosure enable the definition and modeling of resources for industrial systems and the generation of libraries of such resource models. Resource libraries include programming elements (e.g., basic blocks and composite blocks) required to construct resources. In one embodiment, these programming elements are defined according to a distributed control programming standard such as IEC 61499. These models show the relationships between physical and control resources (e.g., different physical model levels), allowing the resource libraries to be mapped to physical devices and control languages ​​or control descriptions.

[0029] Additionally, aspects of the present disclosure provide the ability to simulate the design and operation of industrial systems using resource model libraries, allowing the simulated operation to be evaluated to identify potential improvements to proposed system designs.

[0030] 1 shows the basic structure of an exemplary process control system 100. In one embodiment, at least one process 102 is communicatively connected to a controller 104 and a sensor 106. The process 102 has inputs 108 and 110 that contain inputs the process needs to create an output 112. In one embodiment, the input 108 includes energy to run the process 102, and the input 110 includes physical or chemical raw materials used in the process 102. The output 112 consists of physical or chemical products from the process 102, or energy products in the form of electricity, etc.

[0031] The controller 104 sends data to at least one process 102 to direct its operation according to the controller's 104 goals. The sent data includes commands to operate various types of process elements, devices, or resources 114 in the process 102, such as pumps, motors, valves, actuators, electrostatic precipitators, electrolyzers, vibrators, heaters, etc. The resources 114 may be mechanical, chemical, electrical, biological, or composite mechanisms or sets of mechanisms used to convert energy and materials into value-added products or production. Sensors 106 monitor the process 102 at various points and collect data from those points. The sensors 106 send the collected data to the controller 104. Based on the collected data, the controller 104 can send additional commands to the process 102. In this way, the system 100 forms a control feedback loop, with the controller 104 reacting to changes in the process 102 observed by the sensors 106. Different actions performed by the processes 102 pursuant to the commands of the controller 104 may change the data collected by the sensors 106, thus causing further adjustments by the controller 104 in response to those changes. By implementing this control feedback loop, at least one process 102 is controlled by the controller 104 in an efficient and effective manner.

[0032] To ensure safe operation, the controller 104 includes one or more condition or resource monitoring systems 116 that are responsive to sensors 106 that collect process measurements such as temperature, flow rate, pressure, chemical composition, fluid properties, vibration analysis, motor current signature analysis, ultrasonic analysis, thermal analysis, etc. of critical resources 114. In the illustrated embodiment, the system 100 also includes a historian 118 to capture and store industrial data including process, alarm, and event history data.

[0033] As shown in FIG. 1 , the distributed control system 120, including the controller 104, operates in conjunction with a human-machine interface (HMI) 126. The HMI 126 is an input / output device that presents process information to a human operator. The control system 120 links to the HMI 126 to provide specific sensor or machine maintenance procedures, detailed circuit diagrams, logistical information, trend data, diagnostic data, configuration data transfer, and the like. In an embodiment, the HMI 126 includes a personal computer, smartphone, tablet, touchscreen HMI device, and the like. While illustrated remote from the various industrial resources 114 (e.g., a control room), it should be understood that the HMI 126 can be hosted on the device itself.

[0034] 2 illustrates an industrial automation system 200 that implements the IEC 61499 standard. As known in the art, an IEC 61499 control application is a collection of interconnected function blocks that form a function block network. The runtime (RT) software component 202 in the illustrated embodiment is designed and implemented as a set of software services that host and execute the IEC 61499 control application.

[0035] The RT software components 202 can execute on any computing device or node that includes a processor and memory (with appropriate processing power and memory size). For example, as shown in Figure 2, in industrial control, they execute on programmable logic controllers (PLCs), controllers used in distributed control systems (DCSs), and industrial personal computers (IPCs), generally referred to as controllers 104 in Figure 2. The RT software components 202 utilize services provided by the host operating system of the device or compute node to perform duties such as communicating with other participating entities in the system and storing and retrieving data.

[0036] Referring now to Figure 3, the industry typically requires industrial control system vendors to provide highly available, fault-tolerant control systems that ensure continuous operation even in the event of a fault or failure. To achieve this, such vendors often implement active and backup mechanisms in their controllers. High-availability (HA) controllers are typically designed in a duplex configuration, with a pair of individual controllers 104 operating in active / primary and backup modes, respectively. The primary controller 104A of the pair actively monitors and controls the plant / process, while the secondary (backup) controller 104B of the pair waits to take over in the event of a failure. Synchronization of the control application state from the primary controller 104A to the secondary controller 104B is crucial to ensuring a seamless transition and avoiding undesirable changes in plant / process control. This synchronization can involve full or partial state synchronization and occurs at specific intervals, depending on the design and implementation. Furthermore, in many designs, such as hot standby redundancy, the complete state of each controller 104A, 104B is synchronized, including the control application state, service state, and underlying platform state. Most of these solutions are tightly coupled to the underlying hardware, making them difficult to port to other hardware.

[0037] For example, a high-speed Gigabit Ethernet interlink is used for ultra-fast and efficient synchronization of state from the primary controller to the secondary controller. Preferably, the full bandwidth of the interlink can be used to achieve the expected performance. In one embodiment, the interlink is a dedicated and direct interlink between the two controllers 104A, 104B, with an interlink protocol used for event synchronization and other data synchronization without a switch network. For example, external events are sent over the interlink to the partner / secondary device, which receives the external events and queues them accordingly. The external events are then executed on both the primary and secondary devices.

[0038] Figure 4A shows an example of an IEC 61499 function block (FB). As known to those skilled in the art, a function block is an encapsulation of code / algorithm with input and output interfaces, similar to a class / object in software object-oriented programming. An interface is a set of events and data, and events and data on the same interface are always selectively associated. Figure 4B shows an example of an IEC 61499 function block network (FB network) in which a set of function blocks are connected to each other via event connections. An event connection always has an output event of a function block component as its source and an input event of another function block as its destination.

[0039] A control loop is a fundamental component of control systems in general, and industrial control systems such as process control system 100 in particular. It includes process sensors, controller functions, and final control elements (FCEs) that control the process required to automatically adjust the value of a measured process variable (PV) to equal a desired setpoint (SP) value. In one embodiment, a control loop implementation includes an IEC 61499 FB network. For example, a simple control loop implemented as an IEC 61499 control application includes an FB network with three FBs, as shown in the example of FIG. 4B. It has an input FB, such as an analog or discrete input block, a control block, such as a PID block, and an output block, such as an analog or discrete output block. In the example control loop, a “REQ_PV” is generated when a sensor providing an analog input value reports a change. This triggers the following sequence of events, leading to the execution of the complete control loop, which is nothing other than an IEC 61499 FB network: for example, "Analog input_1 CNF" → "PID_1 REQ" → "PID_1 CNF" → "Analog output_1 REQ_SP" → "Analog output_1 CNF"

[0040] External events can be thought of as "vertical" to the FB / FB network (event connections from one FB to the next) and "independent" from the FB / FB network (event connections from FB to next FB). As such, they are "independent" of the control loop / event connections. In other words, an FB can be scheduled and executed (typically producing output events without the upstream FB being executed or the FB being connected to an upstream FB).

[0041] Referring now to FIG. 5, control application execution is event-driven in accordance with the IEC 61499 standard, where an event represents a change in the system state or condition. External events, such as a change in an IO value, an update of an operator setting from an HMI, or a data change from a subscribed peer controller, trigger the execution of function blocks within the controller. Events generated by internal timer objects are also treated as external events according to the embodiment. If identical applications on two controllers (e.g., controllers 104A and 104B) receive identical external events with identical input data and execute in the same order as other external events, the state of the function blocks after execution will be identical. Synchronizing associated input data and external events from the primary controller to the secondary controller ensures identical control application states at the end of execution.

[0042] According to aspects of the present disclosure, application state synchronicity is achieved between an active (e.g., primary controller 104A) and a backup (e.g., secondary controller 104B) in a duplex configuration by synchronizing the execution of events associated with the same input data. In one embodiment, the software implementation can be used with different deployment models of the RT software component 202, such as bare-metal or embedded deployment as an application on an operating system or deployment as a container that synchronizes event execution. Although referred to as receiving, the primary does not receive external events. Instead, external events are sent to the secondary (along with input data) and synchronized for execution on both devices. Because both devices execute the same events in the same order with the same input data, the computed outputs are the same, and state synchronicity is achieved.

[0043] In addition to synchronous execution of events, additional methods embodying aspects of the present disclosure are implemented to achieve high availability of the entire system, including: Designate one of the controllers 104 as the preferred primary - As part of a duplex configuration, designate one of the controllers 104 as the preferred primary. This is used to avoid conflicts during synchronized startup, i.e., when both controllers 104A and 104B are started at the same time. In one embodiment, the priority designation is only used when a boot project is involved and two controllers are started after a power cycle. When connected to an engineering tool and the user is actively deploying / starting the controllers, the user can configure which device will be the "primary" once a running state is reached. Synchronous startup of dual controllers 104 - When both controllers 104A, 104B start simultaneously, the preferred primary controller (e.g., controller 140A) always takes the lead and follows its partner controller (e.g., controller 104B) until they synchronize and enter run mode. Another scenario in which "resynchronization" is required is when a device fails (a failover occurs) and the failed device is replaced. Asynchronous startup - the primary controller 104A is started and put into a running state first, and then the secondary controller 104B is started. Thus, the already running primary controller 104A takes a short pause (blackout period) and during this pause it informs the participating controller 104B by synchronizing the current application state, i.e., the state and inputs / outputs of each FB component, with the participating controller 104B. The two controllers 104A, 104B then start synchronous execution at an agreed-upon absolute time. Synchronization of IO data and communication processing - The primary controller 104A actively scans IO devices and synchronizes data to the secondary controller 104B. The secondary controller 104B does not actively scan IO data, but establishes the necessary connections and communications with IO devices and maintains keep-alive messages so that it can begin active scanning immediately after taking over the primary role. Event staging on the secondary - Events are also generated on the secondary controller 104B, but are staged in a separate queue. Events in the staging queue are removed when they "match" events synchronized from the primary controller 104A. This ensures that events generated during failover are not lost, and the secondary controller 104B executes these events after taking over. Tight (sub-millisecond accuracy) time synchronization—The system clocks and monotonic clocks of the two controllers 104 in a duplex configuration are kept tightly synchronized to ensure synchronization of timer object events. For example, an NTP-based time synchronization method is used to synchronize the time of the two controllers 104.

[0044] According to a specific embodiment, the two controllers 104 use the same platform (e.g., CPU architecture and operating system). It should be understood that aspects of the present disclosure are hardware-agnostic and do not require the same platform. Using the same OS (and its version) facilitates implementation and verification. Additionally, the output of calculations on the same processor using the same input data is expected to be identical. This is because, for example, different CPU architectures may perform floating-point calculations differently, resulting in application states diverging even when the same FB is executed on both controllers. Also, different endianness may result in different interpretations of binary data, leading to different calculation results. Alternatively, the use of non-identical processors is possible if two processors have the same precision, regardless of whether they are identical (same architecture, same vendor).

[0045] In an embodiment, a controller 104 used to replace a failed controller in a duplex configuration is entrusted with the same duplex configuration as the failed controller, i.e., the same IP address and the same cybersecurity configuration, etc., to facilitate interoperability.

[0046] Advantageously, a runtime component 202 embodying aspects of the present disclosure enables higher performance at lower CPU and network costs compared to traditional scan-based PLC high availability solutions. When a process or plant is in a steady state and process changes are low, synchronizing events consumes less CPU time and network bandwidth than synchronizing the complete controller state, or only the application state, or only the application substate. This makes the runtime component 202 suitable for use with controllers with low CPU power and systems with limited network bandwidth compared to existing solutions.

[0047] Additionally, aspects of the present disclosure enable minute switchover times. Switchover time is the time it takes to switch control functions from the current active or primary controller 104A to the standby or secondary controller 104B. In one embodiment, the runtime component 202 provides switchover times of, for example, tens of milliseconds. Switchovers can be user-requested or can occur automatically due to a fault or failure of the current active or primary controller 104A.

[0048] Figure 6 illustrates an example runtime software component 202 representing a simplex logical device for event execution. The IEC 61499 device model includes resources, with each resource 600 assigned a set of blocks that must execute in its context. In accordance with one or more embodiments, the IEC 61499 resource 600 is modeled as two event queues (an external event queue 602 and an internal event queue 604) and a set of execution threads (hereafter referred to as execution engines 606). In one embodiment, internal events consist of events that are reactive or deterministic, as defined by IEC 61499, and are executed asynchronously and not queued.

[0049] According to one embodiment of the runtime implementation, at startup, both event queues 602, 604 are empty. When an external event is generated, it is added to the external event queue 602. If the external event queue 602 is not empty, the execution engine 606 fetches one external event and adds it to the internal event queue 604. On the other hand, if the external event queue 602 is empty, the execution engine 606 continues to check for events. If the internal event queue 604 is not empty, the execution engine 606 fetches one event from the queue and schedules it for execution. On the other hand, if the internal event queue 604 is empty, the execution engine 606 checks the external event queue 602. Execution of an event consists of executing the function block code pointed to by the event. When a function block is executed, an event is generated with associated data as an output. If the generated event is connected to an event connection to a function block mapped to the current device, it is added to the internal event queue 604. In this way, execution of an external event causes a chain of function blocks to be executed. If the generated events refer to a function block that is mapped to another device, they are sent to the corresponding device via the peer-to-peer communication interface.

[0050] Referring now to FIG. 7, a dual configuration for event synchronization is illustrated according to one or more embodiments. Synchronization event messages are sent only one way, from the current primary instance to the secondary instance. According to aspects of the present disclosure, the primary resource 600A is always assumed to be correct, and the secondary resource 600B simply deferred to it in terms of executing external events. According to one or more embodiments, both the primary and secondary resources 600A, 600B employ a new service named "synchronization service" for sending and receiving synchronization messages. In the primary resource 600A, if the internal event queue 604A is empty, while fetching an external event from the external event queue 602A, the execution engine 606A checks whether the event has been synchronized to the secondary. If the event has been synchronized to the secondary, the execution engine moves the event to the internal event queue. If the event has not been synchronized to the secondary resource 600B, all external events not synchronized to the secondary resource 600B are sent to the secondary resource 600B as synchronization event messages via the synchronization service.

[0051] In the secondary resource 600B, a new queue called staging queue 704 is defined, and all external events due to changes / excitements queued on the secondary are added to this queue instead of external event queue 602B. As synchronization event(s) message(s) are received from the primary resource 600A, these events are added to external event queue 602B, and the corresponding events in staging queue 704 are marked as matching the primary resource 600A. Thus, corresponding events in staging queue 704 are eventually removed from the queue once the corresponding event has finished executing. This prevents the secondary from executing duplicate external events and also provides a way to detect potential faults / failures in either device.

[0052] 8A and 8B each illustrate a first scenario of synchronization initiation in a dual configuration according to one embodiment. At 802, an external event 1 is added to the primary. As shown in FIG. 8B at 804, since the execution pointer (POS 0) == the synchronization pointer (POS 0), a synchronization is triggered at 806. All events in the external queue from the synchronization pointer to the end pointer are sent as synchronization event messages to the secondary. Proceeding to 808, the primary continues executing events. As a result of event chain execution, other events may be added to the internal event queue (1A, 1B, 1C). The secondary continues execution at 810, slightly behind the primary. At 812, new events (2 and 3) being executed are placed in the external event queue. As shown in FIG. 8B at 814, since the execution pointer (POS 1) == the synchronization pointer (POS 1), the next synchronization is triggered at 816 (similar to the synchronization at 806), but this time it synchronizes two events (2 and 3).

[0053] In one embodiment, primary to secondary synchronization is triggered when the execution pointer attempts to overtake the synchronization pointer. No primary or secondary execution occurs during synchronization; an event queue overflow occurs when the end pointer overtakes the synchronization pointer (not the execution pointer).

[0054] 8C and 8D each illustrate a second scenario of synchronization initiation in a dual configuration according to an embodiment. Starting at 818, external event 1 is added to the primary, triggering synchronization at 820. All events in the external queue from the synchronization pointer to the end pointer are sent as synchronization event messages to the secondary. Proceeding to 822, the primary continues executing events. The secondary continues executing at 824, albeit slightly behind the primary. At 826, new events (2 and 3) being executed are added to the external event queue. The next synchronization is triggered at 828 (similar to synchronization at 806), but this time it synchronizes two events (events 2 and 3). While events 2 and 3 are executing, as shown at 830 in FIG. 8D, events 4, 5, and 6 are added to the external event queue, as shown at 832 in FIG. 8D. When the execution pointer moves to event 4 at 834, event 4 becomes the same as the synchronization pointer, so synchronization is invoked to send all events from the synchronization pointer to the end pointer. That is, events 4, 5 and 6 are sent to the secondary as synchronous event messages, as shown in FIG. 8D at 836.

[0055] FIG. 9 illustrates an example of tracking and monitoring event synchronization in a duplex configuration, according to one embodiment. In the illustrated embodiment, Event 1 at 902 arrives late at the secondary, as shown at 904, and has already been acted upon due to a synchronization event received from the primary. This is detected using event tracking information maintained in MAP1 and MAP2. In an embodiment, it is tolerated because it arrived within a timeout period. Event X at 906 is received only at the secondary, as shown at 908, and is added to a staging queue, with tracking information added to MAP1 and MAP2. As shown in FIG. 9, Event X never arrives as a synchronization event from the primary, and a timeout is detected at 910 using the tracking information in MAP1 and MAP2, indicating that Event X has been completely lost by the primary. This potentially indicates a failure in the primary device, and HA diagnostics are notified so that a failover decision can be made.

[0056] In one embodiment, two MAP data structures are maintained to track events added to the staging queue and the external event queue. In this embodiment, MAP1 is used to track the enqueue time of the event, and MAP2 is used to track the arrival of the event in the two queues (i.e., whether it arrived in the staging queue only, the external event queue only, or both) and the offset at which the event resides in the staging queue. If an event only arrived in the staging queue and never arrived as a synchronous event from the primary, and a check against the enqueue time indicates that the event has remained in this state for more than XX (10) ms, it indicates that the event has been lost on the primary. If an event only arrived in the external event queue as a synchronous event and never arrived in the staging queue, and a check against the enqueue time indicates that the event has remained in this state for more than XX (10) ms, it indicates that the event has been lost on the secondary. If an event has only arrived in the external event queue as a synchronous event and never arrived in the staging queue, and the event has remained in this state for more than XX (10) ms, it indicates that the event has been lost on the secondary. If an event loss is detected, HA diagnostics is notified.

[0057] In the event of a failover from primary to secondary, all events that have arrived in the staging queue but have not yet arrived in the external event queue as synchronous events (events that were lost on the primary but arrived on the secondary) are copied from the staging queue to the external event queue and executed after the secondary assumes single responsibility, using the staging queue offset stored in MAP2.

[0058] FIG. 10A illustrates an example of queue processing on a secondary controller during a failover in a duplex configuration, according to an embodiment. Events 2, X, and 3 have arrived in the staging queue but have not yet been received as synchronous events from the primary. Meanwhile, the primary has failed. Therefore, using tracking information from MAP1 and MAP2, all of these events are moved to the external event queue at 1002 before the secondary assumes single responsibility and begins execution.

[0059] FIG. 10B illustrates an example of a change in event ordering during a failover in a dual-tier configuration according to an embodiment. Event 2 is received only by the secondary at 1006, as indicated by 1004, and added to a staging queue, with tracking information added to MAP1 and MAP2. Event 3 is received by both the primary and secondary at 1008. It is added to a staging queue at the secondary, with tracking information added to MAP1 and MAP2. Event 3 arrived at the secondary via a synchronization message from the primary before the primary failed. The tracking information for MAP1 and MAP2 is updated accordingly. Referring to 1010 in FIG. 10B, a change in the ordering of directly related events is illustrated. Event 2 arrives before Event 3 on the secondary, but executes after Event 3 because the synchronization event for Event 2 is pending and not yet received from the primary. As indicated by the tracking information for MAP1 and MAP2, the event is copied to the external event queue just before the secondary assumes the role of the primary and begins execution.

[0060] A method embodying aspects of the present disclosure provides application state synchronization between an active / primary controller and a backup / secondary controller in a dual configuration, where the active / primary controller actively monitors and controls a plant / process and the backup / secondary controller takes over if the active / primary controller fails. The method includes receiving input data at respective inputs of the active / primary controller and the backup / secondary controller. The input data includes one or more of data from sensors associated with the plant / process, data received from an operator interface, and data received from a peer controller. The method further includes classifying which of the received input data is related to or should be treated as an external event, and achieving application state synchronicity between the active / primary controller and the backup / secondary controller by synchronizing execution of events related to the same input data.

[0061] Alternatively or additionally, further aspects of the present disclosure include containerized active / backup controllers, dual active and backup controllers that do not require a dedicated and direct interlink between them, dual active and backup controllers with diverse hardware platforms, dual active and backup controllers with diverse operating systems, and dual active and backup controllers. Advantageously, aspects of the present disclosure provide HA industrial automation systems with very low hardware coupling / dependency, similar to softPLC-based HA industrial automation.

[0062] Embodiments of the present disclosure may include a special purpose computer having various computer hardware, as described in more detail herein.

[0063] For purposes of illustration, programs and other executable program components may be depicted as discrete blocks, however it is recognized that such programs and components reside at various times in different storage components of the computing device, and are executed by the device's data processor.

[0064] Although described in connection with an exemplary computing system environment, embodiments of aspects of the present invention are operational with other special-purpose computing system environments or configurations. The computing system environment is not intended to suggest any limitation as to the scope of use or functionality of any aspect of the present invention. Moreover, the computing system environment should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment. Examples of computing systems, environments, and / or configurations suitable for use with aspects of the present invention include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0065] Embodiments of aspects of the present disclosure may be described in the general context of data and / or processor-executable instructions, such as program modules, stored on one or more tangible, non-transitory storage media and executed by one or more processors or other devices. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types. Aspects of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote storage media, including memory storage devices.

[0066] In operation, a processor, computer, and / or server may execute the processor-executable instructions (e.g., software, firmware, and / or hardware) illustrated herein to implement aspects of the present invention.

[0067] Embodiments may be implemented with processor-executable instructions. The processor-executable instructions may be embodied in one or more processor-executable components or modules on a tangible processor-readable storage medium. Also, embodiments may be implemented with any number and organization of such components or modules. For example, aspects of the present disclosure are not limited to the specific processor-executable instructions or the specific components or modules illustrated and described herein. Other embodiments may include different processor-executable instructions or components having more or less functionality than illustrated and described herein.

[0068] The execution or order of performance of operations in accordance with aspects of the present disclosure illustrated and described herein is not required unless otherwise specified. That is, operations may be performed in any order unless otherwise specified, and embodiments may include additional or fewer operations than those disclosed herein. For example, it is contemplated that performing or executing particular operations before, contemporaneously with, or after other operations is within the scope of the invention.

[0069] When introducing elements of the invention or embodiments thereof, the articles "a," "an," "the," and "said" are intended to mean that there are one or more of the elements. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements other than the listed elements.

[0070] Not all of the depicted components shown or described may be required. Furthermore, some implementations and embodiments may include additional components. Variations in the arrangement and type of components may be made without departing from the spirit or scope of the claims defined herein. Additional, different, or fewer components may be provided, or components may be combined. Alternatively, or additionally, a component may be implemented by multiple components.

[0071] The above description describes embodiments by way of example and not limitation. This specification describes several embodiments, adaptations, variations, alternatives, and uses of aspects of the present invention, including what is currently contemplated to be the best mode of carrying out aspects of the present invention, to enable those skilled in the art to make and use the same. Furthermore, it is to be understood that aspects of the present invention are not limited in their application to the details of construction and the arrangements of components set forth in the following description or illustrated in the drawings. Aspects of the present invention are capable of other embodiments and of being practiced or carried out in various ways. It is also to be understood that the terms and terminology used herein are for the purpose of description and should not be regarded as limiting.

[0072] It will be apparent that modifications and variations are possible without departing from the scope of the invention as defined in the appended claims. Since various changes can be made in the structure and methods described above without departing from the scope of the invention, it is intended that all matter contained in the above description and shown in the accompanying drawings be interpreted as illustrative and not in a limiting sense.

[0073] From the above, it will be seen that the several advantages of the aspects of the invention are achieved and other advantageous results attained.

[0074] The Abstract and Overview are provided to aid the reader in quickly grasping the nature of the technical disclosure. They are submitted with the understanding that they will not be used to interpret or limit the scope or meaning of the claims. The Abstract is provided to introduce some of the concepts further described in the Detailed Description in a simplified form. This Abstract is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the claimed subject matter.

Claims

1. 1. A method for achieving application state synchronicity, comprising: receiving, by a primary controller, an event, the primary controller actively monitoring and controlling at least one process of the plant in response to the received event; determining, by the primary controller, whether the event and associated input data comprises an asynchronous external event; queuing the external event for execution by the primary controller; forwarding the external event to a secondary controller in a duplex configuration with the primary controller, the secondary controller actively monitoring and controlling the at least one process of the plant in response to a failure of the primary controller; queuing the external event for synchronous execution by the secondary controller; In response to the external events and the associated input data received by the primary and secondary controllers, synchronizing execution of the external events by the primary and secondary controllers to achieve application state synchronization between the primary and secondary controllers; A method comprising:

2. the input data associated with the external event includes one or more of an input / output (IO) value change, an operator setting value update from a human machine interface, a data change from a subscribed peer controller, and an event generated by an internal timer object; The method of claim 1.

3. and synchronizing execution of the external event includes triggering execution of one or more function blocks of a control resource of the primary controller and a corresponding control resource of the secondary controller in response to the external event.

3. The method according to claim 1 or 2.

4. and synchronizing execution of the external events includes maintaining a uniform control application state upon completion of execution.

3. The method according to claim 1 or 2.

5. further comprising designating one controller in the dual configuration as a preferred primary.

5. The method according to any one of claims 1 to 4.

6. and actively scanning, by the primary controller, one or more IO devices and synchronizing data from the IO devices to the secondary controller.

6. The method according to any one of claims 1 to 5.

7. and actively scanning, by the secondary controller, for the one or more IO devices after taking over the primary role from the primary controller. The method of claim 6.

8. staging the secondary controller events in a staging queue that is distinct from an external event queue.

8. The method according to any one of claims 1 to 7.

9. and removing the event from the staging queue when the same event is synchronized from the primary controller. The method of claim 8.

10. queuing the external event for synchronous execution by the secondary controller includes executing a runtime software component executing on one or more of a programmable logic controller (PLC), a controller of a distributed control system (DCS), and an industrial personal computer (IPC).

10. The method of any one of claims 1 to 9.

11. the input data includes one or more of data from sensors associated with at least one process of the plant, data received from an operator interface, and data received from a peer controller; 11. The method of any one of claims 1 to 10.

12. A system for achieving application state synchronicity a control resource associated with a primary controller, a corresponding control resource associated with a secondary controller in a duplex configuration with said primary controller, and one or more storage memories; the primary controller actively monitors and controls at least one process of the plant; the secondary controller actively monitors and controls the at least one process of the plant in response to a failure of the primary controller; the one or more storage memories are coupled to the control resource and store processor-executable instructions; The processor-executable instructions, when executed, cause the controlled resource to: receiving, by the primary controller, an event; queuing, by the primary controller, the external event for execution in response to an event and associated input data received by the primary controller, the external event including an asynchronous external event; forwarding the external event to a secondary controller; queuing the external event for synchronous execution by the secondary controller; In response to the external events and the associated input data received by the primary and secondary controllers, synchronizing execution of the external events by the primary and secondary controllers to achieve application state synchronization between the primary and secondary controllers; A system that executes the following.

13. the input data associated with the external event includes one or more of an input / output (IO) value change, an operator setting value update from a human machine interface, a data change from a subscribed peer controller, and an event generated by an internal timer object; The system of claim 12.

14. and synchronizing execution of the external event includes triggering execution of one or more function blocks within a control resource of the primary controller and a corresponding control resource of the secondary controller in response to the external event.

14. A system according to claim 12 or 13.

15. and synchronizing execution of the external events includes maintaining a uniform control application state upon completion of execution.

14. A system according to claim 12 or 13.

16. the one or more storage memories storing processor-executable instructions that, when executed, further cause the control resource to designate one controller in the duplex configuration as a preferred primary.

16. A system according to any one of claims 12 to 15.

17. the one or more storage memories storing processor-executable instructions that, when executed, further cause the control resources of the primary controller to actively scan one or more IO devices and synchronize data from the IO devices to the secondary controller.

17. A system according to any one of claims 12 to 16.

18. the one or more storage memories storing processor-executable instructions that, when executed, further cause the control resources of the secondary controller to actively scan the one or more IO devices after taking over a primary role from the primary controller.

20. The system of claim 17.

19. a staging queue of the secondary controller that is distinct from an external event queue; the one or more storage memories storing processor-executable instructions that, when executed, cause the control resources of the secondary controller to stage an event of the secondary controller in a staging queue.

19. A system according to any one of claims 12 to 18.

20. the one or more storage memories storing processor-executable instructions that, when executed, cause the control resource of the secondary controller to remove an event from the staging queue when the same event is synchronized from the primary controller.

20. The system of claim 19.

21. queuing the external event for synchronous execution by the secondary controller includes executing a runtime software component; the runtime component executes on one or more of a programmable logic controller (PLC), a controller of a distributed control system (DCS), and an industrial personal computer (IPC); 21. A system according to any one of claims 12 to 20.

22. the input data includes one or more of data from sensors associated with at least one process of the plant, data received from an operator interface, and data received from a peer controller; 22. A system according to any one of claims 12 to 21.

23. each of the control resources includes at least one of a programmable logic controller (PLC), a controller of a distributed control system (DCS), and an industrial personal computer (IPC); queuing the external event for synchronous execution by the secondary controller includes executing a runtime software component on at least one of the controlled resources.

23. A system according to any one of claims 12 to 22.

24. 1. A process control system for achieving application state synchronicity, comprising: a primary controller, a secondary controller configured in a duplex configuration with the primary controller, and one or more storage memories; the primary controller actively monitors and controls at least one process of the plant; the secondary controller actively monitors and controls at least one process of the plant in response to a failure of the primary controller; the one or more storage memories are coupled to the primary controller and the secondary controller and store processor-executable instructions; The processor-executable instructions, when executed, cause the controller to: receiving, by the primary controller, an asynchronous external event; queuing the external event for execution by the primary controller; forwarding the external event to the secondary controller; queuing the external event for synchronous execution by the secondary controller; In response to the external events and the associated input data received by the primary and secondary controllers, synchronizing, by the primary and secondary controllers, execution of the external events to achieve application state synchronization between the primary and secondary controllers; A process control system that executes the process.

25. queuing an external event for synchronous execution by the secondary controller includes executing a runtime software component; the runtime component executes on one or more of a programmable logic controller (PLC), a controller of a distributed control system (DCS), and an industrial personal computer (IPC); 25. The process control system of claim 24.

Images