Real-time fault-tolerant checkpointing

Abstract

To provide a real-time fault tolerance check point.SOLUTION: A real-time fault tolerant system may include: a first computing device; a second computing device; and hardware interconnect. The first computing device may include: one or more memory devices; one or more processors; a first network interface operable to receive device data via a time slot-based bus to transmit output data, the output data being generated by processing device data; and a first real-time check point engine. The second computing device may include a component similar or identical to the first computing device.SELECTED DRAWING: None

Description

[Background technology]

[0001] Checkpointing techniques are a means for achieving fault-tolerant behavior in computer systems. Current checkpointing techniques, whether virtualization-based or OS-based, are incompatible with hard real-time operation in that checkpoints are required to occur at any point in time, thereby affecting system response time and potentially causing deadlines to be missed.

[0002] Applications and systems that require both real-time support and fault-tolerant operation are currently designed using custom software and / or hardware that are bespoke and tailored for specific industries and purposes, which requires significant expenditure and time to develop, deploy, and maintain such systems.

[0003] The present disclosure provides systems and methods that address the challenges associated with checkpointing as described above, while providing real-time fault tolerance suitable for use in a variety of industries and support levels. Summary of the Invention [Means for solving the problem]

[0004] In part, this disclosure relates to a real-time fault-tolerant computing system. The system may include a first computing device, a second computing device, and a hardware interconnect. The first computing device may include one or more memory devices, one or more processors, a first network interface operable to receive device data and transmit output data via a timeslot-based bus, where the output data is generated from processing the device data, and a first real-time checkpoint engine. The second computing device may include one or more memory devices, one or more processors, a second network interface operable to receive device data via the timeslot-based bus, and a second real-time checkpoint engine. In one embodiment, the hardware interconnect is operable to enable data exchange between the first computing device and the second computing device. In one embodiment, checkpoints are generated by the first or second real-time checkpoint engine during lower priority communication timeslots allocated on the timeslot-based bus to avoid interference with any real-time communications to or from the first and second computing devices.

[0005] In one embodiment, the first computing device and the second computing device process received device data simultaneously, such that if the first computing device suffers a failure, the second computing device continues to process the device data, generate output data, and transmit the output data using the time slot-based bus. In one embodiment, checkpoint operations are always postponed until the occurrence of a lower priority communication slot, even if the first computing device and the second computing device generate divergent output data prior to the occurrence of the lower priority communication slot. In one embodiment, both the first computing device and the second computing device delay the release of output data to one or more devices communicating with the time slot-based bus until after a subsequent checkpoint to avoid releasing divergent output data if a failure occurs.

[0006] In one embodiment, the first computing device generates a checkpoint every cycle when output data including low priority data is transmitted using a low priority time slot. In one embodiment, the device is a sensor, the output data includes sensor data, and the sensor data is operable as an input for a control system. In one embodiment, both the first computing device and the second computing device are actively processing device data and generating output data such that if the first computing device experiences one or more failure modes, there is about zero or no time delay for the second processing device to continue processing.

[0007] In one embodiment, the duration of one or more time slots is increased by a detection period, the detection period ranging from about 1 microsecond to about 100 microseconds. In one embodiment, both the first computing device and the second computing device delay the release of output data to devices communicating with the time slot-based bus until after the next checkpoint to avoid releasing inconsistent output data. In one embodiment, if the first computing device is operating normally after the checkpoint, the first computing device releases any delayed output data. In one embodiment, the first computing device transmits all of its output data to the second computing device to avoid such output data being dropped in response to a failure of the first computing device.

[0008] In one embodiment, if a first computing device fails, the second computing device takes over the release of the first computing device's output data generated during the last checkpoint. In one embodiment, only the first computing device handles the release of output data when no failure occurs and, in parallel, shares such output data with the second computing device. In one embodiment, device data from one or more devices is transmitted to the first computing device and the second computing device via a time-slot-based bus. In one embodiment, the first computing device and the second computing device receive incoming data directly through broadcast or multicast.

[0009] In one embodiment, the first computing device and the second computing device each include a separate, non-virtual operating system (OS) stored in one or more memory devices and executing on each of the one or more processors, the separate operating system (OS) including a separate network driver, a checkpoint shim, and a storage device driver. In one embodiment, the network driver includes a replicator that communicates with a release coordinator and a release notifier. In one embodiment, one or more networking programs stored in one or more memory devices of the first and second computing devices delay and coordinate transmission of output data using one or more of a hardware interconnect, a first network interface, and a second network interface. In one embodiment, the first computing device and the second computing device each include a separate, virtual operating system (OS) stored in one or more memory devices and executing using separate hypervisors, the separate operating system (OS) including one or more separate network support applications, a checkpoint engine, and one or more storage device support applications. In one embodiment, checkpoints are transmitted using the hardware interconnect.

[0010] In part, this disclosure relates to a method for controlling an operating system in real time, which may include receiving, at a primary node and a secondary node, device data from one or more devices in electrical communication with a timeslot-based bus, the bus being operable to receive data according to timeslots of two or more priority levels; generating a checkpoint at the primary node during a first timeslot and transmitting the checkpoint data to the secondary node, the first timeslot being a lower priority timeslot; generating output data at the primary node and the secondary node in response to processing the received device data; and transmitting the output data from the primary node over the bus using a second timeslot, the second timeslot being a higher priority timeslot, both the first node and the second node being in an active mode.

[0011] In one embodiment, the method further includes transmitting data during non-checkpoint portions of lower priority time slots. In one embodiment, the method further includes gating network data using separate network software modules executing on both the primary and secondary nodes such that outgoing data is released from nodes whose state has been carried forward to the current checkpoint. In one embodiment, the method further includes synchronizing transmission of the outgoing data with high priority time slots, the outgoing data being received by the operating system.

[0012] In one embodiment, the method further includes transmitting data during non-checkpoint portions of lower priority time slots. In one embodiment, the method further includes using the primary node to transmit all of its respective outgoing data to the secondary node, preventing such outgoing data from being dropped in response to the occurrence of a failure of the primary node. In one embodiment, the checkpoint data is transmitted using a hardware interconnect.

[0013] Selectively releasing data to be synchronized with time slots having a priority level that matches or is otherwise compatible with that of data to be transmitted using the time slot-based bus is performed in various embodiments of the present disclosure. Additionally, transmissions of data using the hardware interconnect that are processor / computing device intensive, such as one or more stages of a checkpoint, are transmitted during low priority time slots on the time slot-based bus. In various embodiments, the hardware interconnect is a separate communication channel from the time slot-based bus.

[0014] While the present disclosure relates to different aspects and embodiments, it should be understood that the different aspects and embodiments disclosed herein may be integrated, combined, or used together, as combined systems, or in part, as separate components, devices, and systems, as appropriate. Thus, each embodiment disclosed herein may incorporate each of the aspects to varying degrees, as appropriate for a given implementation. For example, the present invention provides the following items. (Item 1) 1. A real-time fault-tolerant computing system, comprising: a first computing device, one or more memory devices; one or more processors; a first network interface operable to receive device data and transmit output data via a timeslot-based bus, the output data being generated from processing the device data; and First real-time checkpoint engine and a first computing device comprising: a second computing device, one or more memory devices; one or more processors; a second network interface operable to receive device data via the timeslot-based bus; A second real-time checkpoint engine and a second computing device comprising: a hardware interconnection operable to enable data exchange between the first computing device and the second computing device; Equipped with a real-time fault-tolerant computing system, wherein checkpoints are generated by the first or second real-time checkpoint engine during lower priority communication time slots allocated on the time slot-based bus to avoid interference with any real-time communications to or from the first and second computing devices; (Item 2) The real-time fault-tolerant computing system described in the preceding item, wherein the first computing device and the second computing device process received device data simultaneously such that if the first computing device suffers a failure, the second computing device continues to process device data, generate the output data, and transmit the output data using the time slot-based bus. (Item 3) 10. The real-time fault-tolerant computing system of claim 9, wherein a checkpoint operation is always postponed until the occurrence of a lower priority communication slot, even if the first computing device and the second computing device generate divergent output data prior to the occurrence of the lower priority communication slot. (Item 4) 10. The real-time fault-tolerant computing system of claim 9, wherein both the first computing device and the second computing device delay the release of output data to one or more devices communicating with the time slot-based bus until after a subsequent checkpoint to avoid releasing divergent output data in the event of a failure. (Item 5) 2. The real-time fault-tolerant computing system of claim 1, wherein the first computing device generates a checkpoint every cycle when output data comprising low-priority data is transmitted using a low-priority time slot. (Item 6) 10. The real-time fault-tolerant computing system of claim 9, wherein the device is a sensor, the output data comprises sensor data, and the sensor data is operable as input for a control system. (Item 7) 10. The real-time fault-tolerant computing system of claim 9, wherein both the first computing device and the second computing device are actively processing device data and generating output data such that if the first computing device experiences one or more failure modes, there is zero time delay for the second processing device to continue processing. (Item 8) 10. The real-time fault-tolerant computing system of claim 9, wherein the duration of one or more time slots is increased by a detection period, the detection period ranging from about 1 microsecond to about 100 microseconds. (Item 9) 10. The real-time fault-tolerant computing system of claim 9, wherein both the first computing device and the second computing device delay the release of output data to devices communicating with the time slot-based bus until after the next checkpoint to avoid releasing inconsistent output data. (Item 10) 2. The real-time fault-tolerant computing system of claim 1, wherein if the first computing device is operating normally after a checkpoint, the first computing device releases any delayed output data. (Item 11) 10. The real-time fault-tolerant computing system of claim 9, wherein the first computing device transmits all of its output data to the second computing device, preventing such output data from being dropped in response to a failure of the first computing device. (Item 12) 2. A real-time fault-tolerant computing system as described in any one of the preceding items, wherein if the first computing device fails, the second computing device takes over the release of the output data of the first computing device that was generated during the last checkpoint. (Item 13) 10. The real-time fault-tolerant computing system of claim 9, wherein only the first computing device handles the release of output data when no failures occur, and in parallel shares such output data with the second computing device. (Item 14) The real-time fault-tolerant computing system of any one of the preceding items, wherein the device data from one or more devices is transmitted to the first computing device and the second computing device via the time slot-based bus. (Item 15) 2. The real-time fault-tolerant computing system of claim 1, wherein the first computing device and the second computing device receive incoming data directly through broadcast or multicast. (Item 16) The real-time fault-tolerant computing system of any one of the preceding items, wherein the first computing device and the second computing device each comprise a separate non-virtual operating system (OS) stored in the one or more memory devices and running on each of the one or more processors, the separate operating systems comprising separate network drivers, checkpoint shims, and storage device drivers. (Item 17) 2. The real-time fault-tolerant computing system of claim 1, wherein the network driver comprises a replicator that communicates with a release coordinator and a release notifier. (Item 18) The real-time fault-tolerant computing system of any one of the preceding items, wherein one or more networking programs stored in one or more memory devices of the first and second computing devices delay and coordinate transmission of output data using one or more of the hardware interconnect, the first network interface, and the second network interface. (Item 19) The real-time fault-tolerant computing system of any one of the preceding items, wherein the first computing device and the second computing device each comprise a separate virtual operating system (OS) stored in the one or more memory devices and executed using a separate hypervisor, the separate operating system comprising one or more separate network support applications, a checkpoint engine, and one or more separate storage device support applications. (Item 20) 1. A method for controlling an operating system in real time, comprising: receiving, at the primary node and the secondary node, device data from one or more devices in electrical communication with a time slot-based bus, the bus operable to receive data according to time slots of two or more priority levels; generating a checkpoint at the primary node during a first time slot and transmitting the checkpoint data to the secondary node, the first time slot being a lower priority time slot; generating output data at the primary node and the secondary node in response to processing the received device data; transmitting the output data from the primary node over the bus using a second time slot, the second time slot being a higher priority time slot, and both the first node and the second node being in an active mode; A method comprising: (Item 21) 10. The method of claim 1, further comprising transmitting data during non-checkpoint portions of lower priority time slots. (Item 22) 10. The method of claim 9, further comprising gating network data using separate network software modules running on both the primary and secondary nodes so that output data is released from the nodes whose state has been carried forward to the current checkpoint. (Item 23) 10. The method of claim 1, further comprising synchronizing transmission of output data with a high priority time slot, the output data being received by the operating system. (Item 24) 10. The method of claim 1, further comprising transmitting data during non-checkpoint portions of lower priority time slots. (Item 25) 10. The method of claim 9, further comprising using the primary node to transmit all of its respective outgoing data to the secondary node to prevent such outgoing data from being dropped in response to a failure of the primary node. (Item 26) 10. The system of claim 9, wherein checkpoints are transmitted using the hardware interconnect. (Item 27) 10. The method of claim 1, further comprising increasing the duration of one or more time slots to allow for detection of a failure of a primary node or a secondary node. (Item 28) 3. The method of claim 1, wherein the duration of one or more time slots is increased by a detection period, the detection period ranging from about 1 microsecond to about 100 microseconds. (Item 29) 10. The method of claim 1, wherein the checkpoint data is transmitted using a hardware interconnect. (Summary) In part, this disclosure relates to a real-time fault-tolerant system. The system may include a first computing device, a second computing device, and a hardware interconnect. The first computing device may include one or more memory devices, one or more processors, a first network interface operable to receive device data and transmit output data via a timeslot-based bus, where the output data is generated from processing the device data, and a first real-time checkpoint engine. The second computing device may include similar or identical components to the first computing device. The hardware interconnect is operable to enable data exchange between the first computing device and the second computing device. Checkpoints may be generated by the checkpoint engine during lower priority communication timeslots allocated on the timeslot-based bus to avoid interference with any real-time communications to or from the first and second computing devices. [Brief explanation of the drawings]

[0015] The structure and function of the present disclosure can be best understood from this description in conjunction with the accompanying drawings. The drawings are not necessarily to scale, emphasis instead generally being placed on illustrative principles. The drawings are to be considered in all respects illustrative and are not intended to limit the invention, the scope of which is defined only by the claims.

[0016] [Figure 1A] FIG. 1A is a schematic diagram of an exemplary cyclic real-time system that may be implemented using the fault-tolerance features disclosed herein.

[0017] [Figure 1B]FIG. 1B is a schematic diagram of various devices communicating with a real-time fault-tolerant system that shares a bus with the various devices, according to an embodiment of the present disclosure.

[0018] [Figure 2] FIG. 2 is a schematic diagram of a real-time fault-tolerant system showing the paths of real-time device data and output data according to an embodiment of the present disclosure.

[0019] [Figure 3] FIG. 3 is a schematic diagram of a real-time fault-tolerant system including a first computing device and a second computing device, according to an embodiment of the present disclosure.

[0020] [Figure 4] FIG. 4 is a schematic diagram illustrating checkpoint coordination for various time slots of a shared bus for use in a real-time fault-tolerant system, according to an embodiment of the present disclosure.

[0021] [Figure 5] FIG. 5 is a schematic diagram showing three cycles of operation for a primary and secondary RT / FT real-time fault-tolerant implementation, where packets corresponding to output data are retained or gated.

[0022] [Figure 6] FIG. 6 is a schematic diagram illustrating three cycles of operation for a primary and secondary RT / FT real-time fault-tolerant implementation illustrating real-time checkpointing, according to an embodiment of the present disclosure.

[0023] [Figure 7] FIG. 7 is a schematic diagram showing three cycles of operation for a primary and secondary RT / FT real-time fault-tolerant implementation.

[0024] [Figure 8]FIG. 8 is a block diagram of a real-time fault-tolerant virtualization-based checkpointing system according to an embodiment of the present disclosure.

[0025] [Figure 9] FIG. 9 is a block diagram of a real-time fault-tolerant operating system-based checkpointing system according to an embodiment of the present disclosure.

[0026] [Figure 10] FIG. 10 is a block diagram of various networking components suitable for use in a virtualization-based real-time fault-tolerant system, according to certain embodiments of the present disclosure.

[0027] [Figure 11] FIG. 11 is a block diagram of various networking components suitable for use in an OS-based real-time fault-tolerant system, according to one embodiment of the present disclosure.

[0028] [Figure 12A] 12A and 12B are schematic diagrams illustrating four cycles of operation for primary and secondary RT / FT real-time fault-tolerant computing device implementations according to an embodiment of the present disclosure. [Figure 12B] 12A and 12B are schematic diagrams illustrating four cycles of operation for primary and secondary RT / FT real-time fault-tolerant computing device implementations according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0029] In part, this disclosure relates to various systems and methods for real-time, fault-tolerant computing. A real-time computer system provides a guaranteed response time, i.e., the time from an input or multiple inputs to an output or response from the computer system is guaranteed by system design not to exceed a predetermined amount. This predetermined amount of time is often referred to as a deadline. Multiple levels of real-time support exist, including hard real-time support and soft real-time support. Various embodiments of the present disclosure implement real-time checkpointing in conjunction with a time-slot-based bus to process device data, such as from sensors and other devices, such as computing devices, generate output data to control actuators, or otherwise generate input for control systems and other systems, devices, and components, as may be applicable for a given use case or implementation.

[0030] In the context of hard real-time support, missing a deadline results in total system failure. Hard real-time support is used when failure to meet a specified deadline could lead to catastrophic consequences. Examples include, but are not limited to, chemical plant processes, nuclear power plant control, railway signal control, avionics fly-by-wire, and automotive engine control. Conversely, in the context of soft real-time support, the usefulness of a result degrades after its deadline, thereby reducing the system's quality of service. Examples include, but are not limited to, computer voice systems, where missing a deadline can cause some degree of audio distortion, computer gaming systems, where missing a deadline can cause some delay in player actions, or telephone switching systems, where missing a deadline can result in a failed call initiation. The various systems and methods disclosed herein may be used for either hard or soft real-time support.

[0031] This disclosure describes a checkpointing-based solution for achieving application-transparent real-time fault-tolerant operation that can run on standard (commercial) computer systems. Use of this system achieves significant cost and time savings compared to purpose-designed solutions using custom software and / or hardware. In part, this disclosure relates to various checkpointing-based systems that provide application-transparent hard real-time fault-tolerant operation that can run on standard (commercial) computer systems. These systems offer various improvements related to checkpointing.

[0032] Traditional checkpointing is virtualization-based. Specifically, it is a means for achieving virtual machine (VM)-level fault-tolerant operation by periodically stopping a VM, capturing its state, and replicating the captured state on a second VM so that the second VM can continue operation if the first VM fails. The process of stopping a VM, capturing its state, and replicating the state to another VM is called a checkpoint. Checkpointing of VMs is called virtualization-based or hypervisor-based checkpointing.

[0033] It is also possible to implement checkpointing at the OS level, i.e., in an OS driver rather than in a hypervisor. The various systems, methods, combinations thereof, and other embodiments described in this disclosure apply equally well to virtualization-based and OS-based checkpointing.

[0034] In general, using time slot prioritization for device data and selecting low priority time slots for checkpointing, along with improving checkpointing-based approaches in fault-tolerant systems, are some of the features used to implement real-time fault-tolerant systems and methods. With the foregoing as a high-level introduction, it is useful to consider checkpointing in more detail.

[0035] Checkpointing is a technique for keeping pairs of virtual machines (VMs) or operating systems (OSs) in a sufficiently identical state so that if one VM or OS fails, the other VM or OS can continue operation of the failed VM or OS essentially transparently to the client system. To achieve nonstop service, also known as fault tolerance (FT), checkpointing deploys redundant VM or OS pairs on separate physical servers so that if one physical server fails, the VMs or OSs and applications continue to run on the remaining physical server.

[0036] Checkpointing is necessitated by the fact that modern multithreaded or multicore computer systems do not run deterministically. Two computers receiving the same input data do not execute the same instruction stream and, therefore, may generate different output data (or output data timing) due to various factors. For example, a hardware interrupt that is not received at exactly the same point in the instruction stream on both computers can ultimately cause processes to be scheduled differently between the two computers. In addition, disk response time differences (rotational latency variations on spinning drives and garbage collection / remapping activity variations on solid-state drives) can ultimately cause processes to be scheduled differently on the two computers. Furthermore, common nondeterministic processor behaviors such as thermal throttling, different cache hit / miss behavior due to early instruction / data access, and imprecise alignment of fine-grained clocks and counters can ultimately cause processes to be scheduled differently on the two computers. Periodic checkpointing returns pairs of (non-deterministic) OSs or VMs to the same valid state, so that if an OS or VM fails, the partners can resume operation with respect to the other transparently to external client systems.

[0037] At a high level, periodic checkpointing stops a VM or OS from running, captures its state, and sends the state (or state changes) to the partner VM or OS, keeping the partner VM or OS in a roughly equivalent state. In the case of virtualization-based checkpointing, the checkpointing functionality is implemented mostly within the hypervisor. In the case of OS-based checkpointing, the checkpointing functionality is implemented mostly within the operating system driver. The time during which the checkpointed VM or OS is running is called the start-up interval, and the time during which the checkpointed VM or OS is stopped to capture state is called a checkpoint. The sum of these two intervals is called an epoch. Exemplary checkpoint epochs and start-up intervals are shown in the context of scheduling data for release on a TSB in Figures 4, 12A, and 12B, which are discussed in more detail herein.

[0038] In the case of OS-based checkpointing, the OS is largely halted during the checkpoint, but the portion of the OS associated with the checkpointing process continues to run during the checkpoint. Various embodiments of the present disclosure may improve upon and / or utilize some of the features disclosed in co-pending application Ser. No. 16 / 900,909, filed Jun. 13, 2020, and entitled "OPERATING SYSTEM-BASED SYSTEMS AND METHOD OF ACHIEVING FAULT TOLERANCE," the entire disclosure of which is incorporated herein by reference. Real-time Overview

[0039] A real-time computer system provides a guaranteed response time, i.e., the time from an input or multiple inputs to an output or response from the computer system is guaranteed by the system design not to exceed a predetermined amount. This predetermined amount of time is often referred to as a deadline. FIG. 1A is a schematic diagram of an exemplary cyclic real-time system 5 that can be implemented using the fault-tolerance features disclosed herein. Many real-time systems operate in a cyclical manner, i.e., inputs are received and outputs are provided at repeated fixed intervals. The fixed intervals are called cycle times (CT). The real-time fault-tolerant system 7a receives sensor inputs, such as from various devices, including sensors, and generates outputs, such as control outputs. The control outputs can be used to control various devices, such as actuators and other controllers. The real-time fault-tolerant system 7a is operable to implement real-time checkpointing using a virtualization approach or an operating system (OS)-based approach.

[0040] In industrial and process control environments, the communication medium connecting the control computer system and sensors, actuators, valves, and other entities is often a shared bus. As shown in FIG. 1B, the overall manufacturing system 12 has various devices that communicate with the shared bus. The shown shared bus is implemented using a time slot-based bus (TSB). The various devices D1-D7 correspond to a temperature sensor (D1), a gauge (D2), a valve (D3), a motor (D4), a robot arm (D5), a conveyor belt (D6), and an alarm (D7). Devices D1 and D2, i.e., the temperature sensor and gauge, may act as sensors that provide sensor input to the RT / FT system 7b. As shown, system 7b typically includes a first computing device 8a and a second computing device 8b, which are connected to each other and to the TSB. Valve D3, motor D4, robot arm D5, conveyor belt D6, and alarm D7 may be controlled based on control outputs (output data) generated by system 7b in response to sensor inputs from devices D1 and D2. In turn, alarm D7 also serves as an input and output, and the alarm state of device D7 may be fed back to system 7b to adjust other devices or to provide a continuous indication that everything is operating according to acceptable levels.

[0041] Many real-time control systems use a shared communication medium, such as Ethernet, for the TSB embodiment shown in FIG. 1B. For simplicity of presentation, the TSB in FIG. 1B is illustrated as directly connecting to devices (D1-D7), but in practice, there is often an intervening electronic component, typically a programmable logic controller (PLC), between the device and the fieldbus. Various edge computing devices may also be connected to the TSB in various embodiments. The real-time fault-tolerant system 7b is operable to implement real-time checkpointing using a virtualization approach or an operating system (OS)-based approach. Computing devices, such as the first computing device 8a and the second computing device 8b, include network connections such that both devices receive device data from one or more (or all) of devices D1-D7 through the TSB.

[0042] To maintain real-time behavior with a shared bus, processes are implemented to prevent various computing devices accessing the bus from communicating on the shared bus simultaneously. In part, this disclosure relates to protocols and other safeguards for preventing simultaneous access to the shared bus or otherwise transmitting data, such as checkpoint data, during high-priority time slots. Failure to implement such safeguards can result in non-deterministic communication delays, which in turn can cause real-time deadlines to be violated.

[0043] In various embodiments, avoiding communication contention on a shared medium, such as a shared bus, is accomplished by time-slicing the communication medium. In various embodiments, the real-time fault-tolerant system disclosed herein allocates pre-assigned time slots within a processing loop cycle or other period or cycle for each communicating entity on the shared bus. Examples of time-sliced ​​communication buses include, but are not limited to, ProfiNET IRT, CC-Link IE Field, Foundation Fieldbus, and Ethernet POWERLINK. Various TSBs are operable to transmit and receive data according to time slots having various priority levels, such as high, medium, and low priority. Other, more general priority levels may be used to manage how device data is scheduled relative to checkpoint data. A first priority level, a second priority level, a third priority level, etc. may be used to distinguish time slots, facilitate real-time checkpointing, and avoid contention that could result in errors that disrupt real-time operation.

[0044] In various implementations, a time-slot-based bus is used to implement various real-time, fault-tolerant systems that collect data from sensors or other devices, with the ability to allocate specific time slots for time-critical communications and allocate other time slots for non- or less time-critical communications. This allows for the use of a shared communications medium, such as a fieldbus, for non-deterministic communications without interfering with time-critical communications on the same shared bus, such as a TSB.

[0045] For some TSBs, low priority time-sliced ​​communication bus slots may be characterized as non-cyclic, meaning that communication between those slots is not constrained by deadlines dictated by cycle times. Various shared buses suitable for implementing embodiments of the present disclosure may include combinations of time slots, such as time slots with cyclic and non-cyclic characteristics and various levels of priority. Scheduled Traffic in Time-Sensitive Networking (TSN)

[0046] As an example, it is instructive to consider TSN (Time Sensitive Network) technologies that may be used to implement various TSB configurations. TSN supports several methodologies for achieving bounded low-latency network data transport. These are currently incorporated into IEEE 802.1Q-2018. Examples of TSN features may include one or more of the following: credit-based shaper (formerly 802.1Qav), preemption (formerly 802.3br and 802.1Qbu), scheduled traffic (formerly 802.1Qbv), circular queuing and forwarding (formerly 802.1Qch), and asynchronous shaping (formerly P802.1Qcr).

[0047] Various embodiments of the present disclosure may use a suitable TSB, such as TSN scheduled traffic (IEEE 802.1Qbv, also known as TSN time-aware shaper, also known as TSN time-aware scheduler), to demonstrate real-time behavior. TSN is an example of a suitable TSB for use in various embodiments.

[0048] The TSN time-aware scheduler is a time-sliced ​​cyclic bus based on Ethernet capabilities. TSN is an Ethernet-based, non-proprietary standard. Reference to using TSN with respect to a shared bus for the real-time systems and methods disclosed herein is provided as an illustrative example. The present disclosure is not specific to TSN-based networks and may work with any time-sliced ​​cyclic bus, as well as other acyclic real-time solutions in which periodic time slots exist in which checkpoints can be performed without interfering with real-time activity. In general, embodiments of the present disclosure may be implemented using, for example, a TSN-based bus or network and a time-slot-based bus (TSB), such as those disclosed herein or others suitable for providing real-time functionality as disclosed herein. Checkpointing and real-time behavior

[0049] In various aspects, the present disclosure relates to real-time checkpointing and its role in supporting real-time fault tolerance. Existing traditional checkpointing solutions can be grouped into two categories: active-passive checkpointing and active-active checkpointing. These existing traditional checkpointing techniques are described in more detail below. Real-time checkpointing is a different type of checkpointing from these traditional techniques. Real-time checkpointing addresses the shortcomings of active-passive or active-active checkpointing that make them unsuitable for real-time operation. Real-time checkpointing is a different type of checkpointing from active-passive or active-active checkpointing. Real-time checkpointing addresses the shortcomings of active-passive or active-active checkpointing and is therefore suitable for real-time operation.

[0050] A high-level illustrative example of a real-time system 7c is shown with a TSB as part of an overall control system 20. In that example, shown in Figure 2, sensors 1 and 2, i.e., S1, S2, provide real-time (RT) data to a real-time fault-tolerant (RT / FT) checkpointing-based system, labeled RT / FT, which processes the data to control actuator A in real time.

[0051] The illustrated sensors S1, S2 can be various devices (pressure sensors, flow sensors, speed sensors, fluid level sensors, proximity sensors, etc.) and have various implementations. For example, the various devices D1-D6 in system 5 of FIG. 1B may include various sensors and other devices. Various sensors and devices suitable for use with the systems and methods disclosed herein may include, but are not limited to, smart networked sensors, networked PLCs (programmable logic controllers) connected to a sensor or multiple sensors, computers hosting virtualized PLCs connected to sensors, distributed control nodes (industrial control devices with DCN, IO, and computing capabilities), combinations thereof, and others as disclosed herein or otherwise applicable to a given company's products and services. Similarly, actuators can be various device types and have several possible implementations.

[0052] If the sensors are position, proximity, and / or pressure devices associated with a robot arm and the actuators represent the servo motors of the robot arm that control its movement, the sensors provide data to the RT / FT system in real time, which is processed by the RT / FT to control the movement of the robot arm via the servo motors of the arm.

[0053] Still referring to FIG. 2 , two exemplary devices S1 and S2 are shown along with actuator A. In various embodiments, devices S1 and S2 are sensors that collect data for a system or process being monitored or controlled by control system 20. Real-time device data from the sensors is processed by RT / FT 7c for one or more processing cycles and then relayed to the actuator as real-time control information. A given RT / FT may include a network interface for receiving device data (such as sensor data) and transmitting output data. The output data is generated from processing the received device data and, optionally, other inputs and parameters that are specific to the process or system being controlled or monitored. The device data and output data are typically transmitted using a TSB. In the system of FIG. 2 , RT / FT processes data from S1 and S2 and generates output data, such as real-time control information, that will be transmitted to actuator A, causing it to respond in a particular manner, perform a given task, or take a particular action in response to the received output data. RT / FT 7c is simplified in terms of processing details and how one or more computing devices are configured to provide fault tolerance and real-time implementation. Figure 3 shows an example RT / FT with further details and includes two computing devices.

[0054] The RT / FT system 7d of FIG. 3 is a primary and secondary (or vice versa) or RT / FT 1 and RT / FT 2The RT / FT system 7d includes two computing devices, which may be identified as a primary computing device (S1), a secondary computing device (S2), and a secondary computing device (A). The RT / FT system 7d is part of an overall control system 30 that includes TSB, S1, S2, and A. Devices / sensors S1 and S2 and actuator A are identical to those depicted in the system of FIG. 2, but additional details of their connections to the primary and secondary computing devices are shown. The two computing devices work in conjunction to implement a fault-tolerant real-time checkpointing-based system 7d. These computing devices may also be identified as first and second computing devices, servers, nodes, or other terms as disclosed herein. Notwithstanding the foregoing, a given first computing device may initially have a first role, while the second computing device may have a second role. The first and second roles may be switched or changed to other roles if one of the computing devices suffers an error or failure mode.

[0055] The system of Figure 3 includes a shared bus, such as a TSB, and may also include a network connection to a network, such as a corporate information technology (IT) network, the Internet, or other network. 1 and RT / FT 2 are each connected by a hardware interconnect HIC. The sensors, actuators, and RT / FT systems are connected using a redundant shared network 17, represented by thin dark lines. Exemplary data flows are illustrated by thick light lines 18. The various arrows indicate the direction of data flow.

[0056] For the non-fault case, which corresponds to normal operation without a failure in 7d or one of the two computing devices, the data flow is as follows: sensor / device data is received and processed by both RT / FT nodes (so if one node fails, no processing time or state is lost). 1 However, if it breaks down, RT / FT 2 would receive the same sensor / device data and generate output data for the control system 30 and components requiring real-time output to maintain real-time operation, such as actuator A. In one embodiment, the primary node (RT / FT in the diagram above) 2 ) provides control output / output data to actuator A. This is an example requirement. In other instances, a secondary node (RT / FT in the diagram above) 1 ) provides control output / output data to actuator A. In another embodiment, both the primary and secondary computing devices generate output data and transmit the output data redundantly, and the actuator may be operable in some embodiments to be able to process redundant signals and ignore redundant inputs. Checkpointing

[0057] In various embodiments, checkpoints are coordinated with the TSB so that checkpoints overlap or are synchronized for release with only the TSB's lower priority (non-real-time) communication time slots. Coordinating and scheduling checkpoints and checkpoint data using lower priority time slots prevents checkpoint operations from interfering with real-time communications. For example, if checkpoint data were to compete with output data needed by an actuator to maintain the rate of fluid flow in a chemical engineering plant, it could be disastrous if the checkpoint data were sent using the time slot needed to control the actuator. FIG. 4 is a schematic diagram illustrating various scheduling interactions for an exemplary TSB's time slots and checkpointing operations. In general, FIG. 4 illustrates an exemplary approach to coordinating checkpoints with the TSB in an RT / FT system.

[0058] Specifically, as shown in FIG. 4, the RT / FT system processes device data from sensors and generates output data for actuators every cycle. Two processing cycles of RT / FT are shown as TSB cycle 1 and TSB cycle 2. A given TSB cycle includes N time slots. In the example shown for each cycle, four time slots are shown, so N=4. The durations of the time slots may be the same, or they may differ. As shown, time slot 4 is allocated for low-priority traffic and has a longer duration than time slots 1-3. Each individual cycle spans two parallel data generation events. These events include checkpoint scheduling and TSB network scheduling. As mentioned above, the time during which the checkpointed VM or OS is running is called the wake-up interval, and the time during which the checkpointed VM or OS is stopped to capture state is called a checkpoint. The sum of these two intervals is called an epoch.

[0059] First and second checkpoint epochs with their own separate wake-up intervals and checkpoints where the VM or OS is halted during time slot 4 are shown in FIG. 4. The first epoch corresponds to TSB cycle 1, and the second epoch corresponds to TSB cycle 2. These separate time periods may be different in some embodiments. As shown, after the wake-up interval for a given computing device of RT / FT, a checkpoint is released to overlap or synchronize with a time slot, such as time slot 4 in the example shown for each epoch. Fixed-cycle checkpoints are scheduled to occur during TSBs (or TSNs as shown in this example) between low-priority time slots on the TSB to avoid interference with real-time traffic. Various examples of real-time traffic are shown for time slots 1, 2, and 3 for each TSB cycle.

[0060] Another element of this disclosure is that both RT / FT computing devices are active. In contrast, in some fault-tolerant systems, one computing device is passive and requires time to effectively take over from its paired active computing device. By having both nodes active, no processing time or state is lost when / if the primary fails, and thus real-time deadlines can be maintained even when the primary node fails. Additionally, to support real-time fault tolerance, network packets are gated until after the next checkpoint. Figure 5 is a schematic diagram showing three cycles of operation for a primary and secondary RT / FT real-time fault-tolerant implementation in which packets corresponding to output data are held or gated. Figure 5 provides further detail on the embodiment of Figure 4, in which the computing devices (primary RT / FT and secondary RT / FT) each require time slot management over multiple processing cycles (cycle 1, cycle 2, and cycle 3 are shown, although any number of cycles is possible). In turn, during each cycle, a wake-up interval and a checkpoint also occur. Thus, in various embodiments, for N cycles, there are N epochs in the absence of a fault or some other event.

[0061] As shown in FIG. 5, network packets are gated or held during wake-up intervals because two nodes may differ. During each wake-up interval, outgoing network packets that would otherwise be transmitted via the TSB are held. If a failure were to occur as part of RT / FT and the packets were released immediately, packets previously released by the now-failed node may not be in a consistent state with packets released by the previously secondary, now-primary node after the failure. By gating packets, RT / FT ensures that all released packets were generated by nodes whose state has been carried over from the previous epoch. In various embodiments, packet gating may also be referred to as packet delaying. Gated packets are released during the correct TSN time slot. If this is not followed, a given real-time system may not be fault-tolerant and may incur errors, which in turn may disrupt real-time operation. Outgoing packets from the previous wake-up interval are released after a checkpoint occurs during a low-priority time slot.

[0062] Gated outgoing network packets from the primary RT / FT are sent to the secondary RT / FT. The secondary RT / FT generally operates to hold outgoing packets received from the primary and discards those packets if the primary does not fail during the cycle in which the primary released its copy of the corresponding packet. In a non-fault cycle, the secondary RT / FT discards its state from that cycle and continues from the last primary checkpoint. Additionally, in the non-fault case, packets in the secondary RT / FT are discarded upon checkpoint completion. The above points discussed with respect to Figure 5 can be more clearly illustrated by examining the fault case.

[0063] Figure 6 is a schematic diagram showing three cycles of operation for the primary and secondary RT / FT real-time fault-tolerant implementation in case of a failure. As in Figure 5, there are three cycles divided into time slots for scheduling device data and output data to available time slots on the TSB. In the first cycle, cycle 1, there is no failure, and the primary and secondary RT / FT schedule device data and output data (for actuators) in time slots 1, 2, and 3, respectively, and low priority traffic corresponding to the checkpoint is handled on time slot 4.

[0064] As shown, prior to cycle 1, gated network packets from the previous cycle are sent from the primary to the secondary. During the wake-up interval, which is aligned with part of cycle 1, the secondary is waking up independently from the primary following the last checkpoint. During cycle 1, gating packets from the primary are sent to the secondary. In the non-fault case, the secondary discards its state and continues from the last primary checkpoint.

[0065] As shown in Figure 6, in cycle 2 there is a failure event in the primary RT / FT, and cycle 3 shows the details after the failure. When the primary fails during cycle 2, the secondary has gated network packets from the last checkpoint to release. In addition, the secondary is running based on a state consistent with the released packets. As a result, the secondary's execution state is valid, and no execution time is lost. In effect, the secondary can immediately continue from where the primary stopped without losing processing time. While no checkpointing is required after the failure, in some preferred embodiments it is desirable to gate packets as if checkpointing had occurred. In some non-preferred embodiments where packets are not gated as if checkpointing were still occurring, after the failure the secondary may attempt to transmit as many packets as twice as many between the critical (high priority) time slots from the last checkpoint and those from the current checkpoint. In other non-preferred embodiments, where packets are not gated as if checkpointing were still occurring, after a failure the secondary may miss sending packets during the cycle when the primary returns to operation, which may violate the real-time requirements of the overall RT / FT. In cycle 3, after a failure of the primary, the secondary continues to boot from its state and, in preferred embodiments, releases its network packets consistent with its processing state in the following cycle. Networking features and management of outgoing packets

[0066] Dropping outgoing network packets is not acceptable for true real-time operation because the time penalty for higher-level protocol recovery would violate real-time deadline obligations. Furthermore, many real-time buses assume reliable transmission for critical packets and do not have higher-level recovery protocols. For example, TSN is at the Ethernet level, and with TSN, reliable Ethernet transmission can be achieved for critical packets because all collisions can be avoided.

[0067] FIG. 7 is a schematic diagram showing three cycles of operation for a primary and secondary RT / FT real-time fault-tolerant implementation. As illustrated in FIG. 7 below, a real-time checkpointing solution may drop outgoing packets in response to a failover if the mechanism described below is not incorporated into the present invention. Specifically, packets from the previous cycle (labeled cycle 1) are queued by the primary RT / FT for release in the following cycle, labeled cycle 2. If the primary RT / FT fails during cycle 2, not all packets can be released. As shown in FIG. 7, during the primary failure in cycle 3, packet 40 is not transmitted because the active node fails before it can release all of the packets from the last epoch.

[0068] In part, this disclosure also relates to eliminating the possibility of outgoing packets being dropped upon failover. This may be achieved by the primary RT / FT sending its outgoing packets to the secondary either during a wake-up interval or at checkpoint time. Then, depending on system constraints, one of two algorithms or methods can be applied to ensure that no outgoing packets are dropped when a failure occurs. The first algorithm operates such that if the system can tolerate a large number of duplicate packets, both the primary and secondary RT / FT may unconditionally release queued packets outgoing during their assigned time slots. The second algorithm provides that if the system cannot tolerate a large number of duplicate packets, the primary may provide an ongoing indication of released packets to the secondary. If the primary fails, the secondary will resume releasing packets from where the primary stopped prior to the failure. Generally, packets are released during their assigned time slots.

[0069] Also, note that with respect to the second algorithm above, consideration is given to the fact that there may be a delay in the primary's release of the packet relative to providing an indication to the secondary that the packet has been released. One solution is to delay the indication of the packet's release to the secondary by an amount sufficient to ensure that the packet has actually been released until the time notification is received by the secondary. This creates a timing window where duplicate packets are possible, and specifically, in a failover, a packet or portion of a packet released by the primary may be re-released by the secondary. Incoming packets

[0070] With real-time checkpointing, both nodes are operable to receive incoming packets because both nodes are actively running OS and application workloads. This contrasts with an active-active solution that may operate with a primary VM receiving all incoming packets and forwarding them to the secondary so that they are received by both nodes. Such an active-active approach may result in dropped incoming packets. For example, packets intended to be received between the time the primary fails and the time network packets are being rerouted to the secondary in response to it becoming the primary would be dropped. Dropped packets are not acceptable for true real-time operation, and therefore, an active-active implementation is not possible for real-time implementations. Furthermore, in various embodiments to avoid dropped incoming packets in response to failover, an external sender of real-time network packets may use multicast or broadcast so that packets are directly received by both nodes. Overhead Considerations - Checkpoint Interval and Network Gating

[0071] Checkpointing introduces overhead associated with periodically pausing the OS or guest VM. There are limits to how quickly this can be done. Furthermore, input-to-output latency is affected by the need to gate outgoing packets across the checkpoint. Depending on how network time slots are allocated for checkpoints, this added latency can be as short as the worst-case checkpoint interval or as long as an entire checkpoint cycle. Fault Detection

[0072] In some embodiments, the time slot duration or period may be increased to account for failure detection time. For example, if approximately 100 μs worth of data can be transmitted during a time slot, the time allocated to that time slot is increased by the detection time DT. The detection time DT allows for the time it takes for the secondary to detect that the primary computing device has failed and to resume transmitting outgoing data. Without the inclusion of the detection time DT, during a failure case, not all outgoing data would be able to be transmitted during the time slot. In various embodiments, the detection time DT ranges from approximately 0.1 μs to approximately 100 μs. In various embodiments, the detection time DT ranges from approximately 1 μs to approximately 500 μs. Virtualization-based checkpointing

[0073] FIG. 8 is a block diagram of a real-time fault-tolerant virtualization-based checkpointing system according to an embodiment of the present disclosure. As shown in FIG. 8, a virtualization-based real-time FT checkpointing solution is implemented in a fault-tolerant system 70. As shown, real-time FT checkpointing support is implemented through modifications and additions within the hypervisor. For example, if a virtualization-based real-time checkpointing system were implemented on Linux, it could be implemented through extensions and / or modifications to KVM / QEMU (a popular Linux hypervisor). Various configurations of virtual machines (VMs) may be used. Various components that support real-time checkpointing are shaded. Additionally, various components or paths that are inactive are shown with dotted lines.

[0074] System 70 includes Server 1 and Server 2. The servers may be any suitable computing devices. In one embodiment, the servers are commercial or custom servers running an OS that supports virtualization. While the illustration shows a single hypervisor / VM (a single real-time FT hypervisor / VM pair) on each server, various other configurations are possible. For example, system 70 may include multiple hypervisors / VMs on each server, and thus multiple real-time FT hypervisor / VM pairs, non-real-time hypervisor / VM pairs, and unpaired (unprotected) hypervisors / VMs. System 70 includes a primary VM and a secondary VM associated with Server 1 and Server 2, respectively, as shown. The primary VM communicates with the primary hypervisor. Similarly, the secondary VM communicates with the secondary hypervisor. Generally, the components of Server 1 are mirrored in Server 2 in terms of the functional blocks and software components shown for the hypervisor, checkpoint engine, storage support, and network support features shown. In various embodiments, all of the foregoing is implemented using a combination of software modules and software programs.

[0075] Server 1 and Server 2 are connected or linked by a hardware interconnect. In one embodiment, the hardware interconnect is a physical medium, such as a hardware bus or bridge, through which checkpointing state is communicated between the two physical servers. The hardware interconnect may be a network connection, a non-transparent PCI bridge, or any other high-performance computer interconnect. In various embodiments, checkpoints are communicated using the hardware interconnect during the occurrence of low-priority time slots that are active time slots on the TSB. In one embodiment, checkpoint data may be communicated on the hardware interconnect during every time slot (although transmission of the final state on the hardware interconnect occurs during the low-priority time slots here). The “final phase” of the checkpoint is the most processor / system-intensive operation, while other phases of the checkpoint do not affect real-time functionality. In various embodiments, one or more phases of the checkpoint, such as the final phase, impact performance because customer applications or VMs are not running at that time. As a result, the goal is to communicate the checkpoint data as quickly as possible, which has an associated performance cost.

[0076] As used herein, such performance costs can adversely affect real-time performance. The connection of the hardware interconnect to the various components of each computing device can vary depending on the type of hardware interconnect used. For example, if the interconnect is a non-transparent PCI bridge, it does not go through a network as shown at the bottom of FIG. 8. If the hardware interconnect is implemented using a network connection, it may connect to the network shown here at the bottom of FIG. 8 and connect from a networking interface associated with OS networking support. Selectively transmitting checkpoint data that adversely affects real-time performance during low-priority transmissions on the TSB avoids additional burden on a given server, which promotes real-time performance.

[0077] Server 1 and Server 2 each include a checkpoint engine. The checkpoint engines and all of the components depicted in the hypervisors for Server 1 and Server 2 are additions or modifications to each respective hypervisor. The checkpoint engines initiate periodic checkpoints as determined by TSB timing. The purpose of the checkpoint process is to update the state of the secondary VM to match the state of the primary VM so that if the hardware running the primary VM were to fail, the secondary node can take over without leaving the application inoperable.

[0078] The steps of the checkpoint process include one or more of the following: Stopping the OS running on both the primary and secondary VMs. Notifying storage devices and network components that a checkpoint is being initiated. Obtain a list of VM memory that has been modified since the last checkpoint by both the primary and secondary VMs. Compute the union of these two lists. · Transmitting the modified VM memory from the primary hypervisor to the secondary hypervisor via a hardware interface using a union of the modified memory lists. On the primary VM, capturing VM processor and device state, transmitting the processor and device state from the primary hypervisor to the secondary hypervisor via a hardware interface. Informing network support and storage support components on both the primary and secondary hypervisors that the checkpoint process is complete and the VM is about to resume. Resuming VM execution on both the primary and secondary nodes.

[0079] It is important that the system and time slots are appropriately sized to support real-time checkpointing. Time slots may be sized according to the application in which the real-time fault-tolerant system is being used and the cycle time specified by the user. During the checkpoint process, if either the primary or secondary checkpoint engine detects that the checkpoint duration exceeds the allotted time and therefore potentially interferes with real-time deadline obligations, the checkpoint will be aborted and the system will revert to a simplex state to ensure that real-time deadlines are met. Reverting to a simplex state refers to when the secondary and primary stop checkpointing and the primary starts alone (and unprotected because it is no longer checkpointing). When this occurs, one server will repartner with the other server to restore FT operation. Similarly, if either the primary or secondary checkpoint engine detects the need to perform a checkpoint outside of the time slot specified by the TSB timing, the system will revert to a simplex state to ensure that real-time deadlines are met. If this occurs, the system will repartner with itself to restore FT operation. Examples of items that can cause unscheduled checkpoints are an improperly sized system where the buffers associated with memory tracking are too small to record all memory changes that occur between checkpoints, or where the storage cache is too small to cache all storage changes between checkpoints.

[0080] Within the checkpoint engine, there are two subcomponents: a memory tracker and a multi-faceted device state replication component. The memory tracker utilizes processor hardware features to track memory modified by VMs, so the checkpointing engine knows what memory needs to be copied at the checkpoint. In turn, the multi-faceted device state replication component captures the VM device state in the primary hypervisor and applies it to the VM in the secondary hypervisor as part of the checkpoint.

[0081] The checkpoint engine in Server 2's secondary hypervisor waits until the complete checkpoint data, including dirty page data and general device state data, is received, then applies that data and resumes VM operation in the context of the new checkpoint. Storage Support Components

[0082] The storage support component interfaces with the hypervisor storage functionality to support checkpointing. Specific checkpointing-related components are the caching component and the replication component. The caching component caches storage writes that occur between each checkpoint from both the primary and secondary VMs. The primary and secondary VMs are non-deterministic when they boot. As a result, these VMs may not write identical data to storage. For example, if the VMs process transactions in a different order due to non-determinism, the timestamp for a transaction written to storage on the primary may differ from the timestamp written to storage on the secondary. This storage interaction may be referred to as a view of storage. The view is in cache until a checkpoint, after which (in a non-failure case) the primary's data / view is written to disk and the secondary's data / view is discarded. Within a boot interval, the VMs' views of storage are allowed to differ between VMs because each VM operates independently and potentially issues different write requests. Following each checkpoint, the secondary VM's storage is made the same as the primary VM's storage. Caching writes instead of providing them to the OS to be committed to the media provides certain advantages. For example, one advantage is that the secondary cache can simply be discarded following a checkpoint. Also, in the case of a primary node failure and eventual recovery from the secondary, committing primary writes to the media before performing a checkpoint becomes problematic, in which case the primary cache would be discarded as well.

[0083] In addition, replication components are also used to support various storage functionalities. The storage components support replicating storage between primary and secondary VMs. At the start of each startup interval, the VM storage is the same. This requires an initial full synchronization operation before operation begins. Then, in cooperation with the caching component of the storage support component, storage modifications are replicated from the primary hypervisor to the secondary hypervisor as each checkpoint is performed and are delivered to the secondary storage media via the OS. Each storage component may be implemented as one or more software programs, methods, or routines. Network Support Components

[0084] Network support components interface with the hypervisor network functionality to support checkpointing and real-time operations. These components include a replicator, a local outgoing packet queue, a release coordinator, a partner outgoing packet queue, and a release notifier. Each of these may be implemented as one or more software programs, methods, or routines.

[0085] On the primary hypervisor, the replicator is operable to take outgoing network packets from the primary VM and send them to the primary's local outgoing packet queue and the secondary's partner outgoing packet queue. On the secondary hypervisor, the replicator is inactive and simply serves as a conduit for packets from the secondary guests to the secondary's local outgoing packet queue. Along with the packets, the replicator passes to it information identifying the TSB time slot to which each packet is assigned. The local outgoing packet queue is operable to queue packets from the local VM instance for possible later release by the release coordinator. Along with the packets, information identifying the TSB time slot to which each packet is assigned is queued. The partner outgoing packet queue component is operable to queue packets from the partner VM instance for possible later release by the release coordinator. It is active only on the secondary hypervisor.

[0086] Two alternatives for releasing outgoing packets are discussed above. These include: a) If the system can tolerate a large number of duplicate packets, both the primary and secondary RT / FTs can unconditionally release queued outgoing packets. b) If the system cannot tolerate a large number of duplicate packets, the primary only releases queued outgoing packets in the non-fault case. If alternative b is used, the partner outgoing packet queue (active only on the secondary) receives notification from the primary release notifier when a packet is released from the primary. The corresponding packet is then removed from the partner outgoing packet queue.

[0087] The release coordinator is operable to release packets from the appropriate queue at the appropriate time to the OS, and therefore to the external network. In all cases, packets are released during their assigned TSN time slots.

[0088] In the non-fault case, packets from the previous epoch are released from either: a) If alternative a from above is used, both the local outgoing packet queue of the primary hypervisor and the partner outgoing packet queue of the secondary hypervisor; or b) If alternative b above is used, only the primary hypervisor's local outgoing packet queue. If alternative b from above is used, the primary release coordinator notifies the primary release notifier of each released packet. After a delay to ensure that the packet was actually released, the release notifier passes the notification onto the secondary partner outgoing packet queue. If alternative b from above is used and the primary hypervisor or VM fails, the secondary release coordinator releases the remaining packets in the secondary partner outgoing packet queue.

[0089] The release notifier is used only in option b above and is operable to notify the secondary of any packets released by the primary. The primary function of the release notifier is to add sufficient delay to the notification to the secondary to ensure that the packet was actually released by the primary. This is to avoid any dropped packets after failover due to the previous secondary considering the packet to have been released when it was not. OS-based checkpointing

[0090] FIG. 9 is an OS-based block diagram for implementing real-time fault tolerance. OS-based checkpointing is described, which supports real-time operation using various components as depicted and discussed with respect to FIG. 9. A primary node / primary server running a primary OS is connected by a hardware interconnect to a secondary node / secondary server running a secondary OS. As shown, OS-based real-time FT checkpointing support is implemented through modifications and additions within or below the operating system. For example, a Linux® OS-based checkpointing solution would be implemented largely within the Linux® storage driver, Linux® network driver, and Linux® checkpoint driver. System 75 in FIG. 9 is comparable to the previously discussed virtualization-based real-time FT checkpointing of FIG. 8, where support is implemented through hypervisor additions or modifications. Various components that support real-time checkpointing are shaded. Additionally, various components or paths that are inactive are indicated by dotted lines.

[0091] Conceptually, many of the components in Figure 9 perform the same functions as the similarly named components in Figure 8. Differences exist that arise from a virtualization approach versus an operating system-based approach. Because OS-based checkpointing synchronizes systems at the machine level (as opposed to the VM level), there are stricter constraints on the actual hardware for OS-based RT / FT. The primary and secondary nodes / servers are commercial or custom servers that meet one or more of the following requirements: · The servers are configured with the same hardware configuration. The server contains hardware supported by checkpointing network drivers, checkpointing storage drivers, and a checkpointing shim for the target OS. Also, unlike virtualization-based checkpointing, where multiple VMs can be checkpointed and both physical servers can therefore host primary and secondary VMs simultaneously, OS-based checkpointing has a primary server and a secondary server.

[0092] The server OS is protected by and contains checkpointing technology. The server OS can be a standard OS (Windows, Linux, and / or VMware), but various OS-specific driver additions and modifications are required to support checkpointing functionality.

[0093] In one embodiment, the hardware interconnect is a physical medium, such as a hardware bus or bridge, through which checkpointing state is communicated between two physical servers. The hardware interconnect can be a network connection, a non-transparent PCI bridge, or any other high-performance computer interconnect. In various embodiments, the hardware interconnect is separate from the TSB and does not connect to it. Software support for the hardware interconnect is in the checkpoint engine driver. Native OS support for the hardware interface cannot be used because checkpointing operations need to be independent of the checkpointing OS (e.g., checkpoint activity needs to continue even when the host OS is generally paused for a checkpoint). This means, for example, that if TCP / IP is used over a NIC to communicate with the checkpoint driver, a TCP / IP stack that is distinctly different from the TCP / IP stack of the host OS is implemented in the checkpoint driver.

[0094] 9, the checkpoint engine initiates periodic checkpoints as determined by TSB timing. The purpose of the checkpoint process is to update the state of the secondary node to match that of the primary node so that if the primary node were to fail, the secondary node can take over without leaving the application inoperable.

[0095] The steps in the checkpoint process are: Informing network and storage drivers that a checkpoint is imminent. The checkpoint is now suspended until the storage driver performs a callback indicating that outstanding storage accesses have been completed. Stopping the OS running on both the primary and secondary nodes. Obtain the lists of memory modified since the last checkpoint on both the primary and secondary nodes. Compute the union of these two lists. · transmitting the modified memory from the primary node to the secondary node via a hardware interface using the combined modified memory list; On the primary node, capturing processor and device state. · transmitting processor and device states from the primary node to the secondary node via a hardware interface; On both the primary and secondary nodes, notifying the network and storage drivers that the checkpoint process is complete and that OM is about to resume. Restarting the OS running on both the primary and secondary nodes.

[0096] In various implementations of OS-based RT / FT, the system and time slots should be sized appropriately to support real-time checkpointing. During the checkpoint process, if either the primary or secondary checkpoint driver detects that the checkpoint duration will exceed the allocated time on the TSB for low-priority communication (and thus potentially interfere with real-time deadline obligations), the checkpoint will be aborted and the system will be returned to simplex state to ensure that real-time deadlines are met. When this occurs, the system will repartner with itself to restore FT operation.

[0097] Similarly, if either the primary or secondary checkpoint driver detects the need to perform a checkpoint outside of the time slots defined by the TSB timing, the system will revert to a simplex state to ensure that real-time deadlines are met. If this occurs, the system will repartner with itself to restore FT operation. Examples of items that can cause unscheduled checkpoints are that the buffers associated with memory tracking are too small to record all memory changes that occur between checkpoints, and that the storage cache is too small to cache all storage changes between checkpoints.

[0098] The checkpoint shim is transparent to the OS and is a container for a memory tracker and multi-faceted device state replication. This checkpoint shim component tracks memory modified by the server OS so the checkpointing engine knows what memory needs to be copied at checkpoint. In one embodiment, the checkpoint shim utilizes Intel's PML (Page Modification Logging) hardware feature to track memory. In various embodiments, other hardware-specific features, such as motherboard- or processor- or other chipset-specific instructions and features, may be used to track memory. The VT-x hardware virtualization extensions are enabled when using PML under certain scenarios. When VT-x is enabled, the hardware reserves a contiguous area of ​​physical memory for virtual machine monitor (VMM) code to operate on. In a typical virtual machine environment, VMM code is invoked when a virtualized guest operating system executes an instruction that requires special handling, such as an interrupt, a memory fault, an IO access, certain privileged instructions, or an indication that a PML-generated list containing modified memory locations needs to be inspected.

[0099] The memory tracker installs a micro-VMM shim (a minimal hypervisor) whose primary functions are to handle PML FULL vmexit events and maintain dirty page buffers in a reserved memory region. The VMM also handles several other vmexit events from the OS, primarily those vmexits that cannot be disabled in hardware while using PML. These vmexit events are emulated within the micro-VMM so that the server will continue to operate in its normal configuration while the VMM is present.

[0100] When a system workload generates enough dirty pages to fill the hardware PML log, a PML FULL vmexit occurs, a new set of pages is tracked, and workload processing will continue. This allows the memory tracker to generate a list of physical memory pages that have been modified on each server since the previous checkpoint. Memory tracking embodiments track memory modifications at the page level; other embodiments may track memory at a different granularity.

[0101] Multi-way device state replication is a checkpoint shim component that captures device state not captured by the OS-resident component from one node / server and applies it to the other node / server as part of a checkpoint.

[0102] The checkpoint engine on the primary server reads or extracts certain device and processor state data, such as timestamp counters, processor registers, stacks, and other data, and issues a request to the VMM to record that miscellaneous device state data in a reserved memory area. This data is communicated to the secondary server.

[0103] The checkpoint engine on the secondary side requests the VMM to wait until the complete checkpoint data, including dirty page data and general device state data, is received, and then the VMM applies that data and resumes the OS in the context of the new checkpoint. Storage Drivers and Subcomponents

[0104] The storage driver is an OS storage hardware driver that is designed to support checkpointing. Specific components in support of checkpointing are the caching component and the replication component, described immediately below.

[0105] The caching component of the OS storage driver caches storage writes that occur between each checkpoint on both the primary and secondary nodes. Within the checkpoint initiation interval, the OS's view of storage is allowed to differ between nodes because each node operates independently and potentially issues different write requests. Following each checkpoint, the secondary storage is made identical to the primary. Caching writes instead of committing them to the media provides various advantages. For example, the secondary cache can simply be discarded following a checkpoint. Also, in the case of a primary node failure and eventual recovery from the secondary, committing primary writes to the media before performing a checkpoint would be problematic, in which case the primary cache would be discarded as well.

[0106] The OS storage driver's replication component supports replicating storage between primary and secondary nodes. At the start of each startup interval, the storage is the same. This requires an initial full synchronization operation before operation begins. Then, in cooperation with the driver's caching component, storage modifications are replicated from the primary node to the secondary node and committed to the secondary storage media as each checkpoint is performed. Network Drivers and Subcomponents

[0107] The network driver is an OS network hardware driver designed to support real-time FT checkpointing. Components of the network driver include a replicator, a local outgoing packet queue, a release coordinator, a partner outgoing packet queue, and a release notifier, each of which may be implemented as one or more software programs, methods, or routines.

[0108] On the primary node, the replicator component takes outgoing network packets from the OS network stack and sends the packets to the primary's local outgoing packet queue and the secondary's partner outgoing packet queue. On the secondary node, the replicator component is inactive and simply acts as a conduit for packets from the secondary OS network stack to the secondary's local outgoing packet queue. Along with the packets, the replicator passes information identifying the TSB time slot to which each packet is assigned.

[0109] The local outgoing packet queue component queues packets from the local OS network stack for possible later release by the release coordinator. Along with the packets, information identifying the TSB time slot to which each packet is assigned is queued. The partner outgoing packet queue component queues packets from the partner OS network stack for possible later release by the release coordinator. It is active only on secondary nodes.

[0110] Two alternatives for releasing outgoing packets are discussed above, which also apply to OS-based RT / FT systems. These include: a) If the system can tolerate a large number of duplicate packets, both the primary and secondary RT / FT can unconditionally release queued outgoing packets. b) If the system cannot tolerate a large number of duplicate packets, the primary only releases queued outgoing packets in the non-fault case.

[0111] If alternative b is used, the partner outgoing packet queue (active only on the secondary) receives notification from the primary release notifier when a packet is released from the primary. The corresponding packet is then removed from the partner outgoing packet queue.

[0112] The release coordinator releases packets from the appropriate queue at the appropriate time to the OS, and therefore to the external network. In all cases, packets are released only during their assigned TSB time slots.

[0113] In the non-fault case, packets from the previous epoch are either released from both the primary node's local outgoing packet queue and the secondary node's partner outgoing packet queue if alternative a from above is used, or released only from the primary hypervisor's local outgoing packet queue if alternative b above is used.

[0114] If alternative b from above is used, the primary release coordinator notifies the primary release notifier of each released packet, which, after a delay to ensure that the packet was actually released, passes the notification onto the secondary partner outgoing packet queue.

[0115] If alternative b from above is used and the primary hypervisor or VM fails, the secondary release coordinator releases the remaining packets in the secondary partner outgoing packet queue.

[0116] The release notifier is used only in option b above, and notifies the secondary of any packets released by the primary. The primary function of the release notifier is to add enough delay to the notification to the secondary to ensure that the packet was actually released by the primary. This is to avoid any dropped packets after a failover due to the previous secondary thinking that the packet was released when it was not. Modifications and Alternative Embodiments

[0117] Standard active-active checkpointing (eg, COLO) has two performance advantages over the real-time checkpointing described in the previous section: Network packets are not gated, thereby significantly reducing network latency and thereby improving performance. Checkpoints are generally only invoked if there is sufficient difference in the outgoing packets generated by both nodes, reducing the number of checkpoints for most workloads.

[0118] It is not possible to implement a real-time fault-tolerance solution using active-active checkpointing because two nodes (OS-based checkpointing) or two VMs (virtualization-based checkpointing) could generate packets that are sufficiently different at any point in time and therefore require a checkpoint at any point in time, thereby missing real-time deadlines. Notwithstanding the foregoing, it is possible to mitigate the performance benefits of the two above at the expense of some tradeoffs, creating three additional real-time FT checkpointing alternatives. These three alternatives or variations include the following embodiments: 1) Variation 1 - Reduce the checkpoint Omit checkpoints if not needed 2) Variation 2 - Reduce network gating Do not gate packets unless a mismatch exists 3) Variation 3 (Variations 1 and 2 are combined) Do not gate packets unless a mismatch exists Omit checkpoints if not needed

[0119] The three variations listed above require the addition of logic for comparing outgoing packets between the primary and secondary nodes. Specifically, with respect to the virtualization-based real-time checkpointing shown in FIG. 10, logic for comparing outgoing packets from the secondary node with those from the primary node is added to the network support component. Specifically, packet comparison logic is added. This packet comparison logic needs to be active only on one node, but both nodes need to know the results of the comparison. Similarly, FIG. 11 shows a schematic diagram of an OS-based real-time checkpointing network driver with the addition of packet comparison logic added. Generally, references to partners refer to other computing devices or virtual machines / hypervisors, if applicable. Variation 1 - Reduce the checkpoint

[0120] If the outgoing packets generated by both the primary and secondary are effectively equivalent during a cycle, there is no need to perform a checkpoint. This has the advantage that computation cycles and runtime that would otherwise be spent on checkpointing are available for the application. This does have disadvantages, including one or more of the following: 1) Logic is added to the design to compare outgoing packets from the primary and secondary. 2) When a checkpoint is needed, this will be longer, on average, than if checkpoints were done every cycle. The minimum designed possible cycle time is therefore increased to allow for these longer checkpoints. 3) For reasons outlined immediately below, packets are gated for a maximum of just over two cycles if the mismatch occurs too late in the cycle to invoke a checkpoint. As explained later, this disrupts the regular cyclic behavior of the basic implementation and can also further increase the minimum cycle time.

[0121] Figure 12A illustrates the need for packet gating when a mismatch occurs too late in a cycle to invoke a checkpoint. As shown in cycle 3 of Figure 12B, if packets between the primary and secondary mismatch, when it is too late to invoke a checkpoint, the two nodes enter the next cycle in an inconsistent state. As a result, if the secondary needs to take over for the primary in a subsequent cycle (due to a primary failure), the inconsistent state is released to the outside world.

[0122] One solution to this problem is to start gating packets once a mismatch occurs, checkpoint in the following cycle, and continue gating packets until after the checkpoint. This prevents any detectable inconsistencies from being transmitted to the outside world, thereby ensuring that the secondary is in a sufficiently consistent state to take over from the primary. If the primary fails, it is the secondary's packet that is released. This adds at least two cycles of latency to the packet: one for the delay mentioned above and another because this variant always adds a cycle of delay, as in the basic solution. This would cause a cycle in which no packets are transmitted and a subsequent cycle in which two packets are transmitted. This would disrupt cyclic behavior and could affect the minimum possible cycle time if transmitting twice as many packets in a cycle is on a critical timing path.

[0123] Variant 1 provides an additional calculation cycle, but the additional calculation cycle is a discretionary calculation cycle. The system is sized so that it can perform the minimum required work, assuming checkpoints in every cycle. In practice, however, assuming relatively short cycle times, the majority of cycles will not require checkpoints. Variation 2 - Reduce network gating

[0124] In one embodiment, the real-time checkpointing solution gates all outgoing packets at every wake-up interval. This can be relaxed to only begin gating packets once a mismatch occurs, which has the advantage of reducing average network latency. In various embodiments, as part of implementing this variation, logic, such as a hardware or software module, is added to applicable systems and methods to compare outgoing packets from the primary and secondary. Network traffic is no longer strictly cyclical, as illustrated in FIG. 12B. Specifically, referring to FIG. 12B, no packets would be delivered during time slot 3 in cycle 2, while in time slot 3 of cycle 3, packets from both cycles 2 and 3 would be released. If network bandwidth were the critical path in determining the minimum cycle time, the cycle time would be increased because two cycles' worth of packets may need to be released in a single time slot.

[0125] Also, note that only the average network latency is reduced; in the worst case, the latency remains the same or worsens. This occurs because the very first packet may be mismatched and the cycle time is potentially longer due to network bandwidth being the critical path in determining the minimum cycle time. Variation 3 - Reduce network gating and checkpoints

[0126] Variants 1 and 2 may be combined to produce a solution with lower average network latency and more flexible calculation cycles, but this suffers from the disadvantages of variants 1 and 2 outlined above. Improving real-time fault-tolerant systems over traditional checkpointing

[0127] The real-time fault-tolerant systems and methods discussed above improve upon and avoid characteristics of various legacy checkpointing approaches. An active-passive checkpointing solution has an active VM or OS that runs a target workload and a passive (standby) VM or OM that is effectively idle and receives periodic snapshots of the active VM or OS's state to continue running the target workload if the previous active VM or OS fails.

[0128] Active-passive checkpointing solutions synchronize VM or OS state at high frequency and buffer (gate) outgoing network packets during each epoch until the checkpoint is complete (state capture and propagation), ensuring that external clients are provided with functionally consistent network packets across a failover. Additionally, active-active checkpointing improves performance over active-passive checkpointing for most workloads by relaxing checkpoint frequency based on output similarity and eliminating the need to gate outgoing network packets during each epoch.

[0129] In an active-active checkpointing system, both VMs and / or OS instances are running, and outgoing network packets are compared between the instances and immediately released if they are sufficiently similar. Redundant VMs or OSs can differ as long as they generate virtually identical responses to client requests. If corresponding outgoing network packets are no longer virtually identical, a checkpoint is invoked to resynchronize the VMs or OSs.

[0130] Active-passive checkpointing is not suitable for real-time operation for several reasons. Similarly, active-active checkpointing is not suitable for real-time operation for several reasons. Some of these reasons are discussed below and provide rationale as to why traditional active-passive and active-active checkpointing differ from the various RT / FT embodiments of the present disclosure because such RT / FT embodiments avoid one or more or all of the following modes of operation:

[0131] In an active-passive system, checkpoint timing is independent of TSB timing and can be delayed to real-time deadlines. If a failure occurs, the system reverts to the last checkpoint, i.e., up to an epoch of processing time is lost, potentially delaying real-time deadlines. In addition, active-passive checkpointing can drop outgoing packets in response to a failure. This violates real-time obligations and is therefore unacceptable for most real-time applications. Non-real-time solutions rely on higher-level network protocols (e.g., TCP) to compensate for dropped packets. With active-active checkpointing, checkpoints are independent of TSB timing and occur whenever a packet mismatch occurs, thus potentially delaying real-time deadlines. A given RT / FT embodiment operates in a manner that avoids the active-passive and active-active checkpointing issues and design requirements discussed herein.

[0132] Fault-tolerant computer systems frequently include two processor subsystems, generally referred to as primary and secondary nodes. These nodes either operate in parallel, substantially synchronously executing the same program (active-active), or operate as a single system, with the primary node actively performing useful computations and the secondary node standing by awaiting failure of the primary system (active-standby). If the primary system fails, the secondary takes over as the new primary, continuing the work of the failing primary and allowing it to be replaced. General and Specific Computer-Based Implementations and Embodiments

[0133] Unless otherwise specifically stated as will be apparent from the discussion below, throughout the description, discussions utilizing terms such as "processing" or "calculating" or "checkpointing" or "calculating" or "delaying" or "comparing" or "generating" or "determining" or "transferring" or "postponing" or "delivering" or "suspending" or "handling" or "receiving" or "buffering" or "allocating" or "displaying" or "flag" or Boolean logic or other configuration related operations or equivalents will be understood to refer to the actions and processes of a computer system or electronic device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of the computer system or electronic device into other data that is similarly represented as physical quantities in electronic memory or registers or other such information storage, transmission, or display device.

[0134] The algorithms presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the above description. In addition, the present disclosure is not described with reference to any particular programming language, and various embodiments may therefore be implemented using a variety of programming languages.

[0135] Several implementations have been described. Nevertheless, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. For example, various forms of the flows shown above may be used, and steps may be reordered, added, or removed. Accordingly, other implementations are within the scope of the following claims.

[0136] The examples presented herein are intended to illustrate potential and specific implementations of the present disclosure. The examples are intended primarily for purposes of illustration of the present disclosure for those skilled in the art. No particular aspect or aspects of the examples are necessarily intended to limit the scope of the present disclosure.

[0137] The figures and descriptions of the present disclosure have been simplified to illustrate elements relevant for a clear understanding of the present disclosure, while excluding other elements for purposes of clarity. However, those skilled in the art will recognize that these classifications of focused discussion do not facilitate a deeper understanding of the present disclosure, and therefore, a more detailed description of such elements is not provided herein.

[0138] The processes associated with the present embodiment may be executed by a programmable device such as a computer. Software or other sets of instructions that can be employed to cause the programmable device to execute the processes may be stored in any storage device, such as, for example, a computer system (non-volatile) memory, an optical disk, a magnetic tape, or a magnetic disk. Furthermore, some of the processes may be programmed when the computer system is manufactured or via a computer-readable memory medium.

[0139] It should also be understood that certain process aspects described herein may be implemented using instructions stored on a computer-readable memory medium or medium that instructs a computer or computer system to perform the process steps. Computer-readable media may include, for example, memory devices such as diskettes, compact discs of both read-only and read / write varieties, optical disk drives, and hard disk drives. Computer-readable media may also include memory storage devices, which may be physical, virtual, permanent, temporary, semi-permanent, and / or semi-temporary.

[0140] The computer systems and computer-based devices disclosed herein may include memory for storing certain software applications used in acquiring, processing, and communicating information. It is understood that such memory may be internal or external with respect to the operation of the disclosed embodiments. Memory may also include any means for storing software, including hard disks, optical disks, floppy disks, ROM (read-only memory), RAM (random access memory), PROM (programmable ROM), EEPROM (electrically erasable programmable read-only memory), and / or other computer-readable memory media. In various embodiments, a "host," "engine," "loader," "filter," "platform," or "component" may include various computers or computer systems, or may include any reasonable combination of software, firmware, and / or hardware.

[0141] In various embodiments of the present disclosure, a single component may be replaced by multiple components, and multiple components may be replaced by a single component, to perform a given function or functions. Except to the extent that such substitution would not be operational for practicing embodiments of the present disclosure, such substitution is within the scope of the present disclosure. For example, any of the servers may be replaced by a “server farm” or other grouping of networked servers (e.g., a group of server blades) arranged and configured for cooperative functions. It should be understood that a server farm may serve to distribute workloads among / among the farm's individual components and expedite computing processes by utilizing the collective and cooperative capabilities of multiple servers. Such server farms may employ load balancing software to perform tasks such as, for example, tracking requests for processing power from different machines, prioritizing and scheduling tasks based on network demands, and / or providing backup contingency in the event of component failure or reduced operability.

[0142] In general, it may be apparent to those skilled in the art that the various embodiments described herein, or components or portions thereof, may be implemented in many different embodiments of software, firmware, and / or hardware, or modules thereof. The software code or specialized control hardware used to implement some of the present embodiments is not limiting of the present disclosure. Programming languages ​​for computer software and other computer-implemented instructions may be translated into machine language by a compiler or assembler before execution, and / or may be translated directly at runtime by an interpreter.

[0143] Examples of assembly languages ​​include ARM, MIPS, and x86; examples of high-level languages ​​include Ada, BASIC, C, C++, C#, COBOL, Fortran, Java, Lisp, Pascal, Object Pascal; and examples of scripting languages ​​include Bourne script, JavaScript, Python, Ruby, PHP, and Perl. Various embodiments may be employed, for example, within a Lotus Notes environment. Such software may be stored on any type of suitable computer-readable medium or media, such as, for example, a magnetic or optical storage medium. Accordingly, the operation and behavior of embodiments will be described without specific reference to actual software code or specialized hardware components. The absence of such specific reference is appropriate because it is clearly understood that one skilled in the art would be able to design software and control hardware to implement embodiments of the present disclosure based on the description herein with only a reasonable effort and without undue experimentation.

[0144] Various embodiments of the systems and methods described herein may employ one or more electronic computer networks to facilitate communication between different components, convey data, or share resources and information. Such computer networks can be categorized according to the hardware and software technologies used to interconnect devices in the network.

[0145] Computer networks may be characterized based on the functional relationships between the elements or components of the network, such as active networking, client-server, or peer-to-peer functional architectures. Computer networks may be classified according to their network topology, such as, for example, bus networks, star networks, ring networks, mesh networks, star-bus networks, or hierarchical topology networks. Computer networks may also be classified based on the method employed for data communication, such as digital and analog networks.

[0146] Embodiments of the methods, systems, and tools described herein may employ internetworking to connect two or more distinct electronic computer networks or network segments through a common routing technology. The type of internetwork employed may depend on the management and / or participation in the internetwork. Non-limiting examples of internetworks include intranets, extranets, and the Internet. Intranets and extranets may or may not have a connection to the Internet. If connected to the Internet, an intranet or extranet may be protected using appropriate authentication technology or other security measures. As applied herein, an intranet may be a group of networks employing Internet protocols, web browsers, and / or file transfer applications under common control by an administrative entity. Such an administrative entity may restrict access to the intranet, for example, to only authorized users or to another internal network of an organization or commercial entity.

[0147] Unless otherwise indicated, all numbers expressing length, width, depth, or other dimensions used in the specification and claims are to be understood in all instances as indicating both the exact value as indicated and the value as modified by the term "about." As used herein, the term "about" refers to a ±10% variation from the nominal value. Thus, unless indicated to the contrary, the numerical parameters set forth in this specification and the appended claims are approximations that may vary depending on the desired properties sought to be obtained. At the very least, and not as an attempt to limit the application of the doctrine of equivalents to the scope of the claims, each numerical parameter should be construed, at least in light of the number of reported significant digits and by applying ordinary rounding techniques. Any specific value may vary by 20%.

[0148] In various embodiments, one or more of the techniques, methods, systems, subsystems, and combinations described above can be implemented using virtualized operating systems, container-based operating systems, virtualization environments, and various emulators such as VMware, QEMU, KVM, and others.

[0149] The present disclosure may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The foregoing embodiments are therefore to be considered in all respects illustrative and not limiting on the disclosure described herein. The scope of the invention is, therefore, indicated by the appended claims, rather than by the foregoing description, and all changes that come within the meaning and range of equivalency of the claims are intended to be embraced therein.

[0150] It will be understood by those skilled in the art that various modifications and variations can be made without departing from the scope of the described technology. Such modifications and variations are intended to fall within the scope of the described embodiments. It will also be understood by those skilled in the art that features included in one embodiment can be substituted for other embodiments, and that one or more features from a depicted embodiment can be included in another depicted embodiment in any combination. For example, any of the various components described herein and / or depicted in the figures may be combined with, substituted for, or excluded from other embodiments.

Claims

1. 1. A real-time fault-tolerant computing system, comprising: a first computing device, one or more memory devices; one or more processors; a first network interface operable to receive device data and transmit output data via a timeslot-based bus, the output data being generated from processing the device data; and a first real-time checkpoint engine; a first computing device comprising: a second computing device, one or more memory devices; one or more processors; a second network interface operable to receive device data via the time slot-based bus; a second real-time checkpoint engine; a second computing device comprising: a hardware interconnection operable to enable data exchange between the first computing device and the second computing device; Equipped with a first computing device that is connected to the first bus and a second computing device that is connected to the second bus; a second computing device that is connected to the first bus and a second computing device that is connected to the second bus;

2. 2. The real-time fault-tolerant computing system of claim 1, wherein the first computing device and the second computing device process received device data concurrently such that if the first computing device suffers a failure, the second computing device continues to process device data, generate the output data, and transmit the output data using the time slot-based bus.

3. 3. The real-time fault-tolerant computing system of claim 2, wherein a checkpoint operation is always deferred until the occurrence of a lower priority communication slot, even if the first computing device and the second computing device generate divergent output data prior to the occurrence of the lower priority communication slot.

4. 4. The real-time fault-tolerant computing system of claim 3, wherein both the first computing device and the second computing device delay releasing output data to one or more devices communicating with the time slot-based bus until after a subsequent checkpoint to avoid releasing divergent output data in the event of a failure.

5. 3. The real-time fault-tolerant computing system of claim 2, wherein the first computing device generates a checkpoint every cycle when output data comprising low priority data is transmitted using a low priority time slot.

6. The real-time fault-tolerant computing system of claim 2, wherein the output data comprises sensor data, the sensor data operable as input for a control system.

7. 2. The real-time fault-tolerant computing system of claim 1, wherein the first computing device and the second computing device are both actively processing device data and generating output data such that if the first computing device experiences one or more failure modes, there is zero time delay for a second processing device to continue processing.

8. The real-time fault-tolerant computing system of claim 2 , wherein the duration of one or more time slots is increased by a detection period, said detection period ranging from about 1 microsecond to about 100 microseconds.

9. 3. The real-time fault-tolerant computing system of claim 2, wherein both the first computing device and the second computing device delay releasing output data to devices communicating with the time slot-based bus until after a next checkpoint to avoid releasing inconsistent output data.

10. 10. The real-time fault-tolerant computing system of claim 9, wherein the first computing device releases any delayed output data if the first computing device is operating normally after a checkpoint.

11. 3. The real-time fault-tolerant computing system of claim 2, wherein the first computing device transmits all of its output data to the second computing device to prevent such output data from being dropped in response to a failure of the first computing device.

12. 12. The real-time fault-tolerant computing system of claim 11, wherein if the first computing device fails, the second computing device takes over releasing the output data of the first computing device that was generated during the last checkpoint.

13. 3. The real-time fault-tolerant computing system of claim 2, wherein only the first computing device handles the release of output data when no failures occur, and in parallel shares such output data with the second computing device.

14. 2. The real-time fault-tolerant computing system of claim 1, wherein the device data from one or more devices is transmitted to the first computing device and the second computing device via the time slot-based bus.

15. The real-time fault-tolerant computing system of claim 2 , wherein the first computing device and the second computing device receive incoming data directly through broadcast or multicast.

16. 3. The real-time fault-tolerant computing system of claim 2, wherein the first computing device and the second computing device each comprise a separate non-virtualized operating system (OS) comprising separate network drivers, checkpoint shims, and storage drivers stored in the one or more memory devices and executing on each of the one or more processors.

17. 17. The real-time fault-tolerant computing system of claim 16, wherein the network driver comprises a replicator in communication with a release coordinator and a release notifier.

18. 4. The real-time fault-tolerant computing system of claim 3, wherein one or more networking programs stored in one or more memory devices of the first and second computing devices delay and coordinate transmission of output data using one or more of the hardware interconnect, the first network interface, and the second network interface.

19. 3. The real-time fault-tolerant computing system of claim 2, wherein the first computing device and the second computing device each comprise a respective virtual operating system (OS) stored in the one or more memory devices and executed using a respective hypervisor, the respective virtual operating system comprising one or more respective network support applications, a checkpoint engine, and one or more respective storage device support applications.

20. 1. A method for controlling an operating system in real time, comprising: receiving, at the primary node and the secondary node, device data from one or more devices in electrical communication with a time slot-based bus, the bus being operable to receive data according to time slots of two or more priority levels; generating a checkpoint at the primary node during a first time slot and transmitting the checkpoint to the secondary node, the first time slot being a lower priority time slot; generating output data at the primary node and the secondary node in response to processing the received device data; transmitting the output data from the primary node over the bus using a second time slot, the second time slot being a higher priority time slot, and both the primary node and the secondary node being in an active mode; A method comprising:

21. 21. The method of claim 20, further comprising transmitting data during a non-checkpoint portion of a lower priority time slot.

22. 21. The method of claim 20, further comprising gating network data using separate network software modules executing on both the primary and secondary nodes so that output data is released from the nodes whose state has been carried forward to the current checkpoint.

23. 23. The method of claim 22, further comprising synchronizing transmission of output data with a high priority time slot, said output data being received by said operating system.

24. 23. The method of claim 22, further comprising transmitting data during a non-checkpoint portion of a lower priority time slot.

25. 21. The method of claim 20, further comprising using the primary node to transmit all of its respective outgoing data to the secondary node to prevent such outgoing data from being dropped in response to the occurrence of a failure of the primary node.

26. The system of claim 1 , wherein checkpoints are transmitted using the hardware interconnect.

27. 21. The method of claim 20, further comprising increasing the duration of one or more time slots to allow for detection of a failure of a primary node or a secondary node.

28. 28. The method of claim 27, wherein the duration of one or more time slots is increased by a detection period, the detection period ranging from about 1 microsecond to about 100 microseconds.

29. 21. The method of claim 20, wherein the checkpoints are transmitted using a hardware interconnect.

Images