Managing computer cluster using replicated task results

Abstract

To provide a method for processing tasks in a distributed data processing system.SOLUTION: A method for processing tasks in a distributed data processing system including a number of processing nodes. The method includes: maintaining, at a first processing node, a number of counters including a working counter indicating a current time interval of a number of time intervals in the distributed data processing system, and a replication counter in which at least one of all tasks associated with one time interval of the number of time intervals or at least one of all corresponding results associated with that time interval are replicated at multiple processing nodes of the number of processing nodes; and providing messages from the first processing node to another processing nodes of the number of processing nodes, the messages including the working counter and the replication counter.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority from U.S. Patent Application No. 62 / 579,225, filed on October 31, 2017, which is hereby incorporated by reference in its entirety for all purposes.

[0002] This specification relates to managing computer clusters.

Background Art

[0003] One approach to dataflow computation uses a graph-based representation in which computer components corresponding to nodes (vertices) of a graph (called a "dataflow graph") are connected by dataflows corresponding to links (directed edges) of the graph. Downstream components, connected to upstream components by dataflow links, receive an ordered stream of input data elements, process the input data elements in the order they are received, and optionally generate one or more corresponding streams of output data elements. A system for performing such graph-based computation is described in U.S. Patent No. 5,966,072, titled "EXECUTING COMPUTATIONS EXPRESSED AS GRAPHS," which is incorporated herein by reference. In embodiments relating to the approach described in this prior art, each component is implemented as a process typically hosted on one of several computer servers. Each computer server may have several such component processes that are active at any given time, and an operating system (e.g., Unix®) scheduler shares resources (e.g., processor time and / or processor cores) among the components hosted on that server. In such embodiments, data flow between components may be carried out using the operating system's data communication services and data networks connecting servers (e.g., pipes, TCP / IP sessions, etc.). A subset of components typically acts as a source and / or sink for data from the overall computation (e.g., data files, database tables, and external data flows). After component processing and data flow are established, for example, through cooperative processing, the data flows through the entire computer system, performing computations represented as graphs, typically defined by the availability of input data in each component, and scheduling the computing resources of each component.Therefore, parallelism can be achieved by at least enabling various components to be executed in parallel by various processes (hosted on the same or different server computers or processor cores). Here, various components that execute in parallel along various paths via the dataflow graph are referred to as component parallelism, and various components that execute in parallel along different parts of the same path via the dataflow graph are referred to as pipeline parallelism. [Overview of the Initiative] [Means for solving the problem]

[0004] Other forms of parallelism are also supported by such methods. For example, an input dataset may be partitioned according to the partitioning of field values ​​within the dataset's records, and each partition is sent to a separate copy of a component that processes the dataset's records. Such a separate copy (or "instance") of the component may run on a separate server computer or a separate processor core of a server computer, thereby achieving what is referred to herein as data parallelism. The results of the separate components may be merged to form a single data flow or dataset again. The number of computers or processor cores used to run instances of the system will be specified by the developers when the dataflow graph is being developed.

[0005] Various techniques can be used to improve the efficiency of such techniques. For example, each instance of a component does not necessarily need to be hosted in its own operating system processing, for example, by using one operating system processing to implement multiple components (e.g., components that form connected subgraphs of a larger graph).

[0006] At least some embodiments of the methods described above suffer from limitations regarding the efficiency of executing the resulting processing on the underlying computer server. For example, this limitation may relate to the difficulty of reconfiguring running instances of the graph to change the degree of data parallelism, to change the server hosting the various components, and / or to balance the load on various computer resources. Existing graph-based computer systems also suffer from slow startup times, as too many processes are often started unnecessarily, wasting vast amounts of memory. Generally, processing begins with the start of graph execution and ends when graph execution is completed.

[0007] Other systems for distributed computing are used, where the entire computation is divided into smaller parts, and these parts are distributed from a master computer server to various other (e.g., "slave") computer servers, each independently performing computations and returning the results to the master server. Some of these methods are called "grid computing." However, such methods typically rely on the independence of each computation without providing a mechanism for passing data between computational parts or scheduling and / or ordering the execution of these parts, except through the master computer server that calls them. Therefore, such methods do not provide a direct and efficient solution to hosting computations that involve interactions between multiple components.

[0008] Another approach for distributed computing on large datasets leverages the MapReduce framework, as embodied in systems such as Apache Hadoop®. Typically, Hadoop has a distributed file system where portions of each nominated file are distributed. The user defines the computation in terms of two functions: a map function executed in a distributed manner on all portions of the nominated input, and a reduce function executed on portions of the output of the map function execution. The output of the map function execution is partitioned and stored again in intermediate portions within the distributed file system. The reduce function is then executed in a distributed manner to process the intermediate portions, producing the result of the entire computation. Computations that can be represented within the MapReduce framework and whose inputs and outputs can be modified for storage in the map-reduce framework's file system can be performed efficiently, but many computations are incompatible with this framework and / or cannot be easily adapted to have all of their inputs and outputs in a distributed file system.

[0009] In a general embodiment, a method for processing tasks in a distributed data processing system comprising many processing nodes includes the step of processing many sets of tasks using two or more of the many processing nodes. Each set of tasks is configured to produce a corresponding set of results and is associated with a corresponding time interval among many time intervals. The method includes the step of maintaining many counters in a first processing node among many processing nodes. Many counters include a work counter indicating the current time interval among many time intervals in the distributed data processing system, and a replication counter indicating one time interval among many time intervals, wherein at least one of (1) all tasks associated with that time interval or (2) all corresponding results associated with that time interval is replicated in multiple processing nodes among many processing nodes. This method includes providing a message containing a work counter and a replication counter from a first processing node among many processing nodes to other processing nodes.

[0010] The embodiment may include one or more of the following features:

[0011] This process may include the steps of performing calculations related to at least some tasks at a main processing node among many processing nodes, and performing calculations related to replicas of at least some tasks at one or more backup processing nodes among many processing nodes. The step of performing calculations related to the first task at the main processing node may include the step of generating a first result at the main processing node, and the step of performing calculations related to replicas of the first task at the backup processing node includes the step of generating a first result at the backup processing node.

[0012] Both the computations related to the first task performed on the primary processing node and the computations related to the replica of the first task performed on the backup processing node may begin after a commit operation indicating that both the first task and its replica are persistently stored. The first result may include a modified version of the original data replicated on both the primary and backup processing nodes. The computations related to the first task performed on the primary processing node may be identical to the computations related to the replica of the first task performed on the backup processing node.

[0013] The computations related to the first task performed on the main processing node and the computations related to the replica of the first task performed on the backup processing node can be deterministic and depend on which of the multiple processing nodes performs the computation. The term "interceptor" may represent one of many time intervals in which all tasks related to that time interval and all corresponding results related to that time interval are replicated across multiple processing nodes.

[0014] This process may include the steps of performing calculations related to at least some tasks on the main processing node among many processing nodes while a replica of the task is not running on the backup processing node, and transmitting the results of the calculations from the main processing node to the backup processing node that correspond to the tasks performed. The step of performing calculations related to the first task on the main processing node may include the step of generating a first result, which may be transmitted from the main processing node to the backup processing node after the first task has been completed on the main processing node.

[0015] The first result may include a modified version of the original data replicated in the primary processing node and the backup processing node. The replication counter may represent one of many time intervals in which all tasks related to that time interval and all corresponding results related to that time interval are replicated in multiple processing nodes of many processing nodes.

[0016] In another common embodiment, software for processing tasks within a distributed data processing system involving many nodes is stored in a non-temporary format on a computer-readable medium. This software contains instructions for the computer system to use two or more of the many processing nodes to process many sets of tasks. Each set of tasks is configured to produce a corresponding set of results and is associated with a corresponding time interval among many time intervals.

[0017] The above instructions also cause the computer system to maintain many counters within the first processing node of many processing nodes. Many counters include a work counter that indicates the current time interval among many time intervals in the distributed data processing system, and one of the many time intervals. A replication counter indicating a time interval, comprising (1) all tasks associated with the time interval or (2) at least one of all corresponding results associated with the time interval, which is replicated in multiple processing nodes of a large number of processing nodes. The instruction also causes the computer system to provide a message containing the work counter and the replication counter from a first processing node of a large number of processing nodes to other processing nodes.

[0018] In another general embodiment, the device for processing data includes a distributed data processing system comprising many processing nodes, each having at least one processor, and a communication medium connecting the many processing nodes for sending and receiving information between the many processing nodes. The distributed data processing system is configured to use two or more of the many processing nodes to process many sets of tasks, each set of tasks associated with a corresponding time interval among many time intervals, each set of tasks configured to produce a corresponding set of results, and to maintain many counters in a first processing node among the many processing nodes. The many counters include a work counter indicating the current time interval among many time intervals in the distributed data processing system, and a replication counter indicating one of the many time intervals, wherein at least one of (1) all tasks associated with that time interval or (2) all corresponding results associated with that time interval is replicated in multiple processing nodes among the many processing nodes. The device is further configured to provide messages containing the work counters and replication counters from a first processing node among the many processing nodes to other processing nodes.

[0019] In another general embodiment, a computer system for processing tasks in a distributed data processing system comprising many processing nodes includes means for processing many sets of tasks associated with corresponding time intervals in a many time intervals, each set of tasks configured to produce a corresponding set of results, using two or more of the many processing nodes, and means for maintaining many counters in a first processing node of the many processing nodes. The many counters include a work counter indicating the current time interval in a many time intervals in the distributed data processing system, and a replication counter indicating one of the many time intervals, wherein at least one of (1) all tasks associated with that time interval or (2) all corresponding results associated with that time interval is replicated in multiple processing nodes of the many processing nodes. The computer system also includes means for providing messages containing the work counters and replication counters from the first processing node of the many processing nodes to other processing nodes.

[0020] In a general embodiment, a method for managing a distributed data processing system including many processing nodes includes the step of maintaining many data storages within the system, each of which data storages is associated with a corresponding processing node among the many processing nodes and with one of many persistence levels, the many persistence levels including a first persistence level and a second persistence level having a greater level of persistence than the first persistence level. The method also includes the step of processing many sets of data units using two or more processing nodes among the many processing nodes, each data unit in each set of data units is associated with a corresponding time interval among many time intervals. The many sets of data units include a first set of data units associated with a first time interval among many time intervals.

[0021] This processing step includes updating relevant indicators to indicate that, for each specific persistence level, all sets of data units associated with a first time interval are stored at that particular persistence level. This processing step also includes processing many sets of requests using two or more of many processing nodes, where each request in each set of requests is configured to trigger a state update in one of the many processing nodes, and for many time intervals Associated with our response time interval, many sets of requests include a first set of requests associated with a second time interval among many time intervals. This processing step also includes the step of maintaining many counters at a first processing node among many processing nodes.

[0022] Many counters include work counters that indicate the current time interval among many time intervals in a distributed data processing system, and replication counters that indicate one of many time intervals, where all requests related to that time interval are replicated across multiple processing nodes of many processing nodes.

[0023] The method also includes the step of providing a first message, which includes the values ​​of a work counter and a replication counter, from a first processing node among many processing nodes to other processing nodes in a first time.

[0024] The embodiment may include one or more of the following features:

[0025] Many counters may further include a persistence counter indicating one of a plurality of time intervals, where all requests related to the time interval are stored in persistent storage associated with at least one of a plurality of processing nodes. The method includes, for each data unit of a first set of data units, storing the data unit in some of a plurality of data storages associated with each of the plurality of processing nodes, including storing the data unit in some of the plurality of data storages related to a first level of persistence and storing the data unit in one or more of the plurality of data storages related to a second level of persistence.

[0026] In another general aspect, software for managing a distributed data processing system including a plurality of processing nodes is stored in a non-transitory form on a computer-readable medium. The software includes instructions for causing a computer system to hold a plurality of data storages within the system, each of the plurality of data storages being associated with a corresponding processing node of the plurality of processing nodes and associated with one of a plurality of persistence levels, the plurality of persistence levels including a first persistence level and a second persistence level having a greater level of persistence compared to the first persistence level. The instructions also cause the computer system to process a plurality of sets of data units by using two or more of the plurality of processing nodes, each data unit of each set of data units being associated with a corresponding time interval of a plurality of time intervals, the plurality of sets of data units including a first set of data units related to a first time interval of the plurality of time intervals, and the processing includes updating a related indicator to indicate that all sets of data units related to the first time interval are stored at the particular persistence level, for each particular persistence level.

[0027] The above instructions also cause the computer system to process many sets of requests by using two or more of many processing nodes, each request of each set of requests being configured to cause a state update in one of the many processing nodes, associated with a corresponding time interval of many time intervals, and many sets of requests including a first set of requests associated with a second time interval of many time intervals. The above instructions also cause the computer system to hold many counters within a first processing node of many processing nodes. The many counters include a working counter indicating a current time interval of many time intervals within the distributed data processing system, and a replicated counter indicating one of the many time intervals, the replicated counter being such that all requests associated with that time interval are replicated at a plurality of processing nodes of the many processing nodes.

[0028] The above instructions also cause the computer system to provide a first message including the value of the working counter and the value of the replicated counter from a first processing node of many processing nodes to other processing nodes at a first time.

[0029] In another general aspect, an apparatus includes a distributed data processing system including many processing nodes, each processing node including at least one processor and a communication medium connecting many processing nodes for sending and receiving information among the processing nodes of the many processing nodes. The distributed data processing system is configured to hold many data storages within the system, each data storage of the many data storages being associated with a corresponding processing node of the many processing nodes and associated with one of many persistence levels, the many persistence levels including a first persistence level and a second persistence level having a greater level of persistence compared to the first persistence level.

[0030] The device is also configured to process many sets of data units using two or more processing nodes out of many processing nodes, where each data unit in each set of data units is associated with a corresponding time interval out of many time intervals, and the many sets of data units include a first set of data units associated with a first time interval out of many time intervals, and the processing includes updating relevant indicators to indicate that, for each particular persistence level, all sets of data units associated with the first time interval are stored at that particular persistence level.

[0031] The device is also configured to process many sets of requests using two or more processing nodes, and each request in each set of requests is configured to trigger a state update in one of the many processing nodes and is associated with a corresponding time interval in many time intervals, and the many sets of requests include a first set of requests associated with a second time interval in many time intervals. The device is also configured to maintain many counters in the first processing node of the many processing nodes.

[0032] Many counters include a work counter that indicates the current time interval among many time intervals in a distributed data processing system, and a replication counter that indicates one of many time intervals, where all requests related to that time interval are replicated across multiple processing nodes of many processing nodes. The device is also configured to provide a first message containing the values ​​of the work counter and the replication counter from a first processing node among many processing nodes to other processing nodes in a first time.

[0033] In another general embodiment, a method for managing a distributed data processing system including many processing nodes includes the steps of: receiving input data in a distributed data processing system interface component that communicates with the distributed data processing system; providing the received input data to the distributed data processing system, wherein the distributed data processing system assigns an index related to a first time interval among many time intervals to the input data; receiving result data related to the input data from the distributed data processing system in the distributed data processing system interface component, wherein the result data includes an index related to the first time interval; determining an index related to a second time interval among many time intervals at a first processing node among many processing nodes; comparing the index related to the second time interval with the index related to the first time interval included in the result data in the distributed data processing system interface component, and releasing the result data from the distributed data processing system interface component if the index related to the second time interval is the same as or corresponds to a later time interval than the first time interval; and maintaining many data storages within the system, wherein each data in the many data storages The storage comprises the steps of: a process in which storage is associated with a corresponding processing node among many processing nodes and with one persistence level among many persistence levels, the many persistence levels including a first persistence level and a second persistence level having a greater level of persistence compared to the first persistence level; and a process in which processing many sets of data units using two or more processing nodes among many processing nodes, each data unit of each set of data units is associated with a corresponding time interval among many time intervals, the many sets of data units include a first set of data units associated with a third time interval among many time intervals, and the processing comprises the step of updating relevant indicators to indicate that for each particular persistence level, all sets of data units associated with the third time interval are stored at that particular persistence level.

[0034] The embodiment may include one or more of the following features:

[0035] For each data unit in the first set of data units, the data unit may be stored in some of the many data storages associated with each processing node of many processing nodes, and this storage process includes storing the data unit in some of the many data storages associated with a first level of persistence and storing the data unit in one or more of the many data storages associated with a second level of persistence. A metric related to the second time interval may be provided to the distributed data processing system interface component.

[0036] In another common embodiment, software for managing a distributed data processing system, which includes many processing nodes, is stored in a non-temporary format on a computer-readable medium.The above software includes instructions for a computer system to perform the following: receive input data in a distributed data processing system interface component that communicates with a distributed data processing system; provide the received input data to the distributed data processing system, which assigns and provides an index related to a first time interval among many time intervals to the input data; receive result data related to the input data from the distributed data processing system in the distributed data processing system interface component, which includes an index related to the first time interval; determine an index related to a second time interval among many time intervals at a first processing node among many processing nodes; compare the index related to the second time interval with the index related to the first time interval included in the result data in the distributed data processing system interface component, and determine if the index related to the second time interval is the same as or later than the first time interval. In response, the system includes: determining the release of result data from a distributed data processing system interface component; maintaining a large amount of data storage within the system, where each of the large amount of data storage is associated with a corresponding processing node among the large amount of processing nodes and with one of the large amount of persistence levels, where the large amount of persistence levels includes a first persistence level and a second persistence level having a greater level of persistence compared to the first persistence level; and processing a large amount of data units using two or more processing nodes among the large amount of processing nodes, where each data unit in each amount of data units is associated with a corresponding time interval among the large amount of time intervals, where the large amount of data units includes a first amount of data units associated with a third time interval among the large amount of time intervals, and the processing includes updating relevant indicators to indicate that, for each particular persistence level, all amounts of data units associated with the third time interval are stored at that particular persistence level.

[0037] In another general embodiment, the device includes a distributed data processing system comprising many processing nodes, each processing node having at least one processor and between the processing nodes of the many processing nodes The distributed data processing system includes a communication medium that connects many processing nodes for sending and receiving information. The distributed data processing system is configured as follows: A distributed data processing system interface component that communicates with the distributed data processing system receives input data and provides the received input data to the distributed data processing system. The distributed data processing system assigns an index related to a first time interval among many time intervals to the input data. The distributed data processing system interface component receives result data related to the input data from the distributed data processing system, and the result data includes an index related to the first time interval. An index related to a second time interval among many time intervals is determined at the first processing node among many processing nodes. The distributed data processing system interface component compares the index related to the second time interval with the index related to the first time interval included in the result data. If the index related to the second time interval is the same as or corresponds to a later time interval than the first time interval, the distributed data The processing system interface component releases result data, maintains many data storages within the system, each data storage is associated with a corresponding processing node among many processing nodes and with one of many persistence levels, the many persistence levels include a first persistence level and a second persistence level having a greater level of persistence compared to the first persistence level, processes many sets of data units using two or more processing nodes among many processing nodes, each data unit in each set of data units is associated with a corresponding time interval among many time intervals, the many sets of data units include a first set of data units associated with a third time interval among many time intervals, and the processing includes updating relevant indicators to indicate that for each particular persistence level, all sets of data units associated with the third time interval are stored at that particular persistence level.

[0038] In another general embodiment, a method for managing a distributed data processing system including many processing nodes includes the steps of: receiving input data in a distributed data processing system interface component that communicates with the distributed data processing system; providing the received input data to the distributed data processing system, wherein the distributed data processing system assigns an index related to a first time interval among many time intervals to the input data; receiving result data related to the input data from the distributed data processing system in the distributed data processing system interface component, wherein the result data includes an index related to the first time interval; determining an index related to a second time interval among many time intervals at the first processing node among many processing nodes; and determining an index related to the second time interval at the first processing node among many processing nodes. The method includes the steps of: comparing an indicator with an indicator related to a first time interval contained in the result data in a distributed data processing system interface component, and releasing the result data from the distributed data processing system interface component if the indicator related to a second time interval is the same as or corresponds to a later time interval than the first time interval; processing a large number of requests using two or more processing nodes, wherein each request in each set of requests is configured to trigger a state update in one of the many processing nodes, is associated with a corresponding time interval in the many time intervals, and the large number of requests includes a first set of requests related to a third time interval in the many time intervals; and maintaining a large number of counters in the first processing node of the many processing nodes. The large number of counters includes a work counter indicating the current time interval in the many time intervals in the distributed data processing system, and a replication counter indicating one of the many time intervals, wherein all requests related to that time interval are replicated in multiple processing nodes of the many processing nodes. The method also includes the step of providing a first message containing the values ​​of the work counter and the replication counter from the first processing node of the many processing nodes to other processing nodes in a first time.

[0039] The embodiment may include one or more of the following features:

[0040] The metrics related to the second time interval may be provided to the distributed data processing system interface component. Many counters may include persistent counters that indicate one of many time intervals, where all requests related to that time interval are stored in persistent storage associated with at least one of the many processing nodes.

[0041] In another common embodiment, software for managing a distributed data processing system, which includes many processing nodes, is stored in a non-temporary format on a computer-readable medium. The software includes instructions for a computer system to perform the following: receive input data in a distributed data processing system interface component that communicates with the distributed data processing system; provide the received input data to the distributed data processing system, which then assigns and provides an index to the input data relating to a first time interval among many time intervals; receive result data relating to the input data from the distributed data processing system in the distributed data processing system interface component, which includes an index relating to the first time interval; and determine an index relating to a second time interval among many processing nodes in the first processing node among many processing nodes. The distributed data processing system interface component compares a relevant metric with a metric related to a first time interval contained in the result data, and if the metric related to a second time interval is the same as or corresponds to a later time interval than the first time interval, it releases the result data from the distributed data processing system interface component; it processes many sets of requests using two or more of many processing nodes, each request in each set of requests is configured to trigger a state update in one of the many processing nodes, is associated with a corresponding time interval in the many time intervals, and the many sets of requests include a first set of requests related to a third time interval in the many time intervals; and it maintains many counters in the first processing node of the many processing nodes. Many counters include a work counter indicating the current time interval in the many time intervals in the distributed data processing system, and a replication counter indicating one time interval in the many time intervals, where all requests related to that time interval are replicated in multiple processing nodes of the many processing nodes.The software also includes instructions to cause a computer system to provide a first message, containing the values ​​of a work counter and a replication counter, from one processing node to other processing nodes among many processing nodes in a first time interval.

[0042] In another general embodiment, the device includes a distributed data processing system comprising many processing nodes, each processing node comprising at least one processor and a communication medium connecting many processing nodes for sending and receiving information between the processing nodes. The distributed data processing system is configured as follows: A distributed data processing system interface component communicates with the distributed data processing system and receives input data, provides the received input data to the distributed data processing system, the distributed data processing system assigns an index related to a first time interval among many time intervals to the input data, the distributed data processing system interface component receives result data related to the input data from the distributed data processing system, the result data includes an index related to the first time interval, the first processing node among many processing nodes determines an index related to a second time interval among many time intervals, the distributed data processing system interface component compares the index related to the second time interval with the index related to the first time interval included in the result data, and if the index related to the second time interval is the same as or corresponds to a later time interval than the first time interval, the distributed data processing system interface component releases the result data, processes many sets of requests using two or more of the many processing nodes, and each request in each set of requests is sent to one of the many processing nodes. The system is configured to trigger state updates, associated with corresponding time intervals among many time intervals, with many sets of requests including a first set of requests associated with a third time interval among many time intervals, and holding many counters in a first processing node among many processing nodes. Many counters include a work counter indicating the current time interval among many time intervals in the distributed data processing system, and a replication counter indicating one time interval among many time intervals, where all requests associated with that time interval are replicated in multiple processing nodes of many processing nodes. The system is also configured to provide a first message containing the values ​​of the work counter and the replication counter from a first processing node among many processing nodes to other processing nodes in a first time.

[0043] In another aspect, the process of managing a distributed data processing system, which generally includes many processing nodes, includes the following steps: providing input data to the distributed data processing system, where the distributed data processing system assigns an index to the input data that relates to a first time interval among many time intervals; receiving result data from the distributed data processing system that relates to the input data, where the result data includes an index that relates to the first time interval; determining an index related to a second time interval among many time intervals at a first processing node among many processing nodes; comparing the index related to the second time interval with the index related to the first time interval included in the result data, and releasing the result data if the index related to the second time interval is the same as or later than the first time interval; and processing many sets of data units using two or more processing nodes among many processing nodes, where each data unit in each set of data units is associated with a corresponding time interval among many time intervals, and many sets of data units are associated with a corresponding time interval among many time intervals. A process comprising: a first set of data units relating to a third time interval, wherein for each particular persistence level among many particular persistence levels, the process includes updating a relevant indicator to indicate that all sets of data units relating to a third time interval are stored at that particular persistence level, and many persistence levels include a first persistence level and a second persistence level having a greater level of persistence than the first persistence level; a process of processing many sets of requests using two or more of many processing nodes, wherein each request of each set of requests is configured to trigger a state update at one of the many processing nodes and is associated with a corresponding time interval among many time intervals; a first processing node among many processing nodes holds a replication counter indicating one of the many time intervals, where all requests relating to that time interval are replicated at multiple processing nodes of many processing nodes; and a first message containing the value of the replication counter is provided from the first processing node among many processing nodes to other processing nodes.

[0044] In another aspect, the process of managing a distributed data processing system, which generally includes many processing nodes, includes the following steps: providing input data to the distributed data processing system, wherein the distributed data processing system assigns an index to the input data that relates to a first time interval among many time intervals; receiving result data from the distributed data processing system that relates to the input data, wherein the result data includes an index that relates to the first time interval; determining an index related to a second time interval among many time intervals at a first processing node among many processing nodes; releasing the result data based on a comparison between the index related to the second time interval and the index related to the first time interval included in the result data; and processing many sets of data units using two or more processing nodes among many processing nodes, wherein each data unit in each set of data units is associated with a corresponding time interval among many time intervals, and many sets of data units include a first set of data units related to a third time interval among many time intervals, wherein for each particular persistence level among many specific persistence levels, all sets of data units related to the third time interval are stored at that particular persistence level. A process that includes updating relevant metrics to indicate; a process that processes a large set of requests using two or more of many processing nodes, wherein each request of each set of requests is configured to trigger a state update in one of the many processing nodes, associated with a corresponding time interval in many time intervals, and the state update in one or more of the many processing nodes includes an updated state stored in memory managed using memory management that avoids automatic garbage collection (e.g., explicit memory allocation and deallocation or automatic reference counting); a process that a first processing node of many processing nodes holds a replication counter that indicates one of many time intervals, and all requests associated with that time interval are replicated in multiple processing nodes of many processing nodes; and a process that provides a first message containing the value of the replication counter from the first processing node of many processing nodes to at least one other processing node.

[0045] Various aspects may have one or more of the following advantages.

[0046] In general, several features described herein enable increased computational efficiency of computations (especially those whose fundamental specifications are in the view of graph-based program specifications) compared to the above-described methods in which components (or parallel execution copies of components) are hosted on various servers (for example, a distributed data processing system with many processing nodes can increase the number of records processed per unit of a given computing resource). For example, a call cluster component is used to interface a graph-based program specification with a distributed data processing system, where the computations required by the graph-based program specification are arranged in the graph-based program specification and performed in a distributed manner by the processing nodes in the graph-based program specification. Furthermore, several features described herein provide the ability to adapt to changes in computing resources and computational requirements. The computational methods provided herein allow for adaptation to fluctuations in computing resources available during the execution of one or more graph-based computations and / or fluctuations in the computational load or time-varying loads of various components of such computations, for example, due to the characteristics of the data being processed. For example, embodiments can adapt to the addition or removal of processing nodes from a distributed data processing system (or failure and return to online). One way a distributed data processing system provides adaptability is by managing the replication and persistence of data within the system, which includes maintaining a count of messages sent and received by processing nodes and maintaining an indicator of the time interval over which all messages are replicated and / or persisted within the system.

[0047] Computational methods are also provided that can efficiently utilize computing resources with various characteristics (for example, by using servers with varying numbers of processors per server, varying numbers of processor cores per processor, etc.) and efficiently support both homogeneous and heterogeneous environments. Some of the features described herein can also enable rapid startup of graph-based computation. One aspect that provides such efficiency and adaptability provides proper management of clusters of processing nodes, as described herein.

[0048] The configuration is also advantageously fault-tolerant in that "the distributed data processing system can recover from any processing errors that occur by rolling back processing in a timely manner." The system anticipates many possible rollback scenarios and implements an algorithm to perform a rollback in each of the possible rollback scenarios. [Brief explanation of the drawing]

[0049] [Figure 1] This is a block diagram of a system for processing data. [Figure 2] This is a block diagram of a computing system, including a computer cluster. [Figure 3] This is a schematic diagram of a clock representing various repetitive time intervals. [Figure 4] This is a state transition diagram for manipulating the procedure. [Figure 5] This demonstrates the normal operation of the computing system. [Figure 6] This demonstrates the normal operation of the computing system. [Figure 7] This demonstrates the normal operation of the computing system. [Figure 8] This demonstrates the normal operation of the computing system. [Figure 9] This demonstrates the normal operation of the computing system. [Figure 10] This demonstrates the normal operation of the computing system. [Figure 11] This demonstrates the normal operation of the computing system. [Figure 12] This demonstrates the normal operation of the computing system. [Figure 13] The first rollback procedure is shown below. [Figure 14] The first rollback procedure is shown below. [Figure 15] The first rollback procedure is shown below. [Figure 16] The second rollback procedure is shown below. [Figure 17] The second rollback procedure is shown below. [Figure 18] The second rollback procedure is shown below. [Figure 19] The third rollback procedure is shown below. [Figure 20] The third rollback procedure is shown below. [Figure 21] The third rollback procedure is shown below. [Figure 22] The fourth rollback procedure is shown below. [Figure 23] The fourth rollback procedure is shown below. [Figure 24] The fourth rollback procedure is shown below. [Figure 25] The fourth rollback procedure is shown below. [Figure 26] The fifth rollback procedure is shown below. [Figure 27] The fifth rollback procedure is shown below. [Figure 28] The fifth rollback procedure is shown below. [Figure 29] The fifth rollback procedure is shown below. [Figure 30] The sixth rollback procedure is shown below. [Figure 31] The sixth rollback procedure is shown below. [Figure 32] The sixth rollback procedure is shown below. [Figure 33] The seventh rollback procedure is shown below. [Figure 34] The seventh rollback procedure is shown below. [Figure 35] The seventh rollback procedure is shown below. [Figure 36] The eighth rollback procedure is shown below. [Figure 37] The eighth rollback procedure is shown below. [Modes for carrying out the invention]

[0050] Figure 1 shows an example of a data processing system 200 in which computer cluster management technology may be used. System 200 includes a data source 202 which may include one or more sources of data (such as storage devices or one or more connections to online data streams), each of which may store or serve data in any of a variety of formats (e.g., database tables, spreadsheet files, flat text files, or proprietary formats used by mainframes). The execution environment 204 includes a preprocessing module 206 and an execution module 212. The execution environment 204 may be hosted on one or more general-purpose computers under the control of a preferred operating system, such as one version of the UNIX® operating system. For example, execution environment 204 may include a multiple-node parallel computing environment that includes a configuration of a computer system using multiple processing units (e.g., a central processing unit: CPU) or processor cores, and which is a local system (e.g., a multiprocessor system such as a symmetric multi-processing (SMP) computer), a locally distributed system (e.g., multiple processors connected as a cluster or massively parallel processing (MPP)), or a remote or remotely distributed network (e.g., multiple processors connected via a local area network (LAN) and / or a wide-area network (WAN)), or any combination thereof.

[0051] The preprocessing module 206 can perform any configuration settings that may be required before the program specification (e.g., the graph-based program specification described below) is executed by the execution module 212. The preprocessing module 206 can configure program specifications for receiving data from various types of systems (including various forms of database systems) that can embody the data source 202. The data may be organized as records, each having a value for a field (also called an "attribute," "row," or "column"), possibly including null values. When initially configuring a computer program, such as a data processing application for reading data from the data source, the preprocessing module 206 typically begins with some initial formatting information about the records in the data source. The computer program may be represented in the form of a data flow graph, as described herein. In some situations, the record structure of the data source may not be initially known and may instead be determined after the analysis of the data source or data. Initial information about a record may include, for example, the number of bits representing individual values, the order of fields in the record, and the type of value represented by the bits (e.g., string, signed / unsigned integer).

[0052] The storage device providing the data source 202 may be local to the execution environment 204, for example, it may be stored on a storage medium (e.g., a hard drive 208) connected to the computer hosting the execution environment 204, or it may be remote to the execution environment 204, for example, it may be hosted on a remote system (e.g., a mainframe computer 210) that communicates with the computer hosting the execution environment 204 over a remote connection (e.g., provided by a cloud computing infrastructure).

[0053] The execution module 212 executes program specifications configured and / or generated by the preprocessing module 206 to read input data and / or generate output data. The output data 214 may be returned and stored in the data source 202 or in a data storage system 216 that has access to the execution environment 204, or it may be used otherwise. The data storage system 216 also has access to a development environment 218 in which a developer 220 can develop applications for processing data using the execution module 212.

[0054] In other words, the data processing system 200 is: An optional development environment 218 connected to a data storage 216, the development environment 218 is configured to build data processing applications related to a data flow graph that performs graph-based calculations on data flowing from one or more input datasets to one or more output datasets via a graph that processes graph components, the data flow graph is defined by a data structure in the data storage 216, the data flow graph having many nodes is defined by a data structure and represents graph components connected by one or more links, the links are defined by a data structure and represent data flows between graph components, and the development environment 218 and; An execution environment 212 connected to a data storage 216 and hosted on one or more computers, the execution environment 212 includes a preprocessing module 206 configured to read stored data structures defining a data flow graph and to allocate computer resources (such as processing) for performing calculations on graph components to be assigned to the data flow graph by a preprocessing module 206, The execution environment 204 includes an execution module 212 for scheduling and controlling the execution of assigned calculations or processes so that graph-based calculations are performed. That is, the execution module is configured to read data from a data source 202 and process the data using an executable computer program represented in the form of a data flow graph.

[0055] 1. Computer cluster In most common cases, several computer programs (also referred to herein as “applications”) that use the execution module 212 to process data include a call cluster component that the application uses to access a computer cluster. For example, referring to Figure 2, in a technique for pipelined data processing, the call cluster component 110 interacts with components of computer cluster 120 to process records 103 received by the call cluster component 110 from components within an application that is part of it (e.g., a data flow graph or other form of graph-based program specification), and sends corresponding results 105 to one or more other components of the application that is part of it. For each input record 103, the call cluster component 110 sends a request 113 (e.g., a request to perform a data processing task) to the cluster 120, and after some time, receives a response 115 from the cluster 120 to the request 113. Some time after receiving the response 115, the call cluster component 110 sends a result 105 corresponding to the response 115, usually after it is known that the result of processing the request is sufficiently persistent within the cluster 120.

[0056] A graph-based program specification, of which the call cluster component 110 is a part, is not shown in Figure 2. While Figure 2 shows only a single call cluster component 110, it should be recognized that there can generally be many call cluster components (e.g., each call cluster component participating in the same or different application, such as a dataflow graph) that can interact with the same cluster 120. A graph-based program specification can be implemented, for example, as a dataflow graph as described in U.S. Patent No. 5,966,072, U.S. Patent No. 7,167,850, or U.S. Patent No. 7,716,630, or as a data processing graph as described in U.S. Patent Application Publication No. 2016 / 0062776. Such a dataflow graph-based program specification typically includes computer components corresponding to nodes (vertices) of a graph (called a "dataflow graph"), which are connected by dataflows corresponding to links (directed edges) of the graph. Downstream components, connected to upstream components by dataflow links, receive an ordered stream of input data elements, process the input data elements in the order they are received, and optionally generate one or more corresponding streams of output data elements. In some examples, each component is implemented as a process typically hosted on one of several computer servers. Each computer server may have multiple such component processes that are active at any given time, and an operating system (e.g., Unix) scheduler shares resources (e.g., CPU time and / or processor cores) among the components hosted on that server. In such embodiments, data flow between components may be implemented using the operating system's data communication services (e.g., pipes, TCP / IP sessions, etc.) and a data network connecting the servers. A subset of components typically acts as a source and / or sink for data from the overall computation (e.g., data files, database tables, and external data flows).After component processing and data flow are established, for example, through collaborative processing, the data flows through the entire computer system, performing calculations represented as graphs, which are usually determined by the availability of input data in each component, and scheduling the computer resources of each component.

[0057] Cluster 120 includes multiple cluster components 140, 150a-150c connected by a communication network 130 (shown as the “cloud” in Figure 2, which may have various interconnection topologies such as startup, shared medium, and hypercube). Each cluster component (or simply “component”) has a specific role within the cluster. In some embodiments, each component is hosted on a separate computing resource (e.g., a separate computer server, a separate core of a multicore server). It should be understood that these components represent roles within the cluster, and that in some embodiments, multiple roles may be hosted on a single computing resource, and a single role may be distributed across multiple computing resources.

[0058] In Figure 2, the root component 140 (referred to as "root") performs several synchronization functions, which will be explained in detail below, but does not directly participate in the flow or calculation of the data being processed. Many worker components 150a-150c (hereinafter referred to as "workers") process requests 113 from the call cluster component 110. The data 165 is stored redundantly in storage 160 accessible to each worker 150, and each request 113 may need to access (read and / or write) a specific portion of the data stored in storage 160 and identified by a key within the request 113, which is distributed among a specific subset of workers determined by the key. Of the workers holding the data for the key required for a particular request, one worker is designated as the primary worker on which request 113 is executed (e.g., worker 150a), and the other workers —A worker is designated as a backup worker in the sense that it does not normally or necessarily perform the request, but its versions of the data are updated in the same way as or in accordance with the primary worker.

[0059] In Figure 2, the path of a specific input record 103 (which may be considered or include a data unit to be processed) is shown to enter the call cluster component 110. Next, the corresponding request 113 (which has a data unit) is sent by component 110 to the request's primary worker 150a (worker A). The response 115 from primary worker 150a is sent back not only to the request's backup worker 150b (worker B) but also to the call cluster component 110. Finally, the corresponding result 105 is output or sent from the call cluster component 110. Generally, there may be multiple backup components for each request, but for simplicity of explanation, only a single backup component is shown in many of the following examples.

[0060] As will be further discussed below, the call cluster component 110 buffers the request 113 in a replay buffer 112 and may retransmit the request to cluster 120 as needed to ensure that it has been correctly received and / or processed by cluster 120. Component 110 also buffers the response 115 in an escrow buffer 114 and may receive several redundant copies of the response if an error condition is detected. Generally, component 110 holds the response in an "escrow state" until cluster 120 notifies component 110 that the response 115 is properly held within the cluster (i.e., stored in data storage at a suitable persistence level).

[0061] Route 140 performs synchronization by holding time (interval) values ​​and distributing them to other components, and distributing some of the time values ​​to the call cluster component 110. Referring to Figure 3, the clock 142 of Route 140 holds three times. Time T1 is the current working time or time interval, represented, for example, as an integer value, which is repeatedly updated and incremented, for example, by 1 every second.

[0062] When request 113 is received by cluster 120 from call cluster component 110 and response 115 is generated (or sent) by the cluster, each is associated with the work (T1) time (or, equivalently, the time intervals in which time T1 has the same value (i.e., between increments of T1)) in which they were received and generated (or sent). Route maintains and distributes a second time T2 that lags behind time T1. As will be described in considerable detail below, time T2 represents a time (interval) in which all requests and / or responses generated at or before the time they were sent between components 150a-150c of cluster 120 are replicated (e.g., in volatile memory) in multiple components 150a-150c so that they do not need to be resent in the case of an operation rollback to handle an error. In some examples, replication (e.g., in volatile memory) refers to being stored in data storage with a first level of persistence. The route maintains and distributes a third time (interval) T3 (later than times T1 and T2) that represents the time at which the data 165 is stored in persistent memory and made permanent in at least one or more of the workers 150a to 150c where it is stored, so that all requests and / or responses generated at or before that time do not need to be retransmitted or recalculated in the case of a rollback of an operation to handle a failure of a component in cluster 120. In some examples, storing in persistent memory (e.g., to disk) means storing in data storage with a second level of persistence that is relatively more persistent than the first level of persistence. It should be noted that this can be associated with many different levels of persistence. For example, offsite data storage outside the cluster may have a third level of persistence that is relatively more persistent than the first and second levels. In some examples, the time intervals T1, T2, and T3 are alternatively referred to as "state consistency metrics."

[0063] The mechanism of Route 140 for determining when the replication (T2) time or duration (T3) time should be incremented will be described later in this specification, as will the mechanism for distributing the time (T1-T3) values ​​to workers 150a-150c.

[0064] In normal operation, a request 113 received by cluster 120 is processed by worker 150 identified as the primary worker based on the key of the request's data unit (and generally by one or more backup workers 150), and also identified based on the key of the data required. Referring to Figure 4, this processing can be represented as transitions between various states of the request in the call cluster component 110 and the primary and backup workers 150. Note that "various requests are in various states and are generally processed by various workers depending on the data being referenced, and therefore the call cluster component and any particular worker may have many requests in various states."

[0065] Generally, each key is associated with a corresponding subset of 150 workers, selected in a pseudo-random manner based on the key (e.g., the deterministic function of the key, which unpredictably distributes backup workers for each key value). More generally, and preferably, these subsets overlap with other parts of the subset rather than forming a complete set of worker parts according to the key values.

[0066] When a request 113 with a unique identifier rid (or assigned by the call cluster component) is formed in the call cluster component 110 for each input record 103, the request enters state A of the call cluster component. In the following description, each request 113 is in one of three states (denoted A-C) of the call cluster component, and in one of nine different states (denoted A-I) of each worker 150 that processes the request. After recording the request 113, the call cluster component 110 determines which worker 150 will be assigned to be the primary worker for the request and sends the request 113 to that worker 150 (shown as worker A in Figure 2). Note that in an alternative embodiment, "the call cluster component 110 may not be aware of which worker is the designated primary worker, and therefore the request 113 may be internally routed in the cluster 120 to reach the designated primary worker 150a." Request 113 remains in state A in the call cluster component 110 until the response 115 to the request is returned and received from cluster 120.

[0067] When request 113 is received by the primary worker (labeled Worker A in Figure 2), the request enters state A at the primary worker. The primary worker allocates a request time equal to the current work time T1 (represented by ta) that it was informed of as distributed by route 140 (recognizing that there may be a time delay between when route increments T1 and when the worker becomes aware of this increment). At this state, request 113 is stored in volatile memory 155 associated with the request id, rid, and request time represented as ta in this example, and is designated to await execution at the primary worker. At this state A, the primary worker sends request 113 to one or more backup workers 150 (i.e., determined by key) of the request. In this configuration, the request is ultimately allocated to be executed based on resource allocation according to the time (ta) allocated to the request and optionally the order in which the requests arrive at the main worker. When request 113 begins execution at the main worker, the request enters state B at the main worker. When the process generates response 115 (assuming in this example the work time T1 is tb), the state of the request at the main worker enters state C. In state C, response 115 is stored in volatile memory 156 associated with time tb. As will be further discussed below, any updates to response 115 and data storage 160 at the worker are stored associated with time (here time tb) in a manner that allows for the removal of the effects of conventional rollback time, for example by using a revised database or other form of revised data structure. In this state C, response 115 is sent to both the call cluster component 110 and the backup component 150.

[0068] In the call cluster component 110, when response 115 is received from the primary worker, the request enters state B, where the response is stored in association with a time tb generated by the primary worker. Response 115 is held in the call cluster component in the escrow buffer 114 until an escrow time equal to or greater than tb is received from route 140. Depending on the persistence requirements of the request from the call cluster component, route may provide either replication time T2 or duration T3 as the escrow time for the call cluster component. Upon receiving an escrow time equal to or greater than tb, the call cluster component 110 sends result 105 from the call cluster component, and the corresponding request 113 enters a null state C, where no further recording of request 113 or its response 115 is required (for example, it can be completely erased).

[0069] When backup worker 150 receives request 113 from the primary worker, the backup worker enters state F, which is associated with the original request time ta (even if the current work time T1 has been incremented beyond that), and the request is awaiting a response from the primary worker. When backup worker 150b receives response 115 from the primary worker, and therefore response 115 is copied into the volatile memory 156 of the backup, backup worker 150b enters state G.

[0070] As soon as the primary or backup worker has a newly generated response 115, the primary or backup worker is free to begin the process of saving the response to persistent storage 160, such as a disk-based or non-volatile memory-based database or file system (see states D and H). A journal-based approach may be used in which updates to persistent memory are first journalized in a volatile memory-based journal, and portions of this journal are written to persistent storage 160 from time to time. Note that even when portions of this journal of updates are written to persistent storage 160, those updates are not made permanent (i.e., “committed”) until an explicit indicator of the degree to which the update is considered permanent is written to persistent storage.

[0071] T2 reaches or increments to tb when route 140 determines that all requests and responses related to time tb and earlier times have been replicated by all appropriate workers. After time T2=tb is distributed from route 140 to the primary and backup workers 150, these workers make the responses permanent in persistent storage 160. If the journal of updates via time tb has not yet been written to persistent memory, they are written at that time. More generally, the journal of updates via time tb is written by the time T2 reaches or increments to tb. The worker wrote to persistent storage 160. At this time, the only thing that needs to be done is to complete the task of making the update permanent by recording the metric that "the update via time tb is treated as permanent in the persistent journal." While the primary worker makes the journal permanent, which is probably a short time, the journal is in state D. When the primary worker responds to the request shown in Figure 4 in persistent storage, the primary worker enters state E. Similarly, while the backup worker makes the response permanent, the backup worker is in state H, and when the backup worker makes the response permanent in persistent memory, the backup worker enters state I. When the root worker determines that all responses associated with time tb (and earlier time) are permanent in persistent memory (i.e., all are in state E or I), the root worker increments duration T3 into tb. As described above, in a situation where the escrow time is for a request in a call cluster component and has a duration of T3, route 140 notifies call cluster component 110 that the escrow time has exceeded tb, and call cluster component 110 releases the corresponding result 105 for the request 113 and releases the response 115 to one or more other components in the application (e.g., graph).

[0072] As described above, in normal operation, the route updates the work time T1 when a series of requests 113 from the call cluster component are processed in the cluster, the response 115 is returned to the call cluster component and released from the call cluster component to the graph according to the update of the escrow time T2 or T3. Generally, processing a particular request 113 can take a lot of time "ticks" (e.g., 10 or 100 ticks) of the work time T1, and therefore the cluster can have many requests in progress, each with many different request times associated with them. Furthermore, since the data is distributed among the workers, the load is effectively distributed among the workers according to the key of those requests, so that each worker may have multiple requests working as a primary worker (i.e., in one of states A-E) and also multiple requests working as a backup worker (i.e., in one of states F-I).

[0073] It should be noted that some requests to the cluster for performing tasks use the procedures described herein for replicating tasks and replicating the results of performing those tasks. For example, after a task has been tagged and replicated (and not necessarily made persistent) on a backup worker, this task is initialized on the primary worker. If the task acts on data recording, initialization may involve retaining the original version 1 of the record. Next, the task is in a state where it runs on the primary worker but not on the backup worker. After processing is complete, a modified version 2 of the record exists. Next, task finalization may involve sending the modified version 2 of the record from the primary worker to the backup worker. Then, both the primary and backup workers can erase the original version 1 of the record (along with the replicated task). While each of these steps is reasonably efficient, if the duration of the task is very short, the overhead associated with these initialization and finalization steps may make the task less efficient.

[0074] Alternatively, different procedures may be used for some tasks with relatively short durations ("short tasks"). Short tasks are still tagged and replicated on the backup worker. However, initialization does not need to retain the original version 1 of the record. Instead, after a commit operation indicates that both the short task and its replica are persistently stored on the primary and backup workers, respectively, the short task is executed on both workers. At the end of the execution, without any communication required to send the modified record, the primary worker executes the short task. A copy of the revised version 2 of the record will exist in both the main and backup workers. While redundant processing exists in both workers, this redundancy does not significantly impact efficiency because the task is short. This alternative procedure is useful, for example, if the short task is deterministic and produces the same result, regardless of which worker is performing it.

[0075] 2. Example of normal operation Referring to Figures 5-12, an example of the normal operation of the call cluster component 110 and cluster 120 is shown. In Figure 5, input record 103 arrives at the call cluster component 110, and the call cluster component 110 forms a request 113 for input record 103. The call cluster component 110 associates the request 113 with a unique request identifier rid and stores this in the replay buffer 112 of the call cluster component 110.

[0076] The call cluster component 110 sends request 113 to cluster 120, and request 113 is received by the primary worker 150a (worker A) in cluster 120 at time T1=ta. Request 113 is stored in the primary worker 150a's volatile memory 155, and a request time equal to the current work time (T1=ta) is allocated. The request time of request 113 is provided to the call cluster component 110, and the call cluster component 110 associates the request time (i.e., ta) with request 113 stored in the replay buffer 112 of the call cluster component 110. Request 113 stored in the call cluster component 110's replay buffer 112 is in state A (see Figure 4) and awaits a response from cluster 120. Request 113 stored in the primary worker's volatile memory 155 is in state A and awaits computer resources to be allocated for the execution of request 113.

[0077] Referring to Figure 6, the main worker sends request 113 to backup worker 150b (worker B), and request 113 is stored in the volatile memory 155 of backup worker 150b. Request 113 stored in the volatile memory 155 of backup worker 150b is in state F, waiting to receive a response from the main worker.

[0078] Referring to Figure 7, when the main worker 105 allocates computer resources (for example, the main worker's computer resources or computer resources from another part of the cluster) to request 113, request 113 enters state B in the main worker 105 and begins execution.

[0079] Referring to Figure 8, at time T1=tb, the primary worker 105 completes the execution of request 113. The execution of request 113 generates a response 115, which is stored in the primary worker's volatile memory 156. Response 115 is associated with the request identifier (rid) of request 113 and the time (tb) at which it was generated. The primary worker sends response 115 to the call cluster component 110 and the backup worker 150b, and then request 113 enters state C, waiting for duration T3 to reach tb.

[0080] The call cluster component 110 receives response 115 and stores it in its escrow buffer 114. With the response stored in the escrow buffer 114, result 115 enters state B in the call cluster component 110 and waits for the duration T3 (escrow time in this example) to reach tb. The backup worker 150b receives response 115 and stores it in volatile memory 156. Request 113 in backup worker 150b enters state G and waits for the duration T3 to reach tb.

[0081] Although not shown in Figure 8, the replication time (T2) is set to tb by the response 115 stored (replicated) in the volatile memory 156 of the main worker 150a and the backup worker 150b.

[0082] Referring to Figure 9, once the response 115 is stored in the volatile memory 156 of either or both of the primary worker 150a and the backup worker 150b, the primary worker 150a and the backup worker 150b begin storing the response 115 in their respective persistent storage 160s, while the response 115 is also stored in their respective volatile memories 155 and 156.

[0083] Referring to Figure 10, after response 115 is stored in the primary worker and replicated in the backup worker 150b, the duration (T3) is set to tb. Primary worker 150a and backup worker 150b terminate the permanent storage of response 115 in persistent storage 160. Request 113 stored in the primary worker is in state D, and request 113 stored in backup worker 150b is in state H, with request 113 and response 115 still stored in volatile memory 155 and 156, respectively.

[0084] Referring to Figure 11, in this example the escrow time is duration T3, so with T3 updated to tb, the request 113 stored in the call cluster component 110 enters state C, and the response 115 (associated with time tb) is released from its escrow buffer 114.

[0085] Referring to Figure 12, with the response 115 permanently stored in the persistent storage of the main worker 150a, request 113 enters state E, where neither request 113 nor response 115 is stored in its volatile memory 155, 156, respectively. Similarly, with the response 115 permanently stored in the persistent storage of the backup worker 150b, request 113 enters state I, where neither request 113 nor response 115 is stored in its volatile memory 155, 156.

[0086] 3. Rollback Scenario The state transition diagram in Figure 4 represents normal operation, but it is possible, though infrequent, that messages between workers are not successfully received (i.e., not received properly). Furthermore, it is possible that a worker may restart after losing its volatile memory or that a worker may fail completely to process further requests (i.e., in either the primary or backup role). It should be noted that some embodiments of the data processing system described herein implement all of the rollback scenarios described in this section. It should also be noted that other embodiments of the data processing system may implement one or more of the rollback scenarios described in this section, but not all of them.

[0087] 3.1 Scenario 1: tr <ta First, consider the situation where the cluster determines that there are some inter-worker messages that were not successfully received and are associated with time te. Generally, the root notifies all workers that the time must be "rolled back" to time tr prior to time te (i.e., tr < te, for example tr = te - 1). Even with such a rollback, the result provided by the call cluster component 110 is provided to the application or graph as if the rollback did not occur, and the updates to the data distributed among the workers remain consistent with the result provided by the call cluster component. In particular, the result is not released from the call cluster component 110 to the application or graph until it is stored (e.g., replicated or held) at many nodes (e.g., workers), thereby ensuring that the result is never invalidated without being called. In other words, any rollback that occurs must occur prior to the result being provided to the application or graph by the call cluster component 1 10.

[0088] If the root 140 determines that a rollback must be performed because some inter-worker messages were not successfully received, the root notifies the call cluster component 110 of the rollback time tr. The current time T1 is incremented, and generally, all activities from tr + 1 to T1 - 1, inclusive, are processed as if they did not occur. The effect in the call cluster component 110 is that "all requests (stored in the playback buffer 112) in state B (i.e., having a response time not reached by the escrow time) are returned to state A, and any corresponding response 115 in the escrow buffer 114 is discarded." Next, the requests 113 in state A (either already in state A or returned from state B to state A) are resent to the cluster 120.

[0089] Although execution has not yet started, the impact of the cluster (i.e., at worker 150) on requests replicated between the primary worker and the backup worker (i.e., the primary worker is in state A and the backup worker is in state F) is first considered for the situation where the request has a request time ta greater than the rollback time tr (i.e., tr < ta). For this figure, the current working time is represented by tc. Since ta is greater than tr, the call cluster component cannot assume that the request was correctly replicated, and thus the versions of the request stored in the volatile memories 155 of the primary and backup workers are removed. Request 113 is received from call cluster component 110 in cluster 120 with the same request id, rid, and is associated with the new request time tc. Upon receiving request 113, the primary worker stores request 113 in its volatile memory 155 and is in state A. The primary worker sends request 113 to backup worker 150, and backup worker 150 stores request 113 in its volatile memory 155 and is in state F. Next, further processing at the primary and backup workers proceeds in the manner shown in FIG. 4.

[0090] Note that "if the backup worker does not notice the request before receiving the updated request with time tc from the primary worker, the backup worker also proceeds in the same way due to the request that was not correctly replicated."

[0091] Referring to FIGS. 13 - 15, an example of the first rollback scenario is shown. In FIG. 13, request 113 issued at time ta is stored in playback buffer 112 in call cluster component 110 and is in state A. Request 113 is stored in the volatile memory 155 at the primary worker but is in state A since execution has not yet started. Request 113 is also stored at backup worker 150b and is in state F.

[0092] A rollback request is received to roll back the system to time tr < ta. In FIG. 14, after the rollback request is received, request 113 is removed from the volatile memory 155 of the primary worker 150a and the volatile memory 155 of the backup worker 150b. A new request 113' associated with the same request identifier (rid) as the original request 113 is issued by the call cluster component 110 to the cluster 120. At time tc, the new request 113' is received by the cluster 120 and associated with the request time tc. The cluster 120 notifies the call cluster component 110 of the request time tc associated with the new request 113'. The new request 113' in the playback buffer 112 is in state A.

[0093] In the cluster, the new request 113' is sent to the primary worker. The primary worker 150 a stores the new request 113' together with the request time tc in its volatile memory 155. The new request 113' stored in the volatile memory 155 of the primary worker 150a is in state A.

[0094] Referring to FIG. 15, the primary worker sends the new request 113' to the backup worker 150b. The backup worker 150b stores the new request 113' in its volatile memory 155 and associates it with the request time tc. The updated request 113' stored in the volatile memory 155 of the backup worker is in state F.

[0095] Next, the cluster proceeds according to its normal operation (as described in FIGS. 5 - 12).

[0096] 3.2 Scenario 2: tr < ta, execution has started In the second situation, the request time ta of the previous request is greater than the rollback time tr (i.e., tr < ta), but the request started execution and did not complete execution in the primary worker (i.e., the request is in state B (possibly with partial response 115 calculated) in the primary worker, and the request is in state F in the backup worker). In this case, the execution is terminated, the partial response 115 is discarded in the primary worker and the backup worker (or, the execution is allowed to complete and the response is discarded), and the call cluster component 110 resends the request 113 to the cluster 120. The requests stored in the primary worker and the backup worker return to states A and F, respectively. The primary worker notifies the backup worker of the request in the same way as if the request had not started execution in the primary worker.

[0097] Referring to FIGS. 16-18, an example of a second rollback scenario is shown. In FIG. 16, the request 113 issued at time ta is stored in the playback buffer 112 in the call cluster component 110 and is in state A. The request 113 is stored in the volatile memory 155 in the primary worker 150a and is in state B since it started execution. The request is also stored in the backup worker 150b and is in state F.

[0098] A rollback request is received to roll back the system to time tr < ta. In FIG. 17, after the rollback request is received, the request 113 is removed from the volatile memory 155 of the primary worker 150a and the volatile memory 155 of the backup worker 150b. A new request 113' associated with the same request identifier (rid) as the original request 113 is issued by the call cluster component 110 to the cluster 120. At time tc, the new request 113' is received by the cluster 120 and is associated with the request time tc. The cluster 120 notifies the call cluster component 110 of the request time tc associated with the new request 113'. The new request 113' in the playback buffer 112 is in state A.

[0099] In the cluster, the new request 113’ is sent to the primary worker. The primary worker 150a stores the new request 113’ together with the request time tc in its volatile memory 155. The new request 113’ stored in the volatile memory 155 of the primary worker 150a is in state A.

[0100] Referring to FIG. 18, the primary worker 150a sends the new request 113’ to the backup worker 150b. The backup worker 150b stores the new request 113’ in its volatile memory 155 and associates it with the request time tc. The updated request 113’ stored in the volatile memory 155 of the backup worker is in state F.

[0101] Next, the cluster proceeds according to its normal operation (as described in FIGS. 5-12).

[0102] <0000~416>3.3 Scenario 3: tr < ta < tb, the execution has completed In the third situation, the request time ta of the previous request is again greater than the rollback time tr. However, in this case, we assume that the execution completed at time tb (i.e., tr < ta < tb) and the response was replicated in the backup worker and received in the call cluster component 110. That is, request 113 is in state B in the call cluster component 110, the request is in state C in the primary worker 150a, and request 113 is in state G in the backup worker 150b. Instead of simply ending the ongoing execution as in the second situation, the responses 115 stored in the primary and backup workers are removed. As introduced above with reference to FIG. 4, the response generated at time tb is stored within a revised data structure associated with time tb in such a way that all updates at and after the specific time are removed from the data structure. In this state, by removing all data versions updated later than time tr, the illustrated update of the request made at time tb is necessarily removed, and the request is returned to state A in the primary worker having a request time of tc waiting for execution, and returned to state F in the backup worker waiting for a response from the primary worker. In the call cluster component, the response is discarded and the request is returned to state A.

[0103] Referring to FIGS. 19 - 21, a simple example of the third rollback scenario is shown. In FIG. 19, the request 113 issued at time ta is stored within the playback buffer 112 in the call cluster component 110. The response for the request 115 generated at time tb is stored within the escrow buffer 114. Thus, request 113 is in state B in the call cluster component.

[0104] In the cluster, the request 113 and the response 115 are stored in the volatile memories 155, 156 in the primary worker 150a. Therefore, the request 113 is in state C in the primary worker 150a. The request 113 and the response 115 are also stored in the volatile memories 155, 156 in the backup worker. Therefore, this request is in state G in the backup worker 150b.

[0105] A rollback request is received to roll back the system to time tr < ta < tb. In FIG. 20, after the rollback request is received, the response 115 is removed from the escrow buffer 114 of the call cluster component 110. In the cluster 120, both the request 113 and the response 115 are removed from the volatile memory 155 of the primary worker 150a and the volatile memory 155 of the backup worker 150b.

[0106] A new request 113' associated with the same request identifier (rid) as the original request 113 is issued by the call cluster component 110 to the cluster 120. At time tc, the new request 113' is received by the cluster 120 and associated with the request time tc. The cluster 120 notifies the call cluster component 110 of the request time tc associated with the new request 113'. The new request 113' in the playback buffer 112 is in state A.

[0107] In the cluster, the new request 113' is sent to the primary worker 150a. The primary worker 150a stores the new request 113' together with the request time tc in its volatile memory 155. The new request 113' stored in the volatile memory 155 of the primary worker 150a is in state A.

[0108] Referring to FIG. 21, the primary worker 150a sends the new request 113' to the backup worker 150b. The backup worker 150b stores the new request 113' in its volatile memory 155 and associates it with the request time tc. The updated request 113' stored in the backup worker's volatile memory 155 is in state F.

[0109] Next, the cluster proceeds according to its normal operation (as described in FIGS. 5 - 12).

[0110] 3.4 Scenario 4: ta < tr, Execution has not started In the fourth situation, the rollback time tr is at or after the original request time ta (i.e., ta ≤ tr), and the original request has not started execution. The request is resent to the cluster 120 and placed in the queue for execution after the original requests (i.e., {rid, ta}) in the primary and backup workers. The primary worker executes the original request and generates a response (i.e., {rid, tb}). Next, the primary worker proceeds to start the execution of the resent request (i.e., {rid, tc}), but detects that a response associated with the rid of the resent request already exists and defers the execution of the resent request.

[0111] Referring to FIGS. 22 - 25, an example of the fourth rollback scenario is shown. In FIG. 22, the original request 113 issued at time ta is stored in the playback buffer 112 in the call cluster component 110 and is in state A. The original request 113 is stored in the volatile memory 155 in the primary worker 150a and is in state A since execution has not yet started. The original request 113 is also stored in the backup worker 150b and is in state F.

[0112] A rollback request is received to roll back the system to time ta < tr. In FIG. 23, a new request 113' associated with the same request identifier (rid) as the original request 113 is issued by the call cluster component 110 to the cluster 120. At time tc, the new request 113' is received by the cluster 120 and associated with the request time tc. The cluster 120 notifies the call cluster component 110 of the request time tc associated with the new request 113'. The request 113 in the playback buffer 112 remains in state A.

[0113] In the cluster, the new request 113' is sent to the primary worker 150a. The primary worker 150a receives the new request 113' and places the new request 113' in the queue after the original request 113 for execution. Both the original request 113 and the new request 113' stored in the volatile memory 155 of the primary worker 150a are in state A.

[0114] Referring to FIG. 24, the primary worker 150a sends the new request 113' to the backup worker 150b. The backup worker 150b receives the new request 113' and places the new request 113' in the queue after the original request 113 for execution. Both the original request 113 and the new request 113' stored in the volatile memory 155 of the backup worker 150b are in state F.

[0115] Referring to FIG. 25, the primary worker 150a executes the original request 113 to generate a response 115, and the response 115 is held in its persistent storage 160. As a result, the original request 113 is in state D at the primary worker 150a. The new request 113' has not yet started execution at the primary worker 150a and is therefore in state A.

[0116] The response 115 is also sent to the backup worker 150b and the call cluster component It was provided to 110. The backup worker 150b stored the response 115 in its volatile memory 156 and held this response for its persistent storage 160. Thus, the original request 113 is in state H at the backup worker. The call cluster component 110 stored the response 115 in its escrow buffer 114. The request 113 in the playback buffer 112 of the call cluster component is in state B.

[0117] When the new request 113’ starts execution at the primary worker 150a, the primary worker 150a recognizes that the new request 113’ is associated with the same request identifier rid as the response 115 and thus does not execute the new request 113’ since it is a duplicate. In some examples, the response 115 can be resent to the call cluster component and the call cluster component ignores the response 115 as a duplicate.

[0118] Next, the cluster proceeds according to its normal operation (as described in FIGS. 5 - 12).

[0119] 3.5 Scenario 5: ta < tr, execution has started In the fifth situation, the rollback time tr is at or after the original request time ta (i.e., ta ≤ tr), the original request has started execution but has not completed execution at the primary worker (i.e., the request is in state B at the primary worker and the request is in state F at the backup worker). In this situation, the execution is terminated (or allowed to complete) at the primary and backup workers and the response is discarded (i.e., the requests stored at the primary and backup workers return to states A, F respectively).

[0120] The call cluster component 110 resends the request to cluster 120, where the request cluster 120 is queued for execution after the original request (i.e., {rid, ta}) in the primary worker and the backup worker. The primary worker executes the original request and generates a response (i.e., {rid, tb}). Next, the primary worker proceeds to start executing the resend request (i.e., {rid, tc}), but detects that a response associated with the rid of the resend request already exists and defers the execution of the resend request.

[0121] Referring to FIGS. 26-29, an example of the fifth rollback scenario is shown. In FIG. 26, the original request 113 issued at time ta is stored in the playback buffer 112 in the call cluster component 110 and is in state A. The original request 113 is stored in the volatile memory 155 in the primary worker 150a and is in state B since execution has started. The original request 113 is also stored in the backup worker 150b and is in state F.

[0122] A rollback request is received to roll back the system to time ta < tr. In FIG. 27, a new request 113' associated with the same request identifier (rid) as the original request 113 is issued by the call cluster component 110 to cluster 120. At time tc, the new request 113' is received by cluster 120 and associated with the request time tc. Cluster 120 notifies the call cluster component 110 of the request time tc associated with the new request 113'. The request 113 in the playback buffer 112 remains in state A.

[0123] In cluster 120, the execution of the original request 113 stored in the volatile memory 155 of the primary worker 150a is terminated, and the original request 113 is returned to state A. The new request 113' is sent to the primary worker 150a. The primary worker 150a receives the new request 113' and waits after the original request 113 for the new request 113' to be executed It is placed in the queue. The new request 113' stored in the volatile memory 155 of the main worker 150a is in state A.

[0124] Referring to Figure 28, the primary worker 150a sends a new request 113' to the backup worker 150b. The backup worker 150b receives the new request 113' and places it in the queue after the original request 113 for execution. Both the original request 113 and the new request 113', stored in the backup worker 155's volatile memory 155, are in state F.

[0125] Referring to Figure 29, the primary worker 150a executed the original request 113 and generated a response 115. The response 115 is held in its persistent storage 160. As a result, the original request 113 is in state D in primary worker 150a. The new request 113' has not yet started execution in primary worker 150a and is therefore in state A.

[0126] Response 115 was also replicated to backup worker 150b and call cluster component 110. Backup worker 150b stored response 115 in its volatile memory 156 and retained this response in its persistent storage 160. Thus, the original request 113 is in state H in the backup worker. Call cluster component 110 stored response 115 in its escrow buffer 114, and request 113' in the call cluster component's replay buffer 112 is in state B.

[0127] When the new request 113' is initiated by the primary worker 150a, the primary worker 150a recognizes that the new request 113' is associated with the same request identifier rid as response 115, and therefore does not execute the new request 113' as it is a duplicate. In some cases, response 115 may be resent to the call cluster component 110, which ignores response 115 as a duplicate.

[0128] Next, the cluster proceeds according to its normal operation (as described in FIGS. 5 to 12).

[0129] 3.6 Scenario 6: ta < tb < tr, execution has completed In the sixth situation, the rollback time tr is the requested time ta or later, and the request also completed execution at time tb, also at or before the rollback time (i.e., ta ≤ tb ≤ tr). If the response is successfully provided to the call cluster component 110 (i.e., this request is in state B in the call cluster component), the rollback request does not cause the request to be resent, nor does it cause the removal of any response from the escrow buffer 114. That is, any request associated with ta and any response associated with tb remain unchanged.

[0130] However, if the response is not successfully provided to the call cluster component 110, the call cluster component 110 resends the request to the cluster 120. Upon receiving the resent request, the primary worker starts to execute the resent request (i.e., {rid, tc}), but detects that a response 115 associated with the request identifier rid already exists. Therefore, the resent request is not executed, and the response generated by the execution of the original request is resent to the call cluster component 110. The call cluster component 110 receives a response having the response time tb, which is used to determine in the call cluster component when the response can be sent from escrow.

[0131] Referring to FIGS. 30 to 32, an example of the sixth rollback scenario is shown. In FIG. 30 The original request 113 issued at time ta is stored in the playback buffer 112 in the call cluster component 110. A response 115 to the original request 113 was generated at time tb but did not reach the escrow buffer 114 of the call cluster component 110. Therefore, the request 113 is in state A in the call cluster component 110.

[0132] In the cluster, the request 113 and the response 115 are stored in the volatile memories 155, 156 in the primary worker 150a. Therefore, the request 113 is in state C in the primary worker 150a. The request 113 and the response 115 are also stored in the volatile memories 155, 156 in the backup worker. Therefore, this request is in state G in the backup worker 150b.

[0133] A rollback request is received to roll back the system to time ta < tb < tr. In FIG. 31, a new request 113' associated with the same request identifier (rid) as the original request 113 is issued by the call cluster component 110 to the cluster 120. At time tc, the new request 113' is received by the cluster 120 and associated with the request time tc. The cluster 120 notifies the call cluster component 110 of the request time tc associated with the new request 113'.

[0134] The new request 113' is sent to the primary worker 150a within the cluster 120. The primary worker 150a receives the new request 113' and places the new request 113' in the queue in the volatile memory 155 for execution. The original request 113 stored in the volatile memory 155 of the primary worker 150a remains in state C, and the new request 113' stored in the volatile memory 155 of the primary worker 150a is in state A.

[0135] Referring to FIG. 32, when the primary worker 150a starts to execute a new request, the primary worker 150a recognizes that the new request 113’ has the same request identifier rid as the original request 113, and that the response 115 associated with the request identifier rid already exists in the primary worker 150a. Therefore, the primary worker 150a does not execute the new request 113’, but instead resends the response 115 to the call cluster component 110. The call cluster component 110 receives the response 115 and stores it in the escrow buffer 114. Due to the response 115 stored in the escrow buffer 114 of the call cluster component 110, the call cluster component 110 is in state B.

[0136] Next, the cluster proceeds according to its normal operation (as described in FIGS. 5 - 12).

[0137] 3.7 Scenario 7: ta < tr < tb, execution is completed In the seventh situation, the rollback time tr is at or after the request time ta, the request has completed execution at the time tb after the rollback time (i.e., ta ≤ tr < tb), and the replication of responses between workers may not have been successful. The worker discards all responses 115 that have a time after tr. The request 113 stored in the backup worker returns to state F, and the request 113 stored in the primary worker returns to state B. The call cluster component 110 discards all responses 115 in the escrow buffer 114, returns the request 113 stored in the playback buffer 112 to state A, and resends the request 113 to the cluster 120 that reprocesses the request.

[0138] Referring to FIGS. 33 - 35, an example of the seventh rollback scenario is shown. In FIG. 33, the request 113 issued at time ta is stored in the playback buffer 112 in the call cluster component 110. For the request 115 generated at time tb The response is stored in the escrow buffer 114. Therefore, the request 113 is in state B in the call cluster component 110.

[0139] In cluster 120, the request 113 and the response 115 are stored in the volatile memories 155, 156 in the primary worker 150a. Therefore, the request 113 is in state C in the primary worker 150a. The request 113 is also stored in the volatile memories 155, 156 in the backup worker 105, but the response 115 may not have been successfully replicated to the backup worker 150b. Therefore, the request may not be in state G in the backup worker 150b.

[0140] A rollback request is received to roll back the system to time ta < tr < tb. In FIG. 34, the response 115 stored in the escrow buffer 114 of the call cluster component 110 is removed. A new request 113' associated with the same request identifier (rid) as the original request 113 is issued by the call cluster component 110 to the cluster 120. At time tc, the new request 113' is received by the cluster 120 and associated with the request time tc. The cluster 120 notifies the call cluster component 110 of the request time tc associated with the new request 113'. The new request 113' in the playback buffer 112 is in state A.

[0141] In cluster 120, the backup worker 150b removes any response stored in its volatile memory 156 associated with a time after tr, and thus returns to state F. The primary worker 150a returns to state B. The new request 113' is sent to the primary worker 150a. The primary worker receives the new request 113' and places the new request 113' in the queue after the original request 113 for execution. The new request 113' stored in the volatile memory 155 of the primary worker 150a is in state A.

[0142] In FIG. 35, the primary worker 150a completes the execution of the original request 113 and generates a new response 115' at time td. The primary worker 150a sends the new response 115' to the backup worker 150b and the call cluster component 110, and transitions the state of the original request 113 stored in the volatile memory of the primary worker 150a to state C. The backup worker 150b receives the new response 115', stores the new response 115' in its volatile memory 155, and transitions the original request 113 stored in the volatile memory 155 of the backup worker to state G. The call cluster component 110 receives the new response 115', stores it in the escrow buffer 114, and transitions the new request 113' stored in the playback buffer 112 to state B.

[0143] When the new request 113' starts execution in the primary worker 150a, the primary worker 150a recognizes that the new request 113' has the same request identifier rid as the original request 113, and thus does not execute the new request 113' because it is a duplicate.

[0144] Next, the cluster proceeds according to its normal operation (as described in FIGS. 5 - 12).

[0145] 3.8 Scenario 8: ta < tr < tb, execution has completed Finally, in the eighth situation, there is a worker that processes requests when the primary worker is lost (e.g., known to have failed). Generally, any request in the backup worker (i.e., the backup worker is in state F) waiting for the lost primary worker to provide a response causes the backup worker to be promoted to the primary worker. For example, receiving a reply to a message from the worker If Route 140 detects that a worker has been lost due to a lack of trust, it initiates a rollback to time tr (i.e., tr = T2), which is equal to the time it was last replicated. When the backup worker receives a rollback request to time tr, which may include new partition information to address the lost worker, the backup worker begins working as the new primary worker by changing the state of the request to state A, where resources are waiting to perform the request.

[0146] Referring to Figures 36-37, an example of the eighth rollback scenario is shown. In Figure 36, request 113 issued at time ta is stored in the replay buffer 112 in the call cluster component 110 and is in state A. Request 113 is stored in the volatile memory 155 in the primary worker 150a and is in state B because execution started but did not end. The request is also stored in the backup worker 150b and is in state F. While request 113 is being executed, the primary worker 150a fails or is lost.

[0147] In Figure 37, the root requested a rollback to time tr, which is equal to the time of the last replication. At this time, backup worker 150b is prompted to become primary worker 150a and changes its state to state A. Another worker 150c is assigned as a backup worker in state F.

[0148] Next, the cluster proceeds according to its normal operation (as shown in Figures 5-12).

[0149] 4. Root node Next, moving on to the operation of Route 140, as described above, Route periodically increments the current work time (interval) T1 144. Generally, when updating work time, Route distributes a set of time (T1, T2, T3) 144-146 to all workers (e.g., via broadcast communication). Accordingly, the workers provide Route with information that can be used to update T2 and / or T3 time.

[0150] Each worker maintains a set of counters 151-152 associated with a specific work time. One counter 151 is associated with work time t1 called Sent(t1) and counts the number of communications from that worker for requests with request time t1 sent to the backup worker, and the number of responses with response time t1 sent to the backup worker. In Figure 4, Sent(ta) is updated in state A for each request with request time ta sent to the backup worker, and Sent(tb) is incremented for each response generated at time tb sent for replication in the backup worker. Note that the Sent() counter is not incremented for messages sent from the worker to the call cluster component. Another counter 152, Rec(t1), counts the number of communications received by the worker associated with time t1. In particular, the backup worker increments Rec(ta) when it receives a replication of a request with request time ta when it enters state F, and increments Rec(tb) when it receives a replication of a response generated at time tb when it enters state G. Each worker has its own local copy of these counters (represented by Sendw(t) and Recw(t)) of worker w. It should be obvious that the sum of Sendw(t) across all workers w is equal to the sum of Recw(t) across all workers w, insofar as all communications sent associated with time t1 are also received at their destinations.

[0151] Occasionally, for example, in response to receiving a broadcast message from route 140 for the current time (T1, T2, T3), each worker 150 sends its current count Sent(t)151, Rec(t)152 for all times greater than replication time T2. These counts The signals are received and aggregated at the route, and stored in counters 141 and 142, associated with the corresponding time, so that the route can determine the sum of Sent(t) and Rec(t) for each time t greater than T2. ​​If Sent(T2+1) is equal to Rec(T2+1), then all transmissions from time T2+1 onwards have been received, and T2 is incremented to become the next replication time. This process is repeated until Sent(T2+1) is no longer equal to Rec(T2+1) or T2+1 reaches T1. Then, this incremented T2 time (145) is used in the next broadcast communication from the route.

[0152] As described above, data updates in a worker are first journaled in volatile memory, and this journal is periodically written to persistent storage. Each worker can make the journaled changes in persistent memory permanent for up to a maximum replication time T2. Generally, each worker w has the opportunity to make all changes over time T3(w) permanent, and typically different workers reach different times. In addition to sending Rec() and Sent() back to the root in response to the broadcast communication of the current time, each worker also returns its T3(w) time, which is aggregated according to the min() operation, either at the root or along the communication path back to the root. That is, the root determines T3 = minwT3(w) and distributes this new value of T3 at the next time it distributes the current time.

[0153] In some embodiments, the route distributes time tuples (T1, T2, T3) in direct (e.g., unicast) communication between the route and each worker. In other embodiments, the tuples are distributed in a different manner, such as in flooding-based broadcast communication. In another embodiment, the tuples are distributed along a predetermined tree-structured distribution network in which each recipient of the tuples forwards the tuples to several other recipients, so that eventually all workers receive the time tuples.

[0154] The counts from the workers can be aggregated via unicast communication between each worker and the root node, which then performs a full sum across all workers. As a more efficient solution, the counts can be sent back along the same path as the time set, with intermediate nodes along the path performing partial aggregations of the total counts, thereby distributing the burden of aggregation by the root while still obtaining the total count across all workers.

[0155] In the alternative operating mode, responses may be released from the call cluster component if they are replicated rather than retained. In this way, responses may be served to the graph with less delay, and the responses may not yet be persistent within cluster storage.

[0156] As described above, the response to the execution of a request is stored in a revised data structure. In one such data structure, each update of a data item is stored as a separately recoverable version, and the version is tagged with the time associated with the update. For example, the data structure may be stored at least conceptually for each access key as a list of tuples (tb, value), where tb is the time of the value update. Values ​​for different times may share an underlying structure or use other storage optimizations. In some examples, these values ​​are stored based on the editing of data values ​​between times. As an example, values ​​may be represented as a tree-based structure, and each version may be stored as a "forward" increment operation sufficient to generate the next version from the old version, or as a "backward" increment operation sufficient to reconstruct the old version from the current version. As discussed above, this type of revised data structure allows for the rollback of all updates after a rollback time. Rather than retaining all updates to a data item, only updates to the start of an update time are retained, and as a result, a rollback to any start of an update time can be achieved.

[0157] It should be recognized that "after the root increments replication time T2, workers are not required to roll back to a version at that time or earlier." Therefore, the optimization of the revised data structure means that versions at replication time T2 or earlier can be removed from the data structure.

[0158] In some embodiments, some requests are "lightweight" in the sense that their execution time is short, and therefore, executing requests in the backup worker may consume fewer resources than replicating responses from the primary worker to the backup worker. In such embodiments, no replicas of responses are made from the primary worker to the backup worker. Each worker may complete processing at different times. To maintain data synchronization between workers, the primary worker distributes completion times tb as described above, and the backup worker processes locally calculated responses as if they were calculated at that time.

[0159] In an alternative embodiment, the call cluster component joins the cluster in the sense that it receives time sets from the root and returns the Sent() and Rec() counts to the root. In this embodiment, the call cluster component allocates the request time of the request to be used by the worker during request replication. If a rollback occurs, the call cluster component knows the request time of the request it is holding, so it only needs to resend the request after the rollback time and does not discard responses generated during or before the rollback time. The worker behavior is modified to accommodate this behavior of the call cluster component.

[0160] 5.Alternatives More generally, in the rollback scenarios 4 to 8 where ta < tr, when resending a request, the call cluster component 110 does not notice (nor pay attention to) the fact that the original request was sent at time ta. On the other hand, since the cluster 120 uses this time to determine whether to roll back, it is necessary to consider the request time of the original request. Therefore, when the call cluster component 110 resends a request (having a request identifier rid) to the cluster 120 such that ta < tr < tc, the request is received at the primary worker 150a and associated with time tc. The primary worker 150a transfers the request to the backup worker 150b. In this situation, the primary worker may execute the original request (i.e., {rid, ta}) before executing the resended request (i.e., {rid, tc}). When proceeding to execute the resended request (i.e., {rid, tc}), the primary worker 150a will process the resended request as a duplicate because the response of the original request (i.e., {rid, ta}) is already held.

[0161] In some examples, a request spawns subsequent tasks (sometimes called "task chaining"). In such examples, the response to the request is not generated until the spawned tasks are completed. In some examples, if the response to the request {rid, ta} is stored, the response is returned to the call cluster component. However, if the response to the request {rid, ta} does not yet exist because the request {rid, ta} is not yet complete, the subsequent request {rid, tc} having the duplicate rid is ignored because the cluster knows that "the original request will eventually complete and generate a response (which will be returned to the call cluster component)".

[0162] In the example above, when the cluster receives a request, it associates the request with a time (e.g., ta), and then notifies the call cluster component of this time. The call cluster component then associates this time with the request stored in its response buffer. The time associated with requests in the replay buffer of a call cluster component can be used by a call cluster component to selectively replay requests in the event of a rollback. However, in some examples, neither the cluster nor the call cluster component associates requests with time. In these examples, the call cluster component is not as selective when replaying requests in the event of a rollback scenario. For example, in the event of a rollback request, the call cluster component may systematically replay all requests within its replay buffer.

[0163] 6. Embodiments The computer cluster management techniques described above may be implemented, for example, using a programmable computer system that executes preferred software instructions, or in preferred hardware such as a field-programmable gate array (FPGA), or in some hybrid form. For example, in a programmable technique, the software may include procedures within one or more computer programs that run on one or more programmed or programmable computer systems (which may be of various architectures such as distributed client / server or grid), each including at least one processor, at least one data storage system (including volatile and / or non-volatile memory and / or storage elements), and at least one user interface (for receiving input by using at least one input device or port, and for providing output by using at least one output device or port). The software may include one or more modules of a larger program that provide services related to, for example, the design, configuration, and execution of a data flow graph. Modules of the program (e.g., elements of a data flow graph) may be implemented as data structures or other organized data conforming to a data model stored in a data repository.

[0164] Software may be stored in a non-temporary form, for example, by being embodied in a volatile or non-volatile storage medium or any other non-temporary medium by using the physical properties of the medium (e.g., surface pits and lands, magnetic domains, or electric charge) for a certain period of time (e.g., the time between refresh periods of dynamic memory devices such as dynamic RAM). In preparation for loading instructions, software may be provided on a tangible non-temporary medium such as a CD-ROM or other computer-readable medium (e.g., readable by a general-purpose or special-purpose computer system or device), or it may be delivered over a network communication medium to a tangible non-temporary medium of the computer system on which it is executed (e.g., it may be encoded in a propagated signal). Some or all of the processing may be performed on a special-purpose computer or by using dedicated hardware such as a coprocessor or field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC). Processing may be carried out in a distributed manner, with different parts of the software-defined calculations being performed by different computer elements. Preferably, each such computer program is stored on or downloaded onto a computer-readable storage medium (e.g., solid memory or medium, or magnetic or optical medium) of a storage device accessible by a general-purpose or special-purpose programmable computer for configuring and operating the computer, when the storage device medium is read by the computer to perform the processing described herein. The system of the present invention may also be conceived as a tangible, non-temporary medium configured with computer programs, where such a medium causes the computer system to operate in a specific and predetermined manner to perform one or more of the processing described herein.

[0165] Numerous embodiments of the present invention have been described. Nevertheless, it should be understood that the foregoing description is intended to be illustrative and not to limit the scope of the invention as defined by the appended claims. Accordingly, other embodiments are also included in the following claims. For example, various modifications can be made without departing from the scope of the invention. In addition, some of the steps described above may be order-independent and can therefore be performed in a different order than those described.

Claims

1. 1. A method for processing a task in a distributed data processing system including a plurality of processing nodes, comprising: processing a plurality of sets of tasks using two or more of the plurality of processing nodes, each set of tasks configured to generate a corresponding set of processing result output data and associated with a corresponding time interval of a plurality of time intervals; maintaining a plurality of counters at a first processing node of the plurality of processing nodes, the plurality of counters comprising: a work counter associated with a current time interval of the plurality of time intervals within the distributed data processing system; the maintaining step including a duplicate counter associated with one duplicate time interval of the plurality of time intervals that is earlier than the current time interval, the duplicate counter being duplicated in a plurality of processing nodes of the plurality of processing nodes for at least one of (1) all tasks associated with the duplicate time interval, or (2) all the corresponding processing result output data of the tasks associated with the duplicate time interval; providing a message from the first processing node to other processing nodes of the plurality of processing nodes; updating the work counter and the replication counter based on the message; The method comprising:

2. The method described in claim 1, wherein the processing includes performing calculations related to at least some of the tasks on a primary processing node of the plurality of processing nodes, and performing calculations related to replicas of the at least some of the tasks on one or more backup processing nodes of the plurality of processing nodes.

3. performing a computation associated with a first task on the primary processing node includes generating a first result on the primary processing node; The method of claim 2 , wherein performing computations associated with a replica of the first task at the backup processing node includes generating the first result at the backup processing node.

4. The method described in claim 3, wherein both the computations related to the first task executed on the primary processing node and the computations related to the replica of the first task executed on the backup processing node begin after a commit operation indicating that both the first task and the replica of the first task have been persistently stored.

5. The method described in claim 3, wherein the first result includes a modified version of the original data replicated at the primary processing node and the backup processing node.

6. The method described in claim 3, wherein the calculations associated with the first task executed on the primary processing node are identical to the calculations associated with the replica of the first task executed on the backup processing node.

7. The method described in claim 6, wherein the computations related to the first task executed on the primary processing node and the computations related to the replica of the first task executed on the backup processing node are deterministic and do not depend on which of the multiple processing nodes performs the computations.

8. The method described in claim 2, wherein the replication counter indicates one of the multiple time intervals in which all tasks related to that time interval and all the corresponding processing result output data related to that time interval are replicated in multiple processing nodes among the multiple processing nodes.

9. The method described in claim 1, wherein the processing includes performing calculations related to at least some of the tasks on a primary processing node of the plurality of processing nodes while a replica of the task is not being executed on a backup processing node, and transmitting results corresponding to the tasks for which calculations were performed from the primary processing node to the backup processing node.

10. The method described in claim 9, wherein the step of performing a calculation related to a first task at the primary processing node includes a step of generating a first result, and the first result is transmitted from the primary processing node to the backup processing node after the first task is completed at the primary processing node.

11. The method of claim 10, wherein the first result includes a modified version of the original data replicated at the primary processing node and the backup processing node.

12. The method described in claim 9, wherein the replication counter indicates one of the multiple time intervals in which all tasks related to that time interval and all the corresponding processing result output data related to that time interval are replicated in multiple processing nodes among the multiple processing nodes.

13. Software stored in non-transitory form on a computer-readable medium for processing tasks in a distributed data processing system including a plurality of nodes, the software including instructions for causing a computer system to perform all of the steps set forth in any one of claims 1 to 12, 16 to 18, 20 to 22, and 24 to 26.

14. A distributed data processing system including a plurality of processing nodes, each including at least one processor; a communications medium connecting said plurality of processing nodes for transmitting and receiving information between said plurality of processing nodes, said distributed data processing system comprising: using two or more of the plurality of processing nodes to process a plurality of sets of tasks, each set of tasks configured to generate a corresponding set of processing result output data, the plurality of sets of tasks being associated with a corresponding time interval of a plurality of time intervals; maintaining a plurality of counters at a first processing node of the plurality of processing nodes, the plurality of counters comprising: a work counter associated with a current time interval of the plurality of time intervals within the distributed data processing system; a duplicate counter associated with one duplicate time interval of the plurality of time intervals prior to the current time interval, the duplicate counter being duplicated in a plurality of processing nodes among the plurality of processing nodes for at least one of (1) all tasks associated with the duplicate time interval, or (2) all the corresponding processing result output data of the tasks associated with the duplicate time interval; the distributed data processing system provides a message from the first processing node to other processing nodes of the plurality of processing nodes; The apparatus is configured to update the work counter and the replication counter based on the message.

15. A computer system for processing tasks in a distributed data processing system including a plurality of processing nodes, comprising: means for processing, using two or more of the plurality of processing nodes, a plurality of sets of tasks each configured to generate a corresponding set of processing result output data, the plurality of sets of tasks being associated with corresponding time intervals of a plurality of time intervals; Means for maintaining a plurality of counters at a first processing node of the plurality of processing nodes, the plurality of counters comprising: a work counter associated with a current time interval of the plurality of time intervals within the distributed data processing system; the retaining means including a duplicate counter associated with one duplicate time interval of the plurality of time intervals prior to the current time interval, wherein at least one of the corresponding processing result output data of (1) all tasks associated with the duplicate time interval or (2) all of the tasks associated with the duplicate time interval is duplicated in a plurality of processing nodes of the plurality of processing nodes; means for providing a message from the first processing node to other processing nodes of the plurality of processing nodes; means for updating the work counter and the duplication counter based on the message; The computer system.

16. The method of claim 1, maintaining a plurality of data storages within the system, each data storage of the plurality of data storages associated with a corresponding one of the plurality of processing nodes and associated with one persistence level of a plurality of persistence levels, the plurality of persistence levels including a first persistence level and a second persistence level having a greater level of persistence compared to the first persistence level; using the two or more of the plurality of processing nodes to process a plurality of sets of data units, each data unit of each set of data units being associated with a corresponding time interval of the plurality of time intervals, the plurality of sets of data units including a first set of data units associated with a first time interval of the plurality of time intervals, the processing including, for each particular persistence level, updating an associated indicator to indicate that all of the set of data units associated with the first time interval are stored at the particular persistence level; the processing of the plurality of sets of tasks includes processing a plurality of sets of requests using the two or more of the plurality of processing nodes, each request of each set of requests configured to cause a state update in one processing node of the plurality of processing nodes and associated with a corresponding time interval of the plurality of time intervals, the plurality of sets of requests including a first set of requests associated with a second time interval of the plurality of time intervals.

17. The method of claim 16, wherein the plurality of counters further includes a persistent counter indicating one of the plurality of time intervals, wherein all requests associated with that time interval are stored in persistent storage associated with at least one of the plurality of processing nodes.

18. The method described in claim 16, wherein, for each data unit of the first set of data units, storing the data unit in some of the plurality of data storages associated with each processing node of the plurality of processing nodes includes storing the data unit in some of the plurality of data storages associated with the first level of persistence and storing the data unit in one or more of the plurality of data storages associated with the second level of persistence.

19. The apparatus of claim 14, wherein the distributed data processing system: maintaining a plurality of data storages within the system, each data storage of the plurality of data storages associated with a corresponding one of the plurality of processing nodes and associated with one persistence level of a plurality of persistence levels, the plurality of persistence levels including a first persistence level and a second persistence level having a greater level of persistence compared to the first persistence level; processing a plurality of sets of data units using two or more processing nodes of the plurality of processing nodes, each data unit of each set of data units being associated with a corresponding time interval of a plurality of time intervals, the plurality of sets of data units including a first set of data units associated with a first time interval of the plurality of time intervals, the processing including, for each particular persistence level, updating an associated indicator to indicate that all of the set of data units associated with the first time interval are stored at the particular persistence level; said distributed data processing system configured to process said plurality of sets of tasks; the apparatus configured to process a plurality of sets of tasks using two or more of the plurality of processing nodes, wherein each request in each set of requests is configured to cause a state update in one processing node of the plurality of processing nodes and is associated with a corresponding time interval of the plurality of time intervals, the plurality of sets of requests including a first set of requests associated with a second time interval of the plurality of time intervals.

20. The method of claim 1, receiving input data at a distributed data processing system interface component in communication with the distributed data processing system; providing the received input data to the distributed data processing system, the distributed data processing system assigning to the input data an index associated with a first time interval of the plurality of time intervals; receiving, at the distributed data processing system interface component, result data associated with the input data from the distributed data processing system, the result data including an index associated with the first time interval; determining, at a first processing node of the plurality of processing nodes, an index associated with a second time interval of the plurality of time intervals; comparing, in the distributed data processing system interface component, the indicator associated with the second time interval with the indicator associated with the first time interval included in the results data, and releasing the results data from the distributed data processing system interface component if the indicator associated with the second time interval corresponds to a time interval that is the same as or later than the first time interval; maintaining a plurality of data storages within the system, each data storage of the plurality of data storages being associated with a corresponding one of the plurality of processing nodes and associated with one persistence level of a plurality of persistence levels, the plurality of persistence levels including a first persistence level and a second persistence level having a greater level of persistence compared to the first persistence level; the method further comprising: processing a plurality of sets of data units using the two or more processing nodes of the plurality of processing nodes, each data unit of each set of data units being associated with a corresponding time interval of the plurality of time intervals, the plurality of sets of data units including a first set of data units associated with a third time interval of the plurality of time intervals, the processing including, for each particular persistence level, updating an associated indicator to indicate that all of the sets of data units associated with the third time interval are stored at the particular persistence level.

21. The method described in claim 20, wherein, for each data unit of the first set of data units, storing the data unit in some of the plurality of data storages associated with each processing node of the plurality of processing nodes includes storing the data unit in some of the plurality of data storages associated with the first level of persistence, and storing the data unit in one or more of the plurality of data storages associated with the second level of persistence.

22. The method of claim 20, wherein the indicator associated with the second time interval is provided to the distributed data processing system interface component.

23. The apparatus of claim 14, wherein the distributed data processing system: receiving input data at a distributed data processing system interface component in communication with the distributed data processing system; providing the received input data to the distributed data processing system, the distributed data processing system assigning to the input data an index associated with a first time interval of a plurality of time intervals; receiving, at the distributed data processing system interface component, result data associated with the input data from the distributed data processing system, the result data including an index associated with a first time interval; determining, at a first processing node of the plurality of processing nodes, an index associated with a second time interval of the plurality of time intervals; comparing, in the distributed data processing system interface component, the indicator associated with the second time interval with the indicator associated with a first time interval included in the results data, and releasing the results data from the distributed data processing system interface component if the indicator associated with the second time interval corresponds to a time interval that is the same as or later than the first time interval; maintaining a plurality of data storages in the system, each data storage of the plurality of data storages being associated with a corresponding one of the plurality of processing nodes and associated with one persistence level of a plurality of persistence levels, the plurality of persistence levels including a first persistence level and a second persistence level having a greater level of persistence than the first persistence level; the apparatus further configured to: process a plurality of sets of data units using the two or more processing nodes of the plurality of processing nodes, each data unit of each set of data units being associated with a corresponding time interval of the plurality of time intervals, the plurality of sets of data units including a first set of data units associated with a third time interval of the plurality of time intervals, the processing including, for each particular persistence level, updating an associated indicator to indicate that all of the sets of data units associated with the third time interval are stored at the particular persistence level.

24. The method of claim 1, receiving input data at a distributed data processing system interface component in communication with the distributed data processing system; providing the received input data to the distributed data processing system, the distributed data processing system assigning to the input data an index associated with a first time interval of a plurality of time intervals; receiving, at a distributed data processing system interface component, result data associated with said input data from said distributed data processing system, said result data including an index associated with said first time interval; determining, at the first processing node of the plurality of processing nodes, an index associated with a second time interval of the plurality of time intervals; comparing, in the distributed data processing system interface component, the indicator associated with the second time interval with the indicator associated with the first time interval included in the results data, and releasing the results data from the distributed data processing system interface component if the indicator associated with the second time interval corresponds to a time interval that is the same as or later than the first time interval; processing the plurality of sets of tasks includes processing the plurality of sets of tasks using the two or more of the plurality of processing nodes, wherein each request of each set of requests is configured to cause a state update in one processing node of the plurality of processing nodes and is associated with a corresponding time interval of the plurality of time intervals, the plurality of sets of requests including a first set of requests associated with a third time interval of the plurality of time intervals.

25. The method of claim 24, wherein the indicator associated with the second time interval is provided to the distributed data processing system interface component.

26. The method of claim 24, wherein the plurality of counters further includes a persistent counter indicating one of the plurality of time intervals, wherein all requests associated with that time interval are stored in persistent storage associated with at least one of the plurality of processing nodes.

27. The apparatus of claim 14, wherein the distributed data processing system comprises: receiving input data at a distributed data processing system interface component in communication with the distributed data processing system; providing the received input data to the distributed data processing system, the distributed data processing system assigning to the input data an index associated with the first time interval of the plurality of time intervals; receiving, at the distributed data processing system interface component, result data associated with the input data from the distributed data processing system, the result data including an index associated with a first time interval; determining, at a first processing node of the plurality of processing nodes, an index associated with a second time interval of the plurality of time intervals; comparing, in the distributed data processing system interface component, the indicator associated with the second time interval with the indicator associated with a first time interval included in the results data, and releasing the results data from the distributed data processing system interface component if the indicator associated with the second time interval corresponds to a time interval that is the same as or later than the first time interval; and processing a plurality of sets of tasks using the two or more of the plurality of processing nodes, wherein each request of each set of requests is configured to cause a state update in one processing node of the plurality of processing nodes and is associated with a corresponding time interval of the plurality of time intervals, the plurality of sets of requests including a first set of requests associated with a third time interval of the plurality of time intervals.

28. The apparatus described in claim 14, wherein the distributed data processing system configured to process the plurality of sets of tasks is configured to perform computations related to at least some of the tasks on a primary processing node of the plurality of processing nodes and to perform computations related to replicas of at least some of the tasks on one or more backup processing nodes of the plurality of processing nodes.

29. The distributed data processing system configured to perform computations related to a first task at the primary processing node, the distributed data processing system configured to generate a first result at the primary processing node; performing a computation associated with a replica of the first task at the backup processing node includes generating the first result at the backup processing node; 29. The apparatus of claim 28.

30. The apparatus of claim 29, wherein both the computations associated with the first task executed on the primary processing node and the computations associated with the replica of the first task executed on the backup processing node begin after a commit operation indicating that both the first task and the replica of the first task have been persistently stored.

31. The apparatus of claim 29, wherein the first result includes a modified version of the original data replicated at the primary processing node and the backup processing node.

32. The apparatus described in claim 29, wherein the calculations associated with the first task executed on the primary processing node are identical to the calculations associated with the replica of the first task executed on the backup processing node.

33. The apparatus described in claim 32, wherein the computations related to the first task executed on the primary processing node and the computations related to the replica of the first task executed on the backup processing node are deterministic and do not depend on which of the plurality of processing nodes performs the computations.

34. The apparatus described in Claim 28, wherein the replication counter indicates one of the plurality of time intervals in which all tasks associated with that time interval and all the corresponding processing result output data associated with that time interval are replicated at multiple processing nodes among the plurality of processing nodes.

35. The apparatus of claim 14, wherein the distributed data processing system configured to process the plurality of sets of tasks is configured to perform computations related to at least some of the tasks on a primary processing node of the plurality of processing nodes while replicas of the tasks are not being executed on a backup processing node, and to transmit results corresponding to the tasks for which computations were performed from the primary processing node to the backup processing node.

36. The apparatus described in claim 35, wherein the distributed data processing system configured to perform calculations related to a first task at the primary processing node is configured to generate a first result, and the first result is transmitted from the primary processing node to the backup processing node after the first task is completed at the primary processing node.

37. The apparatus of claim 36, wherein the first result includes a modified version of original data replicated at the primary processing node and the backup processing node.

38. The apparatus described in Claim 35, wherein the replication counter indicates one time interval among the plurality of time intervals in which all tasks associated with that time interval and all the corresponding processing result output data associated with that time interval are replicated at multiple processing nodes among the plurality of processing nodes.

Images