Split-based hosting in distributed computing systems

Abstract

A method for fault-tolerant processing of a plurality of data elements using a distributed computing cluster. The distributed computing cluster includes a plurality of data processors associated with a corresponding plurality of data stores. The method includes storing data elements in the distributed computing cluster, wherein the data elements are distributed across the data stores according to a plurality of partitions of the data elements; processing, using a first data processor, a first set of partitioned data elements stored at a first data store to generate first result data for the first set of partitioned data elements; sending the first result data from the distributed computing cluster to a processing component for the first result data that is external to the distributed computing cluster; and storing the first result data in a first buffer located in the distributed computing cluster and associated with the first data processor until the processing component has persistently stored the first result data external to the distributed computing cluster.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 609,517, filed December 13, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to fault tolerance in distributed computing systems. Background Technology

[0004] Distributed computing systems can include computing clusters implemented as networks of interconnected computing devices (sometimes called nodes) that work together to solve a complex task by breaking it down into smaller subtasks and processing them simultaneously. By leveraging the collective power of multiple computers, distributed computing clusters achieve better performance and efficiency than a single computing device can provide.

[0005] With improved performance and efficiency comes increased complexity. One source of increased complexity in distributed computing systems is that data is fragmented across different nodes in the computing cluster and processed at different nodes. Various solutions exist to ensure fault tolerance, data consistency, and coordination among the components of complex distributed computing systems. Summary of the Invention

[0006] This article describes various aspects of fault-tolerant schemes to prevent data loss in the event of component failure in a distributed computing cluster. An example of such a component is the "DoAll" component in a data flow graph that interacts with the distributed computing cluster (where the data flow graph can be implemented as a processing component, such as a networked device with processes running on it). Typically, the DoAll component enables the distributed computing cluster to process collections of data stored within the cluster. These collections are stored in a distributed manner across multiple compute nodes (sometimes called "data processors") within the cluster. Processing the collections in the cluster involves using a "ForAll" procedure to process all data elements by applying a function (e.g., f()) to all data elements stored at the compute nodes. The processed data is then returned to the DoAll component via the compute nodes, which subsequently releases the processed data to downstream components in the data flow graph.

[0007] In the event of a component failure in a distributed computing system, processed data may be lost. For example, if the DoAll component fails, data processed by nodes in the cluster but not delivered to the DoAll component may be lost. The aspects described in this paper implement a managed scheme for storing processed data in an escrow buffer associated with the ForAll process. In the event of a DoAll component failure, the processed data can be retrieved from this escrow buffer (without reprocessing). For large datasets, the escrow buffer can become quite large. To mitigate the impact of large datasets on the escrow buffer, the dataset is over-segmented at each compute node, and the over-segmented data is processed one segment at a time (and stored in escrow), thereby reducing the required size of the escrow buffer.

[0008] In a general sense, a method for fault-tolerant processing of multiple data elements using a distributed computing cluster, the distributed computing cluster including multiple data processors associated with corresponding multiple data repositories, the method comprising: storing data elements in the distributed computing cluster, wherein the data elements are distributed across data repositories according to multiple partitions of the data elements; processing the data elements in a first set of partitions stored at a first data repository using a first data processor to generate first result data of the first set of partitions of the data elements; sending the first result data from the distributed computing cluster to a processing component (sometimes referred to as a “consumer”) of the first result data outside the distributed computing cluster (e.g., a data flow graph including a processing component referred to herein as a “DoAll” component); and storing the first result data in a first buffer (sometimes referred to herein as a “managed buffer”) located in the distributed computing cluster and associated with the first data processor until the processing component has persistently stored the first result data outside the distributed computing cluster.

[0009] Each aspect may include one or more of the following features.

[0010] This processing may include applying the same function (f) to individual data elements across multiple data elements. Over-splitting of data elements is possible within the cluster. The processing component can be a data flow graph, and more specifically, a DoAll component within the data flow graph. Results may also be stored in additional managed buffers at the DoAll component. Over-splitting of results is possible in the DoAll managed buffer, just as it is in the data engine managed buffer.

[0011] The method may include removing the first result data from a first managed buffer after the consumer has persistently stored all result data associated with the first segment outside the distributed computing cluster. At least some of the multiple data repositories may include data elements of two or more segments from a plurality of data elements. The consumer may include a data flow graph containing consumer components. The consumer component in the data flow graph may include a second managed buffer for storing result data, and the method further includes storing the first result data in the second managed buffer. The first result data may be released from the second managed buffer based on an indication that the computing cluster has persistently stored state associated with the first result data. The method may include removing the first result data from the second managed buffer after the consumer has released all result data of the first segment from the second managed buffer and has persistently stored state information for the data flow graph.

[0012] The method may include: resending the first result data from the distributed computing cluster to the consumer based on the determination that the consumer encountered a failure before persistently storing the first result data outside the distributed computing cluster. Resending the first result data may include reading the first result data from a first managed buffer associated with the first data processor.

[0013] The method may include: determining that a first data processor has encountered a failure, and based on this determination, activating a replica of the first data processor, and restoring the consumer to its state prior to receiving first result data from the distributed computing cluster. The method may also include: using the replica of the first data processor to process a first set of segmented data elements to generate regenerated result data of the first set of segmented data elements, sending the regenerated result data from the distributed computing cluster to the consumer, and storing the regenerated result data in a first managed buffer located in the distributed computing cluster and associated with the replica of the first data processor until the consumer has persistently stored the regenerated result data outside the distributed computing cluster.

[0014] Processing the first set of segmented data elements may include applying the same function to each data element. Processing may include marking each processing result in the first result data with a segment number and the value of a counter associated with the cluster. The method may include: in response to a predefined number of data elements having been processed in the distributed computing cluster, incrementing a counter associated with the cluster and sending a message to the processing component indicating that a checkpoint indicated by the counter has been reached. The method may include: determining that a checkpoint has been reached based on the number of data elements that have been processed by the data processor since the last increment of the counter, or by determining whether a predetermined time interval has elapsed since the last increment of the counter.

[0015] The method may include: at a first data processor, receiving a message from a processing component indicating that all data elements associated with the current value of a counter have been removed from the processing component; and in response to receiving the message, removing first result data from a first buffer. The method may also include at the first data processor receiving a message from the processing component requesting the first data processor to resend the first result data to the processing component, and the first data processor sending the first result data to the processing component.

[0016] The method may include determining, by a first data processor, that a second data processor has experienced an operational failure and, in response to determining the failure, replicating the second data processor. Specifically, the operational failure is detected based on a failure-indicating message sent from a second data engine or the second data engine's failure to respond to messages periodically sent by the first data processor. Replicating the second data processor may include: identifying another data processor among a plurality of data processors via the first data processor, particularly by identifying a data processor that responds to messages within a threshold time and / or reports available capacity upon request; and sending a message to the identified data processor requesting it to update its data elements according to a state reflected by a previous value of a first counter, the data elements being associated with a segment previously allocated to the second data processor.

[0017] In another general aspect, a system for fault-tolerant processing of multiple data elements using a distributed computing cluster, the distributed computing cluster including multiple data processors associated with corresponding multiple data repositories, the system comprising: multiple data repositories for storing the multiple data elements, wherein the multiple data elements are distributed across the multiple data repositories according to multiple partitions of the data elements; multiple data processors for processing the data elements, the multiple data processors including: a first processor for processing a first set of partitions of the multiple partitions stored at a first data repository in the multiple data repositories to generate first result data of the data elements of the first set of partitions; an output unit for sending the first result data from the distributed computing cluster to a consumer of the first result data outside the distributed computing cluster; and a first managed buffer located in the distributed computing cluster and associated with the first data processor for storing the first result data until the consumer has persistently stored the first result data outside the distributed computing cluster.

[0018] In another general aspect, a computer-readable medium stores software in a non-transitory form, the software comprising instructions for causing a computing system to process a plurality of data elements in a fault-tolerant manner using a distributed computing cluster, the distributed computing cluster including a plurality of data processors associated with corresponding plurality of data repositories. The instructions cause the computing system to: store the plurality of data elements in the distributed computing cluster, wherein the plurality of data elements are distributed across the plurality of data repositories according to a plurality of partitions of the data elements; process, using a first data processor of the plurality of data processors, a first set of partitions of the data elements stored at a first data repository in the plurality of data repositories to generate first result data of the first set of partitions of the data elements; send the first result data from the distributed computing cluster to a consumer of the first result data outside the distributed computing cluster; and store the first result data in a first managed buffer located in the distributed computing cluster and associated with the first data processor until the consumer has persistently stored the first result data outside the distributed computing cluster.

[0019] In another general aspect, a system is configured for fault-tolerant processing of multiple data elements using a distributed computing cluster, the distributed computing cluster including multiple data processors associated with corresponding multiple data repositories. The system includes: components for storing the multiple data elements, wherein the multiple data elements are distributed across the multiple data repositories according to multiple partitions of the data elements; components for processing the data elements, the multiple data processors including: a first processor for processing a first set of partitions of the multiple partitions stored at a first data repository in the multiple data repositories to generate first result data of the data elements of the first set of partitions; components for sending the first result data from the distributed computing cluster to a consumer of the first result data outside the distributed computing cluster; and a storage component located in the distributed computing cluster and associated with the first data processor for storing the first result data until the consumer has persistently stored the first result data outside the distributed computing cluster.

[0020] Each aspect may possess one or more of the following advantages.

[0021] The various aspects advantageously achieve fault tolerance in distributed computing systems by using a managed solution to store result data until it is determined that the result data will not need to be reproduced due to failure of components in the system (e.g., data flow graph components or data processing components in the computing cluster). The various aspects further achieve the advantage of reducing the size of the managed buffer for large datasets, where the managed buffer can become quite large. The various aspects mitigate the impact of large datasets on the managed buffer by oversplitting the dataset at each computing node and processing one oversplitter at a time (which is also stored in the managed buffer), thereby reducing the required size of the managed buffer.

[0022] Other features and advantages of the invention will be apparent from the following description and the claims. Attached Figure Description

[0023] Figure 1 It is a data processing system.

[0024] Figure 2 The first step in the DoAll component of the data flow graph for processing a collection of data stored in a distributed computing cluster is shown.

[0025] Figure 3 The second step in the DoAll component, which uses a distributed computing cluster to process a dataset, is shown.

[0026] Figure 4The third step in the DoAll component, which uses a distributed computing cluster to process a dataset, is shown.

[0027] Figure 5 The fourth step in the DoAll component, which uses a distributed computing cluster to process a dataset, is shown.

[0028] Figure 6 The fifth step in the DoAll component, which uses a distributed computing cluster to process a dataset, is shown.

[0029] Figure 7 The sixth step in the DoAll component, which uses a distributed computing cluster to process a dataset, is shown.

[0030] Figure 8 The seventh step in the DoAll component, which uses a distributed computing cluster to process a dataset, is shown.

[0031] Figure 9 The eighth step in the DoAll component, which uses a distributed computing cluster to process a dataset, is shown.

[0032] Figure 10 The DoAll component is shown as failing during the ninth step of processing the data set.

[0033] Figure 11 It shows how to reproduce Figure 10 The results determined in the ninth step are used to recover from the failure of the DoAll component.

[0034] Figure 12 The tenth step in the DoAll component, which uses a distributed computing cluster to process a dataset, is shown.

[0035] Figure 13 The eleventh step in the DoAll component, which uses a distributed computing cluster to process a dataset, is shown.

[0036] Figure 14 The data engine that failed during the twelfth step of processing the dataset is shown.

[0037] Figure 15 This demonstrates recovery from a data engine failure during the twelfth step of processing the dataset.

[0038] Figure 16 This illustrates step thirteen of the DoAll component, which uses a distributed computing cluster to process a dataset.

[0039] Figure 17 The fourteenth step in the DoAll component, which uses a distributed computing cluster to process a data set, is shown.

[0040] Figure 18 The fifteenth step in the DoAll component, which uses a distributed computing cluster to process a data set, is shown.

[0041] Figure 19 This illustrates the sixteenth and final step in the DoAll component, which uses a distributed computing cluster to process a dataset. Detailed Implementation

[0042] Figure 1 An example of a data processing system 100 that can utilize computing cluster management technology is shown. System 100 includes a data source 102, which may include one or more data sources (such as storage devices or connections to online data streams), each of which can store or provide data in any of a variety of formats (e.g., database tables, spreadsheet files, plain text files, or native formats used by mainframes). Execution environment 104 includes a preprocessing module 106 and an execution module 112. Execution environment 104 may be hosted on one or more general-purpose computers, for example, under the control of a suitable operating system (such as a version of UNIX). For example, execution environment 104 may include a multi-node parallel computing environment, which includes a configuration of a computer system using multiple processing units (e.g., central processing units (CPUs)) or processor cores, wherein the multiple processing units or processor cores are local (e.g., a multiprocessor system such as a symmetric multiprocessing (SMP) computer), or locally distributed (e.g., multiple processors coupled as a cluster or massively parallel processing (MPP) system), or remote, or remotely distributed (e.g., multiple processors coupled via a local area network (LAN) and / or a wide area network (WAN), or any combination thereof.

[0043] The preprocessing module 106 can perform any configuration tasks that may be required before the execution module 112 executes the program specification (e.g., a graph-based program specification described below). The preprocessing module 106 can configure the program specification to receive data from various types of systems that can represent the data source 102, including different forms of database systems. Data can be organized as records with values ​​for corresponding fields (also referred to as "attributes," "rows," or "columns"), including potentially null values. When first configuring a computer program (such as a data processing application) to read data from a data source, the preprocessing module 106 typically begins with some initial formatting information related to the records in that data source. The computer program can be represented in the form of a data flow diagram as described herein. In some cases, the record structure of the data source may not be initially known but can be determined after analysis of the data source or the data. Initial information related to the records may include, for example, the number of bits representing different values, the order of fields within the record, and the type of value represented by bits (e.g., string, signed / unsigned integer).

[0044] The storage device providing the data source 102 may be local to the execution environment 104, for example, stored on a storage medium (e.g., hard disk drive 108) connected to the computer hosting the execution environment 104, or it may be remote relative to the execution environment 104, for example, hosted on a remote system (e.g., mainframe 110) that communicates with the computer hosting the execution environment 104 via a remote connection (e.g., provided by cloud computing infrastructure).

[0045] Execution module 112 executes the program specifications configured and / or generated by preprocessing module 106 to read input data and / or generate output data. Output data 114 can be stored back in data source 102 or in data storage system 116 accessible to execution environment 104, or used in other ways. Data storage system 116 can also be accessed by development environment 118, in which developer 120 can develop applications for using execution module 112 to process data.

[0046] Very generally, some computer programs (e.g., data flow graphs) used to process data using execution module 112 include components that access a computing cluster. For example, and as described in more detail below, refer to... Figure 2 In data flow graph 111, the DoAll component 110 interacts with computing cluster 120 to process a collection 113 of data elements 114 (e.g., records) stored in computing cluster 120. The result of this processing is returned to the DoAll component 110, which then sends the result downstream to one or more other components in data flow graph 111.

[0047] 1 Data Flow Diagram

[0048] For simplicity, data flow diagram 111 is shown below. Figure 2 Only a portion is shown (i.e., the area above the dashed line), and it should be noted that data flow graph 111 typically includes additional components. More generally, graph-based program specifications can be implemented, for example, as data flow graphs described in U.S. Patent Nos. 5,966,072, 7,167,850, or 7,716,630, or data processing graphs as described in U.S. Publication No. 2016 / 0062776. Such data flow graph-based program specifications typically include computational components corresponding to nodes (vertices) of the graph (referred to as the “data flow graph”), which are coupled via data flows corresponding to links (directed edges) in the graph. Downstream components connected to upstream components via data flow links receive ordered streams of input data elements and process the input data elements in the received order, optionally generating one or more corresponding streams of output data elements. In some examples, the components are implemented as processes hosted on one of typically multiple computer servers. Each computer server can have multiple such component processes active at any given time, and the operating system (e.g., Unix) scheduler shares resources (e.g., processor time and / or processor cores) among the components managed on that server. In such an implementation, data flow between components can be implemented using the operating system's data communication services and data networks connecting the servers (e.g., named pipes, TCP / IP sessions, etc.). A subset of components typically serves as the source and / or destination of data from the entire computation, for example, to and / or from data files, database tables, and external data streams. After component processes and data flows are established, for example, through a coordinating process, the data then flows through the entire computational system used to implement computations represented as a graph, which is governed by the availability of input data at each component and the scheduling of computational resources for each component.

[0049] 2 Computing Cluster

[0050] The computing cluster 120 includes a communication network 130 (in Figure 2 The data engine 122 (sometimes called a "data processor") is coupled together and can have various interconnect topologies, such as star, shared medium, hypercube, etc. In some implementations, each data engine 122 is hosted on different computing resources (e.g., a separate computer server, a single core of a multi-core server, etc.). It should be understood that a data engine represents a role within the cluster, and in some embodiments, multiple roles can be hosted on a single computing resource, and a single role can be distributed across multiple computing resources.

[0051] exist Figure 2In the example, for simplicity, two data engines (first data engine 122a and second data engine 122b) are shown; however, it should be understood that the computing cluster 120 typically has more than two data engines. Each data engine can access a corresponding data repository 124, where the first data engine 122a can access the first data repository 124a, and the second data engine 122b can access the second data repository 124b. Each data repository 124a, 124b stores a portion of a set 113 of data elements 114. In some examples, the preprocessing module 106 distributes the set 113 across the data repositories such that each data repository stores approximately the same number of data elements. In some examples, the data elements are distributed in a specific order to improve processing efficiency (e.g., interleaved between data repositories or stored sequentially across data repositories). In other examples, the execution module 112 distributes the set 113 across the data repositories at runtime.

[0052] In operation, when the DoAll component 110 instructs the computing cluster 120 to process set 113, a "ForAll" process (not shown) is instantiated at each data engine 122. The ForAll process instantiated at a given data engine 122 (which is selected by the execution module 112, for example, based on availability) processes a portion of set 113 stored in its corresponding data repositories 124a, 124b (e.g., by applying a function f() to the data elements in that set). The results of the processing are returned to the DoAll component 110 via the communication network 130.

[0053] A checkpointing scheme is used to provide fault tolerance in both the compute cluster 120 and the data flow graph 111. In some examples, checkpoints include a predetermined number of data elements processed. In other examples, checkpoints are associated with a predetermined processing interval. In some examples, multiple counters are used to coordinate the checkpointing scheme, including a cluster work counter 132, a cluster checkpoint counter 134, and a graph checkpoint counter 136. The cluster work counter 132 leads the other counters and represents the time interval during which the data engine 122 is currently processing data elements in the set 113. The cluster checkpoint counter 134 lags behind the cluster work time 132 by at least one "tick" and represents the time elapsed since the cluster persistently stored its state. In the event of a failure in the cluster, the cluster is able to roll back its state to the state associated with the cluster checkpoint counter 134 and resume execution from that state. The graph checkpoint counter 136 also lags behind the cluster work counter and represents the time elapsed since the data flow graph 111 persistently stored its state. In the event of a failure in data flow graph 111, the graph can roll back its state to the state associated with graph checkpoint counter 136 and resume execution from that state. Further details of the checkpoint system can be found in U.S. Patent No. 11,288,284, the entire contents of which are incorporated herein by reference.

[0054] In this application, as described in more detail below, checkpoint counters are used to mark when result data is generated and to determine when the result data can be released and / or deleted from the managed buffer (e.g., by reading the values ​​of various counters and comparing them with the counter values ​​assigned to the result data). In the following description, for simplicity, the counter values ​​are not explicitly described as being read and compared with the counter values ​​of the result data, but the reading and comparison determine when the result data can be released and / or removed from the managed buffer.

[0055] 3. Hosting Solution

[0056] Data flow graph 111 and compute cluster 120 work together to implement a hosting scheme that ensures any results returned to data flow graph 111 by DoAll component 110 are stored until data flow graph 111 will never need to reproduce those results again (i.e., all results returned to a specific checkpoint of the data flow graph by DoAll component are committed, for example, committed to other entities and reported by DoAll component as no longer needing to be stored in the distributed compute cluster). As part of the hosting scheme, each data engine 122 includes a ForAll managed buffer 119 (sometimes referred to as a "ForAll buffer" or simply a "buffer") that temporarily stores the results of the ForAll process processing a portion of set 113 stored in the corresponding data repository 124 of that data engine. DoAll component 110 includes a DoAll managed buffer 117 (sometimes referred to as a "DoAll buffer" or simply a "buffer") that temporarily stores the results of processing elements of set 113.

[0057] If either the DoAll component 110 or the data engine 122 fails and recovery is required, the processing results stored in managed buffers 117 and 119 are reproduced. Once the processing results are persistently stored and will never need to be reproduced from the managed buffer again, the results are removed from the managed buffer. Counters 132, 134, and 136 are also used to coordinate the removal of such processing results from the managed buffer.

[0058] exist Figure 2 In the simplified example shown, a set 113 of data element 114 is partitioned across two data engines 122, with a first portion of set 113 stored in a first data repository 124a available to the first data engine 122a, and a second portion of set 113 stored in a second data repository 124b available to the second data engine 122b. The corresponding portions of the set stored in the data repositories are “over-segmented” because the various portions of the set are stored in the data repositories as multiple sub-segments. For example, the number of data elements in each sub-segment can be as large as the number of data elements that can be processed in parallel by the respective data engines. In other examples, each sub-segment may include as few as one data element. In the first data repository 124a, the first portion of the set is stored as two sub-segments P1 and P2. In the second data repository 124b, the second portion of the set is stored as two sub-segments P3 and P4.

[0059] The ForAll processes instantiated at each data engine 122 process the portion of that data engine in set 113 one subsegment at a time, and store the result in the ForAll managed buffer 116 of that data engine according to the subsegment. Once the results of the subsegments stored in the ForAll managed buffer 116 of the data engine are no longer needed, they are removed from the ForAll managed buffer. On average, each data engine 122 stores one subsegment result in its ForAll managed buffer at any given time, thereby reducing the total storage required to maintain the ForAll managed buffer 116. As described and illustrated below, this is at least in part due to the fact that result data that is no longer needed to be stored in the managed buffer is quickly cleared from the buffer.

[0060] As shown in more detail below, by over-segmenting the data set 113, the size of the managed buffer is maintained at a manageable size without any reduction in processing power or performance (otherwise the managed buffer might grow too large to be feasible to maintain).

[0061] 3.1 Example

[0062] Continue to refer to Figure 2 A simplified example illustrates a collection 113 that processes data element 114 while using a managed scheme. It should be understood that the following example is provided to facilitate understanding of the scheme, and a typical implementation of the scheme involves significantly more data and complexity than this example.

[0063] exist Figure 2 In the process, the DoAll component 110 has requested the compute cluster 120 to process the set 113 of data elements 114, and the data engines 122 have each instantiated the ForAll process. The cluster work counter 132 has a value K, and both the cluster checkpoint counter 134 and the graph checkpoint counter have a value K-1 (where K represents any time at the start of the example).

[0064] The first data engine 122a reads data elements 1 and 2 from the first sub-segment P1 of set 113. It applies some function f() to each data element to generate a processing result. In some examples, each processing result is labeled with the sub-segment number and the value of the cluster work counter 132. The result of processing data elements 1 and 2 is referred to as "f(1)". 1,K ")" and "f(2 1,K ", because both results are related to the value K of sub-segment P1 and cluster work counter 132. Result f(1 1,K ) and f(2 1,KThe data elements 1 and 2 are stored in the ForAll managed buffer 116a of the first data engine 122a associated with the first sub-segment P1. Data elements 1 and 2 are shaded in gray in the first data repository 124a to indicate that they have been read and processed by the first data engine 122a. Note that abbreviations are used to refer to the results in the diagram, where "1" is used in the abbreviation. 1,K "Corresponds to the result "f(1) 1,K ), and "2 1,K "Corresponds to the result "f(2) 1,K This notation is used throughout the remainder of this application.

[0065] Similarly, the second data engine 122b reads data element 9 from the third sub-segment P3 of set 113 and applies the function f() to that data element to generate a value called "f(9)". 3,K The result of processing f(9) is shown. 3,K The data element 9 is stored in the ForAll managed buffer 116b of the second data engine 122b associated with the third sub-segment P3. Data element 9 is shaded gray in the second data repository 124b to indicate that it has been read and processed by the second data engine 122b.

[0066] The result f(1) 1,K f(2) 1,K ) and f(9 3,K The results are sent from the corresponding data engines 122a and 122b of the computing cluster 120 to the DoAll component 110, where the results are stored in the DoAll managed buffer 117 in association with their respective subsegments.

[0067] refer to Figure 3 When computing cluster 120 finishes persistently storing the state of cluster checkpoint K, the value of cluster checkpoint counter 132 increments from K-1 to K, and cluster work counter 134 increments from K to K+1. A "Checkpoint K Complete" message is sent from computing cluster 120 to DoAll component 110. In one embodiment, this message is sent by the data engine that processed the last data element associated with the checkpoint; in this case, the message is also sent to other data engines in the cluster to notify them that the message has been sent and does not need to be sent again. In another embodiment, a data engine notifies a predetermined data engine of the data elements it has processed, which sends the aforementioned message to the processing component when a checkpoint has been reached. A checkpoint is reached when a predetermined time interval has elapsed or a predetermined number of data elements have been processed.

[0068] Once DoAll component 110 is notified that checkpoint K has been completed in computing cluster 120, it can safely release all results tagged with checkpoint K from DoAll managed buffer 117. In this case, DoAll component 110 releases result f(1 1,K f(2) 1,K ) and f(9 3,K This sends the result to the downstream component in data flow diagram 111. The released results are shaded in gray in DoAll managed buffer 117 to indicate that they have been released from the managed buffer.

[0069] refer to Figure 4 The first data engine 122a reads data elements 3 and 4 from the first sub-segment P1 of set 113 and applies the function f() to each data element to generate the processing result "f(3 1,K+1 ")" and "f(4)" 1,K+1 The result is f(3). 1,K+1 ) and f(4 1,K+1 (Where K+1 is read from the current value of the cluster work counter) is stored in the ForAll managed buffer 116a of the first data engine 122a associated with the first sub-segment P1. Data elements 3 and 4 are shaded in gray in the first data repository 124a to indicate that they have been read and processed by the first data engine 122a. Because all data elements of the first sub-segment P1 have been processed, the first data engine 122a issues a "Sub-segment 1 Complete" message.

[0070] The second data engine 122b reads data elements 10 and 11 from the third sub-segment P3 of set 113 and applies the function f() to the data elements to generate the processing result f(10). 3,K+1 ) and f(11 3,K+1 ). Result f(10 3,K+1 ) and f(11 3,K+1 The data elements 10 and 11 are stored in the ForAll managed buffer 116b of the second data engine 122b associated with the third sub-segment P3. Data elements 10 and 11 are shaded in gray in the second data repository 124b to indicate that they have been read and processed by the second data engine 122b.

[0071] When the data flow graph 111 finishes persistently storing the state of the graph at checkpoint K-1, the graph checkpoint counter 136 increments from K-1 to K.

[0072] Finally, the processing result f(3) 1,K+1 f(4) 1,K+1 f(10) 3,K+1 ) and f(11 3,K+1The results are sent from the computing cluster 120 to the DoAll component 110, where they are stored in the DoAll managed buffer 117 in association with their respective subsegments. A “Subsegment 1 Complete” message is also sent to the DoAll component 110, where it is stored for later use.

[0073] refer to Figure 5 When compute cluster 120 finishes persistently storing the state of cluster checkpoint K+1, the value of cluster checkpoint counter 132 increments from K to K+1, and cluster work counter 134 increments from K+1 to K+2. A "Checkpoint K+1 Complete" message is sent from compute cluster 120 to DoAll component 110. Once DoAll component 110 is notified that checkpoint K+1 has been completed in compute cluster 120, it can safely release all results tagged with checkpoint K+1 from DoAll managed buffer 117. In this case, DoAll component 110 releases result f(4). 1,K+1 f(10) 3,K+1 f(11) 3,K+1 ) and f(3 1,K+1 This sends the result to the downstream component in data flow diagram 111. The released results are shaded in gray in DoAll managed buffer 117 to indicate that they have been released from the managed buffer.

[0074] refer to Figure 6 When the data flow graph 111 has finished persistently storing the state of the graph at checkpoint K, the graph checkpoint counter 136 is incremented from K to K+1. The result of sub-segment 1 is removed from the DoAll managed buffer 117 because this result will never need to be reproduced from the DoAll managed buffer 117 (i.e., the state of the data flow graph is persistently stored until graph checkpoint K+1, and all results of sub-segment 1 have been computed and provided from the DoAll component 110 to downstream components). In some examples, the result to be removed is determined by the DoAll component 110 based on the segmentation information stored in the DoAll managed buffer 117 along with the result.

[0075] DoAll component 110 sends a "Subsegment 1 Complete" message to the first data engine 122a in the computing cluster 120. Upon receiving the Subsegment 1 Complete message, the first data engine 122a removes the result of Subsegment 1 from the first DoAll managed buffer 116a.

[0076] refer to Figure 7The first data engine 122a reads data elements 5 and 6 from the second sub-segment P2 of set 113 and applies the function f() to each data element to generate the processing result "f(5 2,K+2 ")" and "f(6)" 2,K+2 The result is f(5). 2,K+2 ) and f(6 2,K+2 The data elements 5 and 6 are sent to the DoAll component 110 and stored in the ForAll managed buffer 116a of the first data engine 122a associated with the second sub-segment P2. Data elements 5 and 6 are shaded in gray in the first data repository 124a to indicate that they have been read and processed by the first data engine 122a.

[0077] The second data engine 122b reads data elements 12 and 13 from the third sub-segment P3 and the fourth sub-segment P4 respectively, and applies the function f() to the data element to generate the processing result f(12). 3,K+2 ) and f(13 4,K+2 ). Result f(12) 3,K+2 ) and f(13 4,K+2 The data elements 12 and 13 are stored respectively in the ForAll managed buffer 116b of the second data engine 122b associated with the third sub-segment P3 and the fourth sub-segment P4. Data elements 12 and 13 are shaded in gray in the second data repository 124b to indicate that they have been read and processed by the second data engine 122b. Because all data elements of the third sub-segment P3 have been processed, the second data engine 122b issues a "Sub-segment 3 complete" message.

[0078] Processing result f(5) 2,K+2 f(6) 2,K+2 f(12) 3,K+2 ) and f(13 4,K+2 The results are sent from the computing cluster 120 to the DoAll component 110, where they are stored in the DoAll managed buffer 117 in association with their respective subsegments. A “Subsegment 3 Complete” message is also sent to the DoAll component 110, where it is stored for later use.

[0079] refer to Figure 8When compute cluster 120 finishes persistently storing the state of cluster checkpoint K+2, the value of cluster checkpoint counter 132 increments from K+1 to K+2, and cluster work counter 134 increments from K+2 to K+3. A "Checkpoint K+2 Complete" message is sent from compute cluster 120 to DoAll component 110. Once DoAll component 110 is notified that checkpoint K+2 has been completed in compute cluster 120, it can safely release all results tagged with checkpoint K+2 from DoAll managed buffer 117. In this case, DoAll component 110 releases result f(5). 2,K+2 f(6) 2,K+2 f(12) 3,K+2 ) and f(13 4,K+2 This sends the result to the downstream component in data flow diagram 111. The released results are shaded in gray in DoAll managed buffer 117 to indicate that they have been released from the managed buffer.

[0080] refer to Figure 9 When the data flow graph 111 has finished persistently storing the state of graph checkpoint K+1, the graph checkpoint counter 136 increments from K+1 to K+2. The result of the third sub-segment P3 is removed from the DoAll managed buffer 117 because the result will never need to be reproduced from the DoAll managed buffer 117 (i.e., the state of the data flow graph is persistently stored until graph checkpoint K+2, and all results of sub-segment 1 have been computed and provided from the DoAll component 110 to the downstream component).

[0081] DoAll component 110 sends a "Subsegment 3 Complete" message to the second data engine 122b in the compute cluster 120. Upon receiving the Subsegment 3 Complete message, the second data engine 122b removes the result of Subsegment 3 from the second ForAll managed buffer 116b.

[0082] 4. DoAll Faults and Recovery

[0083] refer to Figure 10 The first data engine 122a reads data elements 7 and 8 from the second sub-segment P2 of set 113. It applies the function f() to each data element to generate the processing result "f(7)". 2,K+3 )” and “f(8)” 2,K+3 The result is f(7). 2,K+3 ) and f(8 2,K+3The data elements 7 and 8 are stored in the ForAll managed buffer 116a of the first data engine 122a associated with the second sub-segment P2. Data elements 7 and 8 are shaded in gray in the first data repository 124a to indicate that they have been read and processed by the first data engine 122a. Because all data elements of the second sub-segment P2 have been processed, the first data engine 122a issues a "Sub-segment 2 Complete" message.

[0084] The second data engine 122b reads data elements 14 and 15 from the fourth sub-segment P4 and applies the function f() to the data elements to generate the processing result f(14). 4,K+3 ) and f(15 4,K+3 ). Result f(14) 4,K+3 ) and f(15 4,K+3 The data elements 14 and 15 are stored in the ForAll managed buffer 116b of the second data engine 122b associated with the fourth sub-segment P4. Data elements 14 and 15 are shaded in gray in the second data repository 124b to indicate that they have been read and processed by the second data engine 122b.

[0085] Processing result f(7) 2,K+3 f(8) 2,K+3 f(14) 4,K+3 ) and f(15 4,K+3 The sub-segment 2 completion message is sent from the computing cluster 120 to the DoAll component 110, but the data flow graph 111 fails before the processing result and the sub-segment 2 completion message reach the DoAll component 110.

[0086] refer to Figure 11 The data flow graph 111 restarts and restores its state to graph checkpoint K+2. As part of the restoration, the DoAll component 110 causes the data engines 122 in the compute cluster 120 to reproduce the results stored in their respective ForAll managed buffers 116. In some examples, the DoAll component 110 does this by sending a message to the data engines 122 requesting that the data engines reproduce their respective result data. Figure 11 In the example, all results stored in the first ForAll managed buffer 116a (i.e., f(5)) 2,K+2 f(6) 2,K+2 f(7) 2,K+3 ) and f(8 2,K+3 )) and all results stored in the second ForAll managed buffer 116b (i.e., f(13) 4,K+2 f(14) 4,K+3 ) and f(15 4,K+3The message was resent to DoAll component 110. The sub-segment 2 completion message was also resent because all data elements of the second sub-segment P2 had been processed.

[0087] DoAll component 110 receives and processes the result f(5) 2,K+2 f(6) 2,K+2 f(7) 2,K+3 f(8) 2,K+3 f(13) 4,K+2 f(14) 4,K+3 ) and f(15 4,K+3 These results are then stored in the DoAll managed buffer 117 in association with their respective subsegments. The subsegment 2 completion message is also received by the DoAll component 110, where it is stored for later use.

[0088] refer to Figure 12 When compute cluster 120 finishes persistently storing the state of cluster checkpoint K+3, the value of cluster checkpoint counter 132 increments from K+2 to K+3, and cluster work counter 134 increments from K+3 to K+4. A "Checkpoint K+3 Complete" message is sent from compute cluster 120 to DoAll component 110. Once DoAll component 110 is notified that checkpoint K+3 has been completed in compute cluster 120, it can safely release all results tagged with checkpoint K+3 (or earlier) from DoAll managed buffer 117. In this case, DoAll component 110 releases result f(7). 2,K+3 f(8) 2,K+3 f(14) 4,K+3 ) and f(15 4,K+3 This sends the result to the downstream components in data flow diagram 111. The released results are shaded in gray in DoAll managed buffer 117 to indicate that they have been released from the managed buffer. Note that the DoAll component is aware of the result f(5) from its restored state. 2,K+2 f(6) 2,K+2 ) and f(13 4,K+2 It does not need to be reproduced to the downstream components in data flow diagram 111.

[0089] refer to Figure 13When the data flow graph 111 has finished persistently storing the state of graph checkpoint K+2, the graph checkpoint counter 136 increments from K+2 to K+3. The result of sub-segment 2 is removed from the DoAll managed buffer 117 because the result will never need to be reproduced from the DoAll managed buffer 117 (i.e., the state of the data flow graph is persistently stored until graph checkpoint K+3, and all results of sub-segment 2 have been computed and provided from the DoAll component 110 to the downstream component).

[0090] DoAll component 110 sends a "Subsegment 2 Complete" message to the first data engine 122a in the computing cluster 120. Upon receiving the Subsegment 2 Complete message, the first data engine 122a removes the result of Subsegment 2 from the first ForAll managed buffer 116a.

[0091] 5. Data Engine Failure and Recovery

[0092] refer to Figure 14 The second data engine 122b reads data element 16 from the fourth sub-segment P4 of set 113. It applies the function f() to the data element to generate the processing result "f(16)". 4,K+4 The result is f(16). 4,K+4 The data element 16 is stored in the ForAll managed buffer 116b of the second data engine 122b associated with the fourth sub-segment P4. Data element 16 is shaded gray in the second data repository 124b to indicate that it has been read and processed by the second data engine 122b. Because all data elements of the fourth sub-segment P4 have been processed, the second data engine 122b issues a "Sub-segment 4 Complete" message.

[0093] Processing result f(16) 4,K+4 The result is sent from computing cluster 120 to DoAll component 110, where it is stored in DoAll managed buffer 117 in association with sub-segment 4. Sub-segment 4 completion message is also sent to DoAll component 110, where it is stored for later use.

[0094] Then, the second data engine 122b fails. In some examples, the failure of the second data engine 122b is detected by the DoAll component 110 and / or the first data engine 122a based on periodic messages sent to the second data engine 122b requesting a response and based on a predetermined threshold time elapsed without such a response being received. Typically, the individual data engines are replicated at one or more different computing devices (not shown) in the computing cluster 120 to ensure that the computing cluster can resume processing in the event of a data engine failure. In some examples, the replicas are created and maintained by the execution module 112.

[0095] refer to Figure 15 The replica 122b' of the second data engine takes over the position of the failed second data engine 122b. The data flow graph 111 rolls back to its state at graph checkpoint K+3. Note that the result of previously processed data element 16 is removed from both the second ForAll managed buffer 116b and the DoAll managed buffer 117. The cluster work counter 132 increments from K+4 to K+5, the cluster checkpoint counter increments from K+3 to K+4, and the graph checkpoint counter increments from K+3 to K+4.

[0096] refer to Figure 16 The second data engine copy 122b' reads data element 16 from the fourth sub-segment P4 of set 113 and applies function f() to that data element to generate the processing result "f(16)". 4,K+5 In some examples, the replica reads the current value of the cluster work counter 132, decrements that value, and reads all data elements associated with the decremented value and assigned to the second data processor. These data elements can be requested from the entity responsible for partitioning the data elements and assigning them to data processors.

[0097] The result is f(16) 4,K+5 The data element 16 is stored in the ForAll managed buffer 116b of the replica 122b' of the second data engine associated with the fourth subsegment P4. Data element 16 is shaded gray in the second data repository 124b to indicate that it has been read and processed by the replica 122b' of the second data engine. Because all data elements of the fourth subsegment P4 have been processed, the replica 122b' of the second data engine issues a "Subsegment 4 Complete" message.

[0098] DoAll component 110 receives and processes the result f(5) 2,K+2 f(6) 2,K+2 f(7) 2,K+3 f(8) 2,K+3 f(13) 4,K+2 f(14)4,K+3 ) and f(15 4,K+3 These results are then stored in the DoAll managed buffer 117 in association with their respective subsegments. The subsegment 2 completion message is also received by the DoAll component 110, where it is stored for later use.

[0099] refer to Figure 17 When compute cluster 120 finishes persistently storing the state of cluster checkpoint K+5, the value of cluster checkpoint counter 132 increments from K+4 to K+5, and cluster work counter 134 increments from K+5 to K+6. A "Checkpoint K+5 Complete" message is sent from compute cluster 120 to DoAll component 110. Once DoAll component 110 is notified that checkpoint K+5 has been completed in compute cluster 120, it can safely release all results tagged with checkpoint K+5 from DoAll managed buffer 117. In this case, DoAll component 110 releases result f(16). 4,K+5 This sends the result to the downstream component in data flow diagram 111. The released results are shaded in gray in DoAll managed buffer 117 to indicate that they have been released from the managed buffer.

[0100] refer to Figure 18 When the data flow graph 111 has finished persistently storing the state of graph checkpoint K+5, the graph checkpoint counter 136 increments from K+4 to K+5. The result of sub-segment 4 is removed from the DoAll managed buffer 117 because the result will never need to be reproduced from the DoAll managed buffer 117 (i.e., the state of the data flow graph is persistently stored until graph checkpoint K+5, and all results of sub-segment 4 have been computed and provided from the DoAll component 110 to the downstream component).

[0101] DoAll component 110 sends the sub-segment 4 complete message to the second data engine 122b in the computing cluster 120. Upon receiving the sub-segment 4 complete message, the second data engine 122b removes the result of sub-segment 4 from the second ForAll managed buffer 116b.

[0102] Referring to Figure 25, set 113 was completely processed.

[0103] 6. Achieve

[0104] The methods described above can be implemented, for example, using a programmable computing system that executes suitable software instructions, or it can be implemented in suitable hardware (such as a field-programmable gate array (FPGA)) or in some hybrid form. For example, in a programming approach, the software may include processes in one or more computer programs that execute on one or more programmable computing systems (which may have various architectures, such as distributed, client / server, or network-based), each of which includes at least one processor, at least one data storage system (including volatile and / or non-volatile memory and / or storage elements), and at least one user interface (for receiving input using at least one input device or port and for providing output using at least one output device or port). The software may include one or more modules of a larger program, for example, providing services related to the design, configuration, and execution of the data flow graph. Modules of the program (e.g., elements of the data flow graph) may be implemented as data structures stored in a data store or other organized data conforming to a data model.

[0105] The software may be provided on a tangible, non-transitory medium such as a CD-ROM or other computer-readable medium (e.g., readable by a general-purpose or special-purpose computing system or device), or delivered via a network communication medium (e.g., encoded in a propagating signal) to a tangible, non-transitory medium of a computing system executing the software. Part or all of the processing may be performed on a dedicated computer or using dedicated hardware such as a coprocessor, a field-programmable gate array (FPGA), or a dedicated application-specific integrated circuit (ASIC). The processing may be implemented in a distributed manner, in which different parts of the computation specified by the software are performed by different computing elements. Such computer programs are preferably stored or downloaded to a computer-readable storage medium (e.g., solid-state memory or medium, or magnetic or optical medium) accessible by a general-purpose or special-purpose programmable computer for configuring and operating the computer when the storage medium is read by the computer to perform the processing described herein. The system of the present invention may also be considered as a tangible, non-transitory medium configured with a computer program, wherein such a medium causes the computer to operate in a specific and predefined manner to perform one or more of the processing steps described herein.

[0106] Several embodiments of the invention have been described. However, it should be understood that the foregoing description is intended to be illustrative and not to limit the scope of the invention, which is defined by the appended claims. Therefore, other embodiments are also within the scope of the appended claims. For example, various modifications can be made without departing from the scope of the invention. Additionally, some of the steps described above may be sequentially independent and thus may be performed in a different order than that described.

Claims

1. A method for fault-tolerant processing of multiple data elements using a distributed computing cluster, the distributed computing cluster comprising multiple data processors associated with corresponding multiple data repositories, the method comprising: The plurality of data elements are stored in the distributed computing cluster, wherein the plurality of data elements are distributed across the plurality of data repositories according to the plurality of partitions of the data elements; The first data processor among the plurality of data processors processes the data elements of a first group of segments stored at a first data repository in the plurality of data repositories to generate first result data of the first group of segmented data elements; Sending the first result data from the distributed computing cluster to a processing component outside the distributed computing cluster; and The first result data is stored in a first buffer located in the distributed computing cluster and associated with the first data processor until the processing component has persistently stored the first result data outside the distributed computing cluster.

2. The method according to claim 1, further comprising: After the processing component has persistently stored all the result data associated with the first segment outside the distributed computing cluster, the first result data is removed from the first buffer.

3. The method according to claim 1, wherein, At least a portion of the plurality of data repositories includes two or more split data elements from the plurality of data elements.

4. The method according to claim 1, wherein, The processing component includes a data flow graph containing consumer components.

5. The method according to claim 4, wherein, The consumer component of the data flow graph includes a second buffer for storing result data, and the method further includes storing the first result data in the second buffer.

6. The method according to claim 5, wherein, Based on an indication that the computing cluster has persistently stored an indication of the state associated with the first result data, the first result data is released from the second buffer.

7. The method according to claim 5, further comprising: After the processing component has released all the result data of the first segment from the second buffer and has persistently stored the state information for the data flow graph, the first result data is removed from the second buffer.

8. The method according to claim 1, further comprising: Based on the determination that the processing component encountered a failure before persistently storing the first result data outside the distributed computing cluster, the first result data is resent from the distributed computing cluster to the processing component.

9. The method according to claim 8, wherein, Retransmitting the first result data includes reading the first result data from the first buffer associated with the first data processor.

10. The method according to claim 1, further comprising: It was determined that the first data processor had encountered a fault, and based on this determination: The first data processor is activated as a copy of itself based on the determination that the first data processor has encountered a fault. Restore the processing component to the state it was in before receiving the first result data from the distributed computing cluster.

11. The method of claim 10, further comprising: The copy of the first data processor is used to process the first set of segmented data elements to generate regenerated result data of the first set of segmented data elements; The regenerated result data is sent from the distributed computing cluster to the processing component; as well as The regenerated result data is stored in the first buffer located in the distributed computing cluster and associated with the copy of the first data processor until the processing component has persistently stored the regenerated result data outside the distributed computing cluster.

12. The method according to claim 1, wherein, Processing the first set of segmented data elements involves applying the same function to each data element.

13. The method according to claim 1, wherein, The process also includes: Each processing result in the first result data is labeled with a segmentation number and the value of a counter associated with the cluster.

14. The method according to claim 1, further comprising: In response to a predefined number of data elements having been processed in the distributed computing cluster, a counter associated with the cluster is incremented, and a message is sent to the processing component indicating that a checkpoint indicated by the counter has been reached.

15. The method of claim 14, further comprising: The checkpoint is determined to have been reached based on the number of data elements that have been processed by the data processor since the last increment of the counter. or The checkpoint is determined to have been reached by determining whether a predetermined time interval has elapsed since the last increment of the counter.

16. The method of claim 14, further comprising: At the first data processor, a message is received from the processing component indicating that all data elements associated with the current value of the counter have been removed from the processing component; as well as In response to receiving the message, the first result data is removed from the first buffer.

17. The method according to claim 1, further comprising: At the first data processor, a message is received from the processing component, the message requesting the first data processor to resend the first result data to the processing component; as well as The first data processor sends the first result data to the processing component.

18. The method according to claim 1, further comprising: The first data processor determines that the second data processor has experienced an operational failure, specifically, wherein the operational failure is detected based on a fault indication message being sent from the second data engine or the second data engine failing to respond to a message periodically sent by the first data processor; and In response to determining the fault, the second data processor is copied.

19. The method according to claim 18, wherein, The second data processor includes: Another data processor among the plurality of data processors is identified through the first data processor, particularly by identifying a data processor that responds to messages within a threshold time and / or reports available capacity upon request; and A message is sent to the identified data processor requesting the identified data processor to update its data elements based on a state reflected by a previous value of a first counter, the data elements being associated with a segment previously assigned to the second data processor.

20. A system for fault-tolerant processing of multiple data elements using a distributed computing cluster, the distributed computing cluster comprising multiple data processors associated with corresponding multiple data repositories, the system comprising: Multiple data repositories for storing the multiple data elements, wherein the multiple data elements are distributed across the multiple data repositories according to multiple partitions of the data elements; Multiple data processors are used to process data elements, the multiple data processors including a first processor, the first processor being used to process a first set of segments among the multiple segments stored at a first data repository in the multiple data repositories, to generate first result data of data elements of the first set of segments; An output unit, used to send the first result data from the distributed computing cluster to a processing component of the first result data outside the distributed computing cluster; and A first buffer, located in the distributed computing cluster and associated with the first data processor, is used to store the first result data until the processing component has persistently stored the first result data outside the distributed computing cluster.

21. A computer-readable medium storing software in a non-transitory form, the software including instructions for causing a computing system to process multiple data elements in a fault-tolerant manner using a distributed computing cluster, the distributed computing cluster including multiple data processors associated with corresponding multiple data repositories, the instructions causing the computing system to: The plurality of data elements are stored in the distributed computing cluster, wherein, The plurality of data elements are distributed across the plurality of data repositories based on the plurality of data element partitions; The first data processor among the plurality of data processors processes the data elements of a first group of segments stored at a first data repository in the plurality of data repositories to generate first result data of the first group of segmented data elements; The first result data is sent from the distributed computing cluster to a processing component for the first result data outside the distributed computing cluster; as well as The first result data is stored in a first buffer located in the distributed computing cluster and associated with the first data processor until the processing component has persistently stored the first result data outside the distributed computing cluster.

22. A system for fault-tolerant processing of multiple data elements using a distributed computing cluster, the distributed computing cluster comprising multiple data processors associated with corresponding multiple data repositories, the system comprising: Components for storing the plurality of data elements, wherein the plurality of data elements are distributed across the plurality of data repositories according to a plurality of partitions of the data elements; The components for processing data elements include a first processor for processing a first set of segments in a first data repository stored in the plurality of data repositories to generate first result data of data elements of the first set of segments; A component for sending the first result data from the distributed computing cluster to a processing component for the first result data outside the distributed computing cluster; and A storage component, located in the distributed computing cluster and associated with the first data processor, is used to store the first result data until the processing component has persistently stored the first result data outside the distributed computing cluster.

Images