Distributed data stream asynchronous snapshot method and system based on flink

Abstract

The invention relates to the technical field of distributed data stream processing, in particular to a distributed data stream asynchronous snapshot method and system based on flink, and the method comprises the following steps: designing and executing a snapshot algorithm which is completed in limited time under the condition that all processes are in an active state, thereby meeting the termination requirement; when the snapshots are executed, the causal sequence of the data streams is maintained, and it is ensured that the delivery sequence recorded in the snapshots is consistent with the sequence in actual task execution, so that the feasibility requirement is met; the flink-based distributed data stream asynchronous snapshot method and system provided by the invention have the beneficial effects that an asynchronous obstacle snapshot algorithm is introduced, and the flink-based distributed data stream asynchronous snapshot method and system are a new snapshot technology for realizing good throughput. The asynchronous obstacle snapshot algorithm is the first algorithm which considers the minimum state required by the loop-free execution topology. In addition, an asynchronous barrier snapshot algorithm is extended to accommodate a loop execution graph by storing only records that need to be reprocessed when recovery is required.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data stream processing, and in particular to a method and system for distributed data stream asynchronous snapshot based on Flink. Background Art

[0002] In order to ensure that distributed processing systems can recover reliably when task failures occur, a common approach is to use global snapshot technology. A global snapshot is the complete state of the execution graph at a specific point in time, which captures all relevant data and state information.

[0003] In distributed systems, the main purpose of global snapshots is to provide a consistent recovery point so that computing can be quickly resumed in the event of a failure. By capturing global snapshots regularly, the system can quickly recover to a consistent state after a failure, ensuring the reliability and consistency of computing. The capture of a global snapshot usually involves stopping the operation of the entire computing system and capturing all tasks and data at once. Once the snapshot is completed, the system will continue its operation. A major disadvantage of this approach is that it requires stopping the operation of the entire system, which can result in long downtime, especially in large-scale systems.

[0004] To solve this problem, some systems use asynchronous global snapshot technology. These systems usually use distributed logs to record the metadata and data status of tasks. When the system needs to recover to a consistent global state, it traverses the log and applies the corresponding operations to rebuild the snapshot. The benefit of this approach is that it does not require stopping the operation of the entire system, but can take a snapshot without interrupting normal processing. However, the algorithm for implementing asynchronous global snapshots is usually complex and needs to be carefully designed to ensure the correctness and reliability of the system.

[0005] In addition, to improve the efficiency of global snapshots, some systems use incremental snapshot technology. Incremental snapshots only capture data and state information that have changed since the last snapshot, rather than the state of the entire execution graph. This approach can reduce the size and capture time of snapshots, thereby improving the performance and responsiveness of the system. In general, global snapshots are an important technology in distributed processing systems that help ensure the reliability and consistency of computing. However, implementing efficient global snapshots requires careful design of algorithms and system architecture, and consideration of various factors to ensure the correctness and reliability of the system. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for asynchronous snapshot of distributed data streams based on Flink to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: a method for asynchronous snapshot of distributed data stream based on Flink, the method comprising the following steps:

[0008] Design and implement a snapshot algorithm that completes in bounded time while all processes are active, thus satisfying termination requirements;

[0009] When executing snapshots, the causal order of data flows is maintained to ensure that the delivery order recorded in the snapshot is consistent with the order in the actual task execution to meet feasibility requirements.

[0010] Preferably, in an acyclic data flow environment, the method further includes the following steps:

[0011] Special barrier markers are injected periodically into the input data stream to simulate the execution phase;

[0012] When each task receives a barrier representing the execution phase, its current set of operating states is included in the snapshot, implementing a snapshot mechanism without persistent channel states.

[0013] Preferably, in a data flow environment including a directed cycle, the method further comprises the following steps:

[0014] Identify backward edges in loops through static analysis;

[0015] During the snapshot process, for each task, after receiving a barrier from the regular input, a downstream backup log is created for all records received from the backward edge;

[0016] The barrier is pushed to the entire execution graph, including backward edges, to ensure that records within the loop are consistently included in the snapshot and can be put back into the transfer when resuming.

[0017] Preferably, a fault recovery mechanism is also included, the mechanism comprising:

[0018] In case of failure, the entire execution graph is restarted starting from the last global snapshot;

[0019] Each task retrieves its snapshot state from persistent storage and restores the records in its backup log;

[0020] The task continues to ingest records from its input channels to resume processing of the data stream.

[0021] Preferably, the fault recovery mechanism also includes a partial graph recovery solution, and the specific steps include:

[0022] Reschedule upstream tasks and their dependencies, especially those tasks whose output channels point to the failed task;

[0023] Sequence records to add a unique identifier to each record sent from the source;

[0024] During the recovery process, each downstream node ignores records with sequence numbers less than the processed sequence numbers to avoid duplicate processing, thereby achieving fault recovery with exactly-once semantics.

[0025] A distributed data stream asynchronous snapshot system based on Flink includes multiple processing nodes, each of which performs distributed data stream processing tasks. The system asynchronously generates globally consistent snapshots without interrupting data stream processing to support system fault recovery and fault tolerance.

[0026] Preferably, the system includes a snapshot algorithm that simulates execution stages by injecting special barrier markers into the data stream and captures the set of operating states of the processing nodes at the end of each stage to construct a global snapshot, thereby ensuring the termination of the snapshot.

[0027] Preferably, the system handles both acyclic data flows and data flows containing cycles. For the latter, the system identifies backward edges in the cycles through static analysis and performs additional processing on records received from backward edges during the snapshot to ensure that all relevant records are included in the snapshot, meeting the feasibility requirements of the snapshot.

[0028] Preferably, the system includes a snapshot management mechanism for generating, storing, retrieving and versioning snapshots, wherein snapshot data is persistently stored in a reliable storage medium so that the system state can be quickly restored when needed.

[0029] Preferably, the system also includes a fault recovery mechanism, which supports both full-graph recovery and partial-graph recovery. When a processing node fails, the latest global snapshot data is used to restore the system's operating status, while ensuring the data's exact one-time processing semantics to avoid repeated calculations.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The method and system for asynchronous snapshot of distributed data streams based on Flink proposed in this invention introduce the asynchronous barrier snapshot algorithm, which is a new snapshot technology that achieves good throughput. The asynchronous barrier snapshot algorithm is the first algorithm that considers the minimum state required for a loop-free execution topology. In addition, the asynchronous barrier snapshot algorithm is extended to adapt to cyclic execution graphs by only storing records that need to be reprocessed when recovery is required. The asynchronous barrier snapshot algorithm is implemented on Apache Flink and the effect of the method compared with synchronous snapshots is evaluated. In the early stage, the asynchronous barrier snapshot algorithm shows good results, has little impact on the overall execution throughput, and has linear scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] For example, see Figure 1 The present invention provides a technical solution: a method for asynchronous snapshot of distributed data stream based on Flink, the method comprising the following steps:

[0035] Design and implement a snapshot algorithm that completes in bounded time while all processes are active, thus satisfying termination requirements;

[0036] When executing snapshots, the causal order of data flows is maintained to ensure that the delivery order recorded in the snapshot is consistent with the order in the actual task execution to meet feasibility requirements.

[0037] In an acyclic data flow environment, the following steps are also included:

[0038] Special barrier markers are injected periodically into the input data stream to simulate the execution phase;

[0039] When each task receives a barrier representing the execution phase, its current set of operating states is included in the snapshot, implementing a snapshot mechanism without persistent channel states.

[0040] In a data flow environment with directed loops, the following steps are also included:

[0041] Identify backward edges in loops through static analysis;

[0042] During the snapshot process, for each task, after receiving a barrier from the regular input, a downstream backup log is created for all records received from the backward edge;

[0043] The barrier is pushed to the entire execution graph, including backward edges, to ensure that records within the loop are consistently included in the snapshot and can be put back into the transfer when resuming.

[0044] A failure recovery mechanism is also included, which includes:

[0045] In case of failure, the entire execution graph is restarted starting from the last global snapshot;

[0046] Each task retrieves its snapshot state from persistent storage and restores the records in its backup log;

[0047] The task continues to ingest records from its input channels to resume processing of the data stream.

[0048] The fault recovery mechanism also includes a partial graph recovery solution, and the specific steps include:

[0049] Reschedule upstream tasks and their dependencies, especially those tasks whose output channels point to the failed task;

[0050] Sequence records to add a unique identifier to each record sent from the source;

[0051] During the recovery process, each downstream node ignores records with sequence numbers less than the processed sequence numbers to avoid duplicate processing, thereby achieving fault recovery with exactly-once semantics.

[0052] Embodiment 2, based on embodiment 1, proposes the following technical solution:

[0053] (1) Problem definition

[0054] To ensure correct results after recovery, such as termination and feasibility as described in Tel

[14] , we need to ensure that each snapshot satisfies certain properties.

[0055] Termination guarantees that the snapshot algorithm will eventually complete in a bounded time after it is started, as long as all processes are active.

[0056] Feasibility refers to the meaningfulness of a snapshot, i.e., no information about the computation is lost during the snapshot. Formally, this means that causal ordering [9] is maintained across snapshots, and records are sent as they are delivered in tasks from the snapshot’s perspective.

[0057] In short, to ensure that a distributed processing system can correctly handle task failures and provide consistent results, it is necessary to capture snapshots of the execution graph periodically. The snapshots should satisfy properties such as termination and feasibility to ensure that the system can recover from failures and continue to perform computations.

[0058] (2) ABS of Acyclic Data Flow

[0059] Snapshotting without persisting channel state is feasible when the execution is divided into multiple stages. Stages divide the injected data flow and all related computations into a series of possible executions where all previous inputs and generated outputs have been fully processed. The set of operational states at the end of a stage reflects the entire execution history and can therefore be used only for snapshots. The core idea behind our algorithm is to use staged snapshots to create identical snapshots while maintaining continuous data injection.

[0060] In our approach, we simulate the stages of execution of a continuous dataflow by periodically injecting special barrier markers into the input dataflow. These barriers are pushed throughout the execution graph to the sinks. As each task receives a barrier representing a stage of execution, a global snapshot is gradually built.

[0061] (3) ABS of Circular Data Stream

[0062] When there are directed cycles in the execution graph, the ABS algorithm introduced earlier will cause a deadlock and fail to terminate, because the tasks in the cycle will wait indefinitely to receive barriers from all inputs. In addition, records in any transmission within the cycle will not be included in the snapshot, thus violating feasibility. Therefore, it is necessary to consistently include records generated within the loop in the snapshot to achieve feasibility, and put these records back into the transmission when resuming.

[0063] Our approach handles cyclic graphs, extending the basic algorithm without incurring any additional channel blocking. First, we identify backward edges L in the cycle by static analysis. In control flow theory of directed graphs, a backward edge is an edge pointing to a vertex that has already been visited during the depth-first search. The execution graph G(T,E\L) is a directed acyclic graph (DAG) containing all tasks in the topology. From the perspective of the DAG, the algorithm operates the same as before, but we additionally make a downstream backup of the records received from the identified backward edges during the snapshot. This is achieved in the following way: each task t, as a consumer of the backward edge Lt A backup log is created for all records received from Lt from the time the barrier is forwarded until the barrier is received from Lt. The barrier pushes all in-flight records within the loop into the downstream log so that they are included in a consistent snapshot only once.

[0064] The algorithm is guaranteed to terminate because each task eventually receives barriers from all inputs (including backward side channels) and completes its snapshot. By broadcasting immediately after receiving barriers from regular inputs, we can avoid the deadlock condition mentioned earlier.

[0065] The FIFO ordering property still holds for backward edges, which is demonstrated by the following properties. 1) Each task state included in the snapshot is a copy of the state of the corresponding task before processing any subsequent events after the barrier received on the regular input. 2) The downstream log included in the snapshot is complete and contains all subsequent records pending before the barrier received on the backward edge, due to the FIFO guarantee.

[0066] (4) Fault recovery

[0067] Although not the focus of this article, a working failure recovery scheme can motivate us to apply the snapshot approach. Therefore, we briefly explain its operation here. There are several failure recovery schemes that work with consistent snapshots. In the simplest case, the entire execution graph can be restarted starting from the last global snapshot: each task t does the following: 1) retrieves the state associated with snapshot st from persistent storage and sets it as the initial state; 2) restores its backup log and processes all records contained in it; 3) starts ingesting records from its input channels.

[0068] Partial graph recovery schemes are also possible, similar to TimeStream, by simply rescheduling upstream task dependencies (those tasks whose output channels point to the failed task) and their respective upstream tasks up to the source. To provide exactly-once semantics, duplicate records should be ignored in all downstream nodes to avoid recomputation. To achieve this, we can adopt a scheme similar to SDGs to tag records with sequence numbers from the source, so that each downstream node can discard records with sequence numbers less than the processed sequence number.

[0069] Example 3, based on Example 2, proposes a distributed data stream asynchronous snapshot system based on Flink, including multiple processing nodes, each node performs distributed data stream processing tasks, and the system asynchronously generates globally consistent snapshots without interrupting data stream processing to support system fault recovery and fault tolerance.

[0070] The system includes a snapshot algorithm that simulates execution stages by injecting special barrier markers into the data stream and captures the set of operating states of the processing nodes at the end of each stage to build a global snapshot, thereby ensuring the termination of the snapshot.

[0071] The system handles both acyclic data flows and data flows containing cycles. For the latter, the system identifies backward edges in the cycles through static analysis and performs additional processing on records received from backward edges during the snapshot to ensure that all relevant records are included in the snapshot and meet the feasibility requirements of the snapshot.

[0072] The system includes a snapshot management mechanism for generating, storing, retrieving and versioning snapshots, where snapshot data is persistently stored in a reliable storage medium so that the system state can be quickly restored when needed.

[0073] The system also includes a fault recovery mechanism that supports both full-graph recovery and partial-graph recovery. When a processing node fails, the latest global snapshot data is used to restore the system's operating status, while ensuring the data's exactly-once processing semantics to avoid repeated calculations.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A distributed data stream asynchronous snapshot method based on Flink, characterized by: The method comprises the following steps: Design and implement a snapshot algorithm that completes in bounded time while all processes are active, thus satisfying termination requirements; When executing snapshots, the causal order of data flows is maintained to ensure that the delivery order recorded in the snapshot is consistent with the order in the actual task execution to meet feasibility requirements.

2. According to claim 1, a distributed data stream asynchronous snapshot method based on Flink, in an acyclic data stream environment, is characterized in that: The following steps are also included: Special barrier markers are injected periodically into the input data stream to simulate the execution phase; When each task receives a barrier representing the execution phase, its current set of operating states is included in the snapshot, implementing a snapshot mechanism without persistent channel states.

3. According to claim 1, a distributed data stream asynchronous snapshot method based on Flink, in a data stream environment containing directed cycles, is characterized in that: The following steps are also included: Identify backward edges in loops through static analysis; During the snapshot process, for each task, after receiving a barrier from the regular input, a downstream backup log is created for all records received from the backward edge; The barrier is pushed to the entire execution graph, including backward edges, to ensure that records within the loop are consistently included in the snapshot and can be put back into the transfer when resuming.

4. According to a method for asynchronous snapshot of distributed data stream based on Flink according to claim 1, it is characterized in that: A failure recovery mechanism is also included, which includes: In case of failure, the entire execution graph is restarted starting from the last global snapshot; Each task retrieves its snapshot state from persistent storage and restores the records in its backup log; The task continues to ingest records from its input channels to resume processing of the data stream.

5. According to a method for asynchronous snapshot of distributed data stream based on Flink according to claim 4, it is characterized in that: The fault recovery mechanism also includes a partial graph recovery solution, and the specific steps include: Reschedule upstream tasks and their dependencies, especially those tasks whose output channels point to the failed task; Sequence records to add a unique identifier to each record sent from the source; During the recovery process, each downstream node ignores records with sequence numbers less than the processed sequence numbers to avoid duplicate processing, thereby achieving fault recovery with exactly-once semantics.

6. A distributed data stream asynchronous snapshot system based on Flink according to the method for distributed data stream asynchronous snapshot based on Flink according to any one of claims 1 to 5, comprising a plurality of processing nodes, each node executing a distributed data stream processing task, characterized in that: The system asynchronously generates globally consistent snapshots without interrupting data flow processing to support system failure recovery and fault tolerance.

7. A distributed data stream asynchronous snapshot system based on flink according to claim 6, characterized in that: The system includes a snapshot algorithm that simulates execution stages by injecting special barrier markers into the data stream and captures the set of operating states of the processing nodes at the end of each stage to build a global snapshot, thereby ensuring the termination of the snapshot.

8. A distributed data stream asynchronous snapshot system based on flink according to claim 6, characterized in that: The system handles both acyclic data flows and data flows containing cycles. For the latter, the system identifies backward edges in the cycles through static analysis and performs additional processing on records received from backward edges during the snapshot to ensure that all relevant records are included in the snapshot and meet the feasibility requirements of the snapshot.

9. A distributed data stream asynchronous snapshot system based on flink according to claim 6, characterized in that: The system includes a snapshot management mechanism for generating, storing, retrieving and versioning snapshots, where snapshot data is persistently stored in a reliable storage medium so that the system state can be quickly restored when needed.

10. A distributed data stream asynchronous snapshot system based on flink according to claim 6, characterized in that: The system also includes a fault recovery mechanism that supports both full-graph recovery and partial-graph recovery. When a processing node fails, the latest global snapshot data is used to restore the system's operating status, while ensuring the data's exactly-once processing semantics to avoid repeated calculations.

Images