A fault-tolerant method for edge flow system based on parallel recovery
Patent Information
- Application Number
- CN202511652357.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-12
- Publication Date
- 2026-11-03
- Estimated Expiration
- 2045-11-12
AI Technical Summary
Existing technologies have slow fault recovery speeds for stream processing systems in edge computing environments, especially in terms of large state recovery and tuple stacking processing, which are inefficient and cannot meet the requirements of application scenarios with high real-time requirements.
A parallel recovery-based approach is adopted, which involves adding a warm backup replica operator to rebuild the state in parallel, using multiple warm backup replica operators to rebuild checkpoints and flight logs in parallel, and combining this with a fine-grained state transition mechanism to realize a parallel state recovery framework.
It significantly accelerates the fault recovery speed of the streaming system, reduces recovery delay, improves resource utilization, and meets the real-time requirements of the edge computing environment.
Smart Images
Figure CN121523799B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a fault-tolerant method for edge streaming systems, and more particularly to a fault-tolerant method for edge streaming systems based on parallel recovery, belonging to the field of edge computing technology. Background Technology
[0002] With the rapid development of 5G communication technology and the widespread adoption of the Internet of Things (IoT), edge computing, as a new computing paradigm, is gradually becoming an important infrastructure for intelligent applications. In an edge environment, computing and storage resources are deployed at the network edge close to the data source; this deployment model can significantly reduce data transmission latency and improve the real-time performance of data processing, thereby better meeting the application scenarios with extremely high requirements for real-time performance and low latency, such as smart cities, autonomous driving, telemedicine, and industrial IoT.
[0003] In edge environments, stream processing is widely used because it can continuously process real-time data streams from data sources. However, computing resources in edge environments are often limited, while systems typically need to process large amounts of heterogeneous data streams, which places higher demands on the reliability and fault tolerance of stream computing systems. Against this backdrop, traditional fault tolerance strategies, such as checkpointing, rollback, causal logging, and hot backup, exhibit significant limitations in resource-constrained edge computing environments. Traditional methods not only lead to high recovery latency but may also fail to complete tasks normally due to insufficient computing resources. Therefore, researching fault tolerance strategies suitable for resource-constrained and real-time-critical edge environments is particularly important.
[0004] In existing technologies, the checkpoint-based local rollback recovery technology Clonos solves a key fault tolerance problem in stream processing systems through innovative architectural design. Its core mechanism is as follows: Figure 5 As shown, a master-slave operator collaborative working mode is adopted. The master operator is responsible for real-time processing and maintenance of the causal log, while the standby operator maintains a warm backup state through state snapshot synchronization. The Clonos system utilizes a dual-buffer design (after a network transmission buffer is completed, it is directly used as the flight log without copying; subsequent network transmission buffering occurs in the next buffer), effectively solving the flight log overhead problem. Simultaneously, an epoch segmentation mechanism is used to manage the flight log (i.e., only the flight log after the most recent checkpoint is saved, and the rest are deleted), ensuring accurate replay of the data stream during fault recovery; see reference. Figure 6The Clonos fault recovery process includes six key steps: After the JobManager activates the standby operator in sub-seconds, the Clonos system dynamically rebuilds the network connection and retrieves the causal logs stored downstream. The upstream operator replays the non-persistent data by epoch. The standby operator uses the causal logs to ensure the consistency of nondeterministic operation results. Finally, duplicate records are eliminated by outputting the channel logs. Although the Clonos system has made a major breakthrough in the field of stream processing fault tolerance, it still has key shortcomings in edge computing scenarios with extremely high real-time requirements: (1) Although clonos uses warm standby operators, it still requires a lot of recovery time for GB-level large states; (2) After the state reconstruction is completed, the flight logs must be replayed, which also consumes a lot of time; (3) After the fault recovery, a large number of tuples will accumulate to be processed, and it is too slow to digest them by relying solely on the original configuration.
[0005] In summary, a fault-tolerant method for edge streaming systems based on parallel recovery is needed. Summary of the Invention
[0006] A brief overview of the invention is given below to provide a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.
[0007] In view of this, in order to solve the problem of slow fault recovery speed in traditional fault tolerance methods for streaming systems in the prior art, this invention provides a fault tolerance method for edge streaming systems based on parallel recovery.
[0008] The technical solution is as follows: A fault-tolerant method for edge streaming systems based on parallel recovery, comprising the following steps:
[0009] S1. Parallel state reconstruction is completed by replacing the faulty operator with the warm backup replica operator corresponding to the added operator;
[0010] S2. Based on fine-grained state transition, the warm backup replica operator is used as the container operator. The operator in step S1 is transferred to the container operator. After the container operator is transformed, the state is collected again.
[0011] S3. Through the process of parallel state reconstruction and state recollection, the parallel state recovery framework of the flow system is obtained, and parallel recovery is completed.
[0012] Furthermore, in S1, during job deployment, each operator additionally starts a warm standby replica operator, which is achieved by adding a warm standby replica operator standby execution to the execution graph. After job deployment, the operator enters a blocked state until it is awakened by the control node jobManager and then unblocked to execute the recovery task. After being awakened by the control node jobManager, the operator executes the recovery task.
[0013] When an operator in the streaming system encounters a failure, all warm backup replica operators of the same parallel instance participate in parallel recovery. The control node jobManager wakes up all warm backup replica operators via remote RPC. Each warm backup replica operator is responsible for rebuilding part of the state of the failed operator. After being woken up by the control node jobManager, the warm backup replica operator modifies its own data partition to be the same as the data partition it is responsible for rebuilding, obtains the corresponding state, and rebuilds the part of the state it is responsible for until it is restored to the same state as when the failure started. The warm backup replica operator establishes connections with upstream and downstream, and enters the running state. Each warm backup replica operator temporarily replaces part of the data partition of the failed operator.
[0014] Further, in step S2, the control node jobManager takes the warm backup replica operator corresponding to the source operator (i.e., the operator in step S1) as the final container operator, and notifies all warm backup replica operators through a streaming system call to migrate the state to the container operator. The migration process includes the following steps:
[0015] S21. Control information barrier is injected into the data channel of the source operator through the control node jobManager. All source operators that receive the control information continue to send the control information to all downstream operators through the data channel.
[0016] S22. For an operator that receives control information, if the downstream operator of the current operator needs to be scaled, after receiving the control information, establish a data connection with the container operator and change the route of the data partition that needs to be migrated to the container operator.
[0017] S23. For operators that receive control information, if the current operator itself needs to perform state transition, it enters the alignment state when it receives control information from an upstream operator. At this time, it starts to receive the key of the data partition of the corresponding upstream operator. If the corresponding key does not exist locally, it actively sends a request to the operator that holds the corresponding key.
[0018] S24. After receiving control information from all upstream channels, the downstream operator enters the aligned state and no longer receives old data partitions. At the same time, it begins to migrate the state to the new operator, that is, it sends the state to be migrated to the container operator in batches according to the set granularity.
[0019] S25. After the state transition is completed, the container operator sends an acknowledgment signal to the control node jobManager. When the control node jobManager receives acknowledgment signals from all container operators, the transition phase ends.
[0020] S26. After the migration phase ends, the container operator will be converted into the primary operator and start a new replica operator as its own replica operator. Other warm standby replica operators will be converted from the running state to the standby replica state and continue to be blocked, releasing the consumed resources. When the control node jobManager confirms that all container operators have been converted, the state collection phase ends.
[0021] Furthermore, in S2, when performing fine-grained state management, the overall key value range is divided into a fixed number of key groups, and each key group is assigned to an operator. When the job performs a checkpoint, the key value state of each operator is serialized and stored in the distributed file system by key group. For the operator state, it will be serialized and stored in the distributed file system separately.
[0022] Furthermore, in S2, during parallel recovery based on divisible states, when an operator fails, the control node first splits the state to be rebuilt of the failed operator into two parts, checkpoint and flight log, according to the time dimension, and assigns them to two warm backup replica operators. Then, it notifies each warm backup replica operator through a stream system call. The two warm backup replica operators are responsible for restoring the checkpoint and flight log states of the failed operator, respectively. After being woken up by the control node, the warm backup replica operators immediately switch to the running state and rebuild the checkpoint state using the checkpoint in the distributed file system and rebuild the flight log state using the flight log replayed by the upstream operator. After the two states are rebuilt, the two states are merged in the state recollection phase to achieve parallel recovery.
[0023] The specific definition of a separable state is as follows:
[0024] Let the processing logic of a certain operator be the operation ⊕, formally represented as e -> SR -> (e', SR'), where e represents a tuple, R' = R∪{e}, and SR represents the state generated after processing the set R of tuples with the same tuples. Under a certain splitting method, the states are merged by the ⊕ operation, i.e., x⊕y, where x and y represent the split states.
[0025] If the operation ⊕ satisfies x⊕y = y⊕x for any elements x and y, then the operation ⊕ is said to be commutative.
[0026] If the operation ⊕ satisfies x⊕(y⊕z) = (x⊕y)⊕z for any elements x, y, z, then the operation ⊕ is said to be associative.
[0027] When the operation ⊕ has both commutativity and associativity, it is called a spatially separable state; if it only has associativity, it is called a temporally separable state.
[0028] When the state R of a certain operator is tup1⊕tup2⊕tup3⊕tup4⊕tup5⊕tup6⊕tup7, it creates a checkpoint after processing tup3, and fails after processing tup5. The state saved by the checkpoint is tup1⊕tup2⊕tup3. The flight log contains tup4 and tup5, and the tuples accumulated after the failure are tup6 and tup7.
[0029] If the current operator's state R is a separable state, it is represented as: R = (tup1⊕tup2⊕tup3)⊕(tup4⊕tup5⊕tup6⊕tup7). Let R1 = (tup1⊕tup2⊕tup3) and R2 = (tup4⊕tup5⊕tup6⊕tup7), then we get R = R1⊕R2.
[0030] The beneficial effects of this invention are as follows: This invention further accelerates the fault recovery speed of the flow system (by using parallel methods and employing redundant resources of the system), that is, by using multiple warm standby operators to rebuild checkpoints in parallel, the state reconstruction speed is accelerated; by using multiple operators to consume flight logs in parallel, the recovery speed is accelerated; and by using multiple reconstructed operators to process piled-up tuples in parallel, the running speed is accelerated.
[0031] The parallel state recovery framework for streaming systems of this invention mainly refers to parallel state recovery and state collection based on state transition of streaming systems based on warm standby operators, especially the process of warm standby-parallel reconstruction-state re-collection. This invention combines checkpoint mechanism, standby operator, causal logging and fine-grained state transition mechanism to achieve efficient fault recovery through parallel recovery and dynamic resource scheduling. The core concept of this invention is to split the state of the fault operator into multiple parts, use warm standby replica operator to rebuild in parallel, and then merge the states through fine-grained transition, thereby significantly reducing recovery latency.
[0032] This invention verifies the fault recovery effect of the proposed method through overhead experiments and failure experiments. The overhead experiments measure the throughput and latency overhead of the proposed method (PRSP) under normal operation, while the failure experiments investigate the fault recovery efficiency of PRSP. In both cases, comparisons are made with Flink's native method and clonos. Flink's native method is the foundational engine introduced in this invention, while clonos is currently a more advanced engine. (References) Figure 7 In the overhead experiments, the performance of PRSP under normal operation (i.e., no failures) was observed, and the runtime management overhead was quantified. A complete Nexmark benchmark query was executed, with the parallelism of each operator set to 20. The average latency of different applications was compared in the overhead experiments, with DEBS, NEXMARK, and WORDCOUNT as the test applications. From the overhead experiment results, under relatively high throughput, the difference between the present invention and the native Flink method is about 5%, while under relatively low throughput (not exceeding 10kb / s), the difference is within 1%. The throughput of the parallel recovery method of the present invention is basically the same as that of the clonos method using causal logs for local recovery under various conditions. Next, the average latency of different applications was compared, with DEBS, NEXMARK, and WORDCOUNT as the test applications. From the experimental results, the difference between the present invention and the native Flink method is very small, within 5%. The average latency of the parallel recovery method is basically the same as that of the clonos method using causal logs for local recovery. Figure 8 and Figure 9 For the failure experiment, detailed throughput and latency metrics for two of the most noteworthy queries (Nexmark and DEBS) are selected for display; additionally, refer to Figure 10 We compared the recovery times of PRSP and Flink under multiple concurrent failure scenarios using simulated workloads. Recovery time was defined as the time from the occurrence of a failure to the observed latency of the recovery system being reduced to within 10% of the latency before the failure. This metric is used to evaluate the mechanism's ability to recover quickly from failures.
[0033] Verification experiments show that, on benchmark tests such as Nexmark and DEBS, PRSP's recovery time is at least an order of magnitude faster than Flink (the native method) and 33% faster than Clonos, while guaranteeing exactly-once semantics. PRSP is implemented based on Apache Flink and optimizes resource utilization by partitioning the state along the key range and time dimension. This invention provides a lightweight and highly available solution for edge stream processing. Attached Figure Description
[0034] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:
[0035] Figure 1 This is a flowchart illustrating a fault-tolerant method for edge streaming systems based on parallel recovery.
[0036] Figure 2 This is a schematic diagram of the system architecture of the present invention;
[0037] Figure 3 This is a schematic diagram of the system fault recovery process of the present invention;
[0038] Figure 4 This is a schematic diagram of the fault recovery process under the optimization of the shardable state.
[0039] Figure 5 This is a schematic diagram of the Clonos system architecture;
[0040] Figure 6 This is a schematic diagram of the Clonos system's fault recovery process;
[0041] Figure 7 This is a schematic diagram comparing the average processing latency of the system of this invention with that of other existing systems;
[0042] Figure 8 This is a schematic diagram comparing the average throughput of the system of this invention with that of other current systems;
[0043] Figure 9 This is a schematic diagram comparing the recovery time of the first set of data from the system of this invention with that of other existing systems;
[0044] Figure 10 This is a schematic diagram comparing the recovery time of the second set of data of the system of the present invention with that of other existing systems;
[0045] Figure 11 This is a schematic diagram comparing the average processing delay changes in a single fault recovery process between the system of this invention and other existing systems. Detailed Implementation
[0046] To make the technical solutions and advantages of the embodiments of the present invention clearer, the exemplary embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0047] refer to Figures 1-10This embodiment describes a fault-tolerant method for edge streaming systems based on parallel recovery, specifically including the following steps:
[0048] S1. Parallel state reconstruction is completed by replacing the faulty operator with the warm backup replica operator corresponding to the added operator;
[0049] S2. Based on fine-grained state transition, the warm backup replica operator is used as the container operator. The operator in step S1 is transferred to the container operator. After the container operator is transformed, the state is collected again.
[0050] S3. Through the process of parallel state reconstruction and state recollection, the parallel state recovery framework of the flow system is obtained, and parallel recovery is completed.
[0051] Specifically, an operator is the smallest unit of processing logic in a distributed stream processing system. Multiple jobs can run on a single stream system, and each job runs on multiple physical nodes. A job is divided into multiple tasks (i.e., operators) based on its processing logic, and one or more tasks can run on a single physical node.
[0052] State: The operator's connection information and the key-value pair data stored in the operator's memory;
[0053] Checkpoints: Streaming systems periodically record the state of all operators. This recording action can be called a checkpoint, and the persistent file containing the state of all operators is also called a checkpoint.
[0054] Furthermore, in S1, during job deployment, each operator additionally starts a warm standby replica operator, which is achieved by adding a warm standby replica operator to the execution graph. After job deployment, the operator enters a blocked state, consuming only a very small amount of system resources, until it is awakened by the control node jobManager and then unblocked to execute the recovery task. After being awakened by the control node jobManager, the operator executes the recovery task, eliminating the time overhead of redeployment.
[0055] When an operator in the streaming system encounters a failure, all warm backup replica operators of the same parallel instance participate in parallel recovery. The control node jobManager wakes up all warm backup replica operators via remote RPC. Each warm backup replica operator is responsible for rebuilding part of the state of the failed operator. After being woken up by the control node jobManager, the warm backup replica operator modifies its own data partition to be the same as the data partition it is assigned to rebuild, obtains the corresponding state (such as checkpoints, causal logs), and rebuilds the part of the state it is responsible for until it is restored to the same state as when the failure started. The warm backup replica operator establishes connections with upstream and downstream, and enters the running state. Each warm backup replica operator temporarily replaces part of the data partition of the failed operator.
[0056] Specifically, with the development of the stream processing field, there are more and more stream processing systems to choose from. However, Flink occupies a leading position in real-time processing among stream processing engines, and is known for its high performance and low latency stream processing capabilities. This invention chooses to rewrite the original Flink and implements a parallel recovery framework based on operators and fine-grained migration, which solves the problem of slow recovery.
[0057] In existing local recovery approaches, checkpoints and causal logs are typically used for fault recovery. This method has significantly reduced time overhead compared to Flink's native global fault recovery strategy, but it still falls short of meeting the needs of edge environments with high real-time requirements. Therefore, this invention introduces a parallel recovery strategy, which utilizes the idle resources of multiple nodes to perform parallel state reconstruction, further improving the efficiency of state reconstruction.
[0058] Traditional fault recovery approaches typically involve reconstructing the state using existing operators or by starting a new operator. However, both approaches have several drawbacks. First, deploying a new operator in a large-scale stream processing system incurs significant time overhead. Although this time is considered a fixed constant, it is still considerable and significantly reduces the efficiency of state reconstruction. Second, while using existing operators eliminates the overhead of deploying a new operator, the selected operator still requires a series of switching operations, which are still costly. Therefore, a parallel state reconstruction approach based on operators is introduced. In traditional stream processing systems, a logical operator in the logical execution flow graph corresponds to a physical operator in the physical execution flow graph. (Refer to...) Figure 2 After adding the warm standby replica operator in this invention, one logical operator in the logical execution flow graph will correspond to two physical operators in the physical execution flow graph, one of which is the main operator (current execution) and the other is the replica operator. The warm standby replica operator will be deployed on different worker nodes (taskManager) as much as possible to avoid failure with the main operator and to make full use of the idle resources of multiple nodes.
[0059] Further, in step S2, the control node jobManager takes the warm backup replica operator corresponding to the source operator (i.e., the operator in step S1) as the final container operator, and notifies all warm backup replica operators through a streaming system call to migrate the state to the container operator. The migration process includes the following steps:
[0060] S21. Control information barrier is injected into the data channel of the source operator through the control node jobManager. All source operators that receive the control information continue to send the control information to all downstream operators through the data channel.
[0061] S22. For an operator that receives control information, if the downstream operator of the current operator needs to be scaled, after receiving the control information, establish a data connection with the container operator and change the route of the data partition that needs to be migrated to the container operator.
[0062] S23. For operators that receive control information, if the current operator itself needs to perform state transition, it enters the alignment state when it receives control information from an upstream operator. At this time, it starts to receive the key of the data partition of the corresponding upstream operator. If the corresponding key does not exist locally, it actively sends a request to the operator that holds the corresponding key.
[0063] S24. After receiving control information from all upstream channels, the downstream operator enters the aligned state and no longer receives old data partitions. At the same time, it begins to migrate the state to the new operator, that is, it sends the state to be migrated to the container operator in batches according to the set granularity.
[0064] S25. After the state transition is completed, the container operator sends an acknowledgment signal to the control node jobManager. When the control node jobManager receives acknowledgment signals from all container operators, the transition phase ends.
[0065] S26. After the migration phase is completed, the container operator will be converted into the primary operator and will no longer serve as a warm standby replica operator. A new replica operator will be started as its own replica operator. Other warm standby replica operators will be converted from the running state to the replica state standby and continue to be blocked, releasing the resources consumed. When the control node jobManager confirms that all container operators have been converted, the state collection phase ends.
[0066] Specifically, after step S1, the state of the faulty operator is reconstructed on multiple warm backup replica operators, which is not conducive to unified management. Moreover, the resource overhead of the warm backup replica operators increases significantly after they are put into running state. There is a certain waste of resources when multiple warm backup replica operators run simultaneously to replace the data partition of the faulty operator. Therefore, it is necessary to collect the state on multiple warm backup replica operators into one operator (i.e., perform one elastic scaling) to solve the data partitioning and resource waste problems. For this purpose, the present invention introduces a state re-collection based on a fine-grained migration mechanism.
[0067] Among the more advanced elastic scaling mechanisms for streaming systems, there are already no-downtime elastic scaling mechanisms. However, current elastic scaling mechanisms usually target all data partitions, rather than specific data partitions. Therefore, an improved partial elastic scaling based on fine-grained migration has been introduced.
[0068] refer to Figure 3 Based on the parallel reconstruction and recollection of the state in steps S1-S2, this invention realizes a parallel recovery framework for the streaming system. When an operator fails, the control node notifies each warm backup replica operator through a system call. Each warm backup replica operator is responsible for restoring a part of the state of the failed operator. After being woken up by the control node, the warm backup replica operator immediately switches to the running state and obtains the persistent storage required to rebuild the state it is responsible for. The warm backup replica operator persists the state and rebuilds the part of the state it is responsible for, and establishes connections with upstream and downstream operators according to the operator state, replacing some of the functions of the failed operator.
[0069] After completing state reconstruction at the checkpoint, the upstream operator will replay the causal log and flight log to restore the state lost from the last checkpoint to the time of the failure. During parallel reconstruction, the function of the original fault operator is replaced by multiple replica operators.
[0070] After each replica operator recovers to its state at the time of the failure, a fine-grained state transition will be performed, smoothly collecting the reconstructed states from each replica operator into the replica operator of the failed operator. The data partitions of the container operator will be gradually modified to be the same as the data partitions of the failed operator until the failed operator is completely replaced.
[0071] After state reconstruction and recollection are completed, the control node will notify each replica operator to switch to a blocked state, the container operator to switch to a master operator and replace the function of the original operator, and the parallel recovery will be completed.
[0072] Furthermore, when performing fine-grained state management (sending states to be migrated to container operators in batches according to a set granularity), the overall key range is divided into a fixed number of key groups, and each key group is assigned to an operator. When the job performs a checkpoint, the key state of each operator is serialized and stored in the distributed file system by key group, instead of being serialized and stored at the operator level in the traditional method. For operator states, they will be serialized and stored separately in the distributed file system, instead of being stored together with the key state.
[0073] Specifically, this invention designs a decomposition based on fine-grained state management, introducing fine-grained state management, that is, managing state at the key group level. In traditional stream processing systems, when a job performs checkpoints, the state is usually serialized at the operator level and stored in a distributed file system (HDFS). The checkpoint storage content of traditional stream systems is divided into two parts: operator state and keyed state. Operator state mainly includes information such as the upstream and downstream of the operator and the resources it holds. Keyed state refers to the state stored by the operator in a broad sense. The keyed state that each operator is responsible for is a continuous part of the overall keyed state range. The keyed state ranges of each operator do not intersect, and the concatenation of the keyed state ranges of all operators is the overall keyed state range. The keyed state range of each operator is managed as a whole.
[0074] Furthermore, in parallel recovery based on divisible state, when an operator fails, the control node first splits the state to be rebuilt of the failed operator into two parts, checkpoint and flight log, according to the time dimension, and assigns them to two warm backup replica operators. Then, it notifies each warm backup replica operator through a stream system call. The two warm backup replica operators are responsible for restoring the checkpoint and flight log parts of the failed operator, respectively. After being woken up by the control node, the warm backup replica operators immediately switch to the running state and rebuild the checkpoint state using the checkpoint in the distributed file system and rebuild the flight log state using the flight log replayed by the upstream operator. After the two parts of the state are rebuilt, the state recollection phase is used to merge the two parts of the state and realize parallel recovery.
[0075] The specific definition of a separable state is as follows:
[0076] Let the processing logic of a certain operator be the operation ⊕, formally represented as e -> SR -> (e', SR'), where e represents a tuple, R' = R∪{e}, and SR represents the state generated after processing the set R of tuples with the same set of tuples. Under a certain splitting method, states are merged through the ⊕ operation, i.e., x⊕y, where x and y represent the split states;
[0077] If the operation ⊕ satisfies x⊕y = y⊕x for any elements x and y, then the operation ⊕ is said to be commutative.
[0078] If the operation ⊕ satisfies x⊕(y⊕z) = (x⊕y)⊕z for any elements x, y, z, then the operation ⊕ is said to be associative.
[0079] When the operation ⊕ has both commutativity and associativity, it is called a spatially separable state; if it only has associativity, it is called a temporally separable state.
[0080] When the state R of a certain operator is tup1⊕tup2⊕tup3⊕tup4⊕tup5⊕tup6⊕tup7, it creates a checkpoint after processing tup3, and fails after processing tup5. The state saved by the checkpoint is tup1⊕tup2⊕tup3. The flight log contains tup4 and tup5, and the tuples accumulated after the failure are tup6 and tup7.
[0081] If the current operator's state R is a separable state, it is represented as: R = (tup1⊕tup2⊕tup3)⊕(tup4⊕tup5⊕tup6⊕tup7). Let R1 = (tup1⊕tup2⊕tup3) and R2 = (tup4⊕tup5⊕tup6⊕tup7) to get R = R1⊕R2.
[0082] Specifically, for fault operators in a stream processing system, their state can be split into two parts in the time dimension: checkpoint part and flight log part. Both parts are reconstructed in parallel by warm backup replica operators and finally merged through state recollection.
[0083] According to the definition of a separable state, in the fault recovery of a streaming system, the two parts of state R1 and R2 can be recovered in parallel, and then merged into a complete state. That is, if the state is separable, in the actual recovery process, checkpoints and tuples in the flight log or real-time tuples in the data stream can be processed in parallel without waiting for the checkpoint reconstruction to be completed before processing begins. In other words, this invention can establish a temporary state during the checkpoint reconstruction process, and then merge the two parts of state after the operator's checkpoint reconstruction is completed to obtain the final state. In this case, real-time tuples can be processed without waiting for the operator's checkpoint reconstruction to be completed, which greatly reduces the processing latency of tuples during fault recovery. Furthermore, this method allows checkpoints and flight logs to be processed separately in parallel, further improving the parallelism of recovery and making better use of the system's idle resources to achieve a faster recovery speed.
[0084] refer to Figure 2Each operator has multiple backup operators. The state of the main operator is backed up and stored in multiple backup operators. At the same time, the output queue will be recorded in the flight log and truncated periodically for fault recovery.
[0085] refer to Figure 3 When the master operator encounters a failure, the backup operator is activated first. Then, the upstream and downstream network connections are reconfigured to connect with the backup operator. The backup operator then uses the stored backup to rebuild the state, replays the flight log, and finally collects the state again, collecting the state into one operator to complete the fault recovery.
[0086] refer to Figure 4 When the primary operator encounters a failure, the backup operator is activated first, and then the upstream and downstream network connections are reconfigured to connect with the backup operator. Figure 3 The difference is that the backup operator will perform two steps in parallel: state reconstruction and flight log replay using the stored backup. The final step is also to collect the state again and collect the state into an operator to complete the fault recovery.
[0087] refer to Figure 5 The system adopts a master-slave operator collaborative working mode. The master operator is responsible for real-time processing and maintenance of causal logs, while the standby operator maintains a warm backup state through state snapshot synchronization. During system operation, the system effectively solves the flight log overhead problem through a dual-buffer design (after a network transmission buffer is completed, it is directly used as the flight log without copying, and the subsequent network transmission buffer will be in the next buffer). At the same time, the system adopts an epoch segmentation mechanism to manage the flight log, ensuring that the data stream can be accurately replayed during fault recovery.
[0088] refer to Figure 6 It consists of six key steps: JobManager activates standby operators, the system dynamically rebuilds network connections, retrieves causal logs stored downstream, upstream operators replay unpersisted data by epoch, standby operators use causal logs to ensure consistency of nondeterministic operation results, and finally eliminates duplicate records through output channel logs.
[0089] refer to Figure 7 The average processing latency of this invention is slightly better than that of the Clonos system, and the difference is not significant compared with that of the non-fault-tolerant system (Flink) (proving that the fault-tolerant overhead is small).
[0090] refer to Figure 8 The average throughput of this invention is slightly better than that of the Clonos system, and the difference is not significant compared to the system without fault tolerance (Flink) (proving that the fault tolerance overhead is small).
[0091] refer to Figure 9The recovery time of this invention is significantly better than that of the Clonos system under the first set of data, and is more than an order of magnitude better than that of the fault-intolerant system (Flink).
[0092] refer to Figure 10 The recovery time of this invention under the second set of data is still significantly better than that of the Clonos system, and more than an order of magnitude better than that of the fault-intolerant system (Flink).
[0093] refer to Figure 11 The fault recovery speed of this invention is slightly better than that of the Clonos system, and significantly better than that of the non-fault-tolerant system (Flink).
[0094] Although the invention has been described with reference to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of interpreting or limiting the subject matter of the invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the invention is illustrative and not restrictive, and the scope of the invention is defined by the appended claims.
Claims
1. A fault-tolerant method for edge streaming systems based on parallel recovery, characterized in that, Includes the following steps: S1. Parallel state reconstruction is completed by replacing the faulty operator with the warm backup replica operator corresponding to the added operator; S2. Based on fine-grained state transition, the warm backup replica operator is used as the container operator. The operator in step S1 is transferred to the container operator. After the container operator is transformed, the state is collected again. S3. Through the process of parallel state reconstruction and state recollection, the parallel state recovery framework of the flow system is obtained, and parallel recovery is completed; In S1, when the job is deployed, each operator additionally starts a warm standby replica operator, which is achieved by adding a warm standby replica operator standby execution to the execution graph. After the job is deployed, the operator enters a blocked state until it is awakened by the control node jobManager and then unblocked to execute the recovery task. After being awakened by the control node jobManager, the operator executes the recovery task. When an operator in the streaming system encounters a failure, all warm backup replica operators of the same parallel instance participate in parallel recovery. The control node jobManager wakes up all warm backup replica operators through remote RPC calls. Each warm backup replica operator is responsible for rebuilding part of the state of the failed operator. After being woken up by the control node jobManager, the warm backup replica operator modifies its own data partition to be the same as the data partition it is assigned to rebuild, obtains the corresponding state, and rebuilds the part of the state it is responsible for until it is restored to the same state as when the failure started. The warm backup replica operator establishes connections with upstream and downstream, and enters the running state. Each warm backup replica operator temporarily replaces part of the data partition of the failed operator. In step S2, the control node jobManager takes the warm backup replica operator corresponding to the source operator (i.e., the operator in step S1) as the final container operator, and notifies all warm backup replica operators through a streaming system call to migrate the state to the container operator. The migration process includes the following steps: S21. Control information barrier is injected into the data channel of the source operator through the control node jobManager. All source operators that receive the control information continue to send the control information to all downstream operators through the data channel. S22. For an operator that receives control information, if the downstream operator of the current operator needs to be scaled, after receiving the control information, establish a data connection with the container operator and change the route of the data partition that needs to be migrated to the container operator. S23. For operators that receive control information, if the current operator itself needs to perform state transition, it enters the alignment state when it receives control information from an upstream operator. At this time, it starts to receive the key of the data partition of the corresponding upstream operator. If the corresponding key does not exist locally, it actively sends a request to the operator that holds the corresponding key. S24. After receiving control information from all upstream channels, the downstream operator enters the aligned state and no longer receives old data partitions. At the same time, it begins to migrate the state to the new operator, that is, it sends the state to be migrated to the container operator in batches according to the set granularity. S25. After the state transition is completed, the container operator sends an acknowledgment signal to the control node jobManager. When the control node jobManager receives acknowledgment signals from all container operators, the transition phase ends. S26. After the migration phase is completed, the container operator will be converted into the master operator and start a new replica operator as its own replica operator. Other warm standby replica operators will be converted from the running state to the standby replica state and continue to be blocked, releasing the consumed resources. When the control node jobManager confirms that all container operators have been converted, the state collection phase ends. In S2, when performing fine-grained state management, the overall key value range is divided into a fixed number of key groups, and each key group is assigned to an operator. When the job performs a checkpoint, the key value state of each operator is serialized and stored in the distributed file system by key group. For the operator state, it will be serialized and stored in the distributed file system separately. In S2, during parallel recovery based on divisible states, when an operator fails, the control node first splits the state to be rebuilt of the failed operator into two parts, checkpoint and flight log, according to the time dimension, and assigns them to two warm backup replica operators. Then, it notifies each warm backup replica operator through a stream system call. The two warm backup replica operators are responsible for restoring the checkpoint and flight log states of the failed operator, respectively. After being woken up by the control node, the warm backup replica operators immediately switch to the running state and rebuild the checkpoint state using the checkpoint in the distributed file system and rebuild the flight log state using the flight log replayed by the upstream operator. After the two states are rebuilt, the two states are merged in the state recollection phase to achieve parallel recovery. The specific definition of a separable state is as follows: Let the processing logic of a certain operator be the operation ⊕, formally represented as e -> SR -> (e', SR'), where e represents a tuple, R' = R∪{e}, and SR represents the state generated after processing the set R of tuples with the same tuples. Under a certain splitting method, the states are merged by the ⊕ operation, i.e., x⊕y, where x and y represent the split states. If the operation ⊕ satisfies x⊕y = y⊕x for any elements x and y, then the operation ⊕ is said to be commutative. If the operation ⊕ satisfies x⊕(y⊕z) = (x⊕y)⊕z for any elements x, y, z, then the operation ⊕ is said to be associative. When the operation ⊕ has both commutativity and associativity, it is called a spatially separable state; if it only has associativity, it is called a temporally separable state. When the state R of a certain operator is tup1⊕tup2⊕tup3⊕tup4⊕tup5⊕tup6⊕tup7, it creates a checkpoint after processing tup3, and fails after processing tup5. The state saved by the checkpoint is tup1⊕tup2⊕tup3. The flight log contains tup4 and tup5, and the tuples accumulated after the failure are tup6 and tup7. If the current operator's state R is a separable state, it is represented as: R = (tup1⊕tup2⊕tup3)⊕(tup4⊕tup5⊕tup6⊕tup7). Let R1 = (tup1⊕tup2⊕tup3) and R2 = (tup4⊕tup5⊕tup6⊕tup7), then we get R = R1⊕R2.