Edge flow system fault tolerance method based on parallel recovery

Abstract

The invention discloses an edge flow system fault tolerance method based on parallel recovery, and belongs to the technical field of edge computing. The problem that in the prior art, a traditional flow system fault tolerance method is low in fault recovery speed is solved. The method comprises the following steps: S1, replacing a fault operator with a warm backup replica operator corresponding to an added operator to complete parallel state reconstruction; s2, according to state transition based on fine granularity, a warm backup copy operator is adopted as a container operator, the operator in the step S1 is migrated into the container operator, and after the container operator carries out conversion, state recollection is completed; and S3, obtaining a parallel state recovery framework of the stream system through parallel state reconstruction and state recollection processes, and completing parallel recovery. According to the method, the fault recovery speed of the flow system is effectively improved, the warm backup replica operator is used for parallel reconstruction, the recovery delay is remarkably reduced through fine-grained migration and merging states, and the method can be applied to intelligent city planning and automatic driving.

Description

TECHNICAL FIELD

[0001] The application relates to an edge flow system fault-tolerant method, in particular to an edge flow system fault-tolerant method based on parallel recovery, and belongs to the technical field of edge computing. BACKGROUND

[0002] With the rapid development of 5G communication technology and the wide popularity of the Internet of Things, edge computing as a new computing paradigm has gradually become an important infrastructure for intelligent applications. In the edge environment, computing and storage resources are deployed near the network edge close to the data source; this deployment mode can significantly reduce the delay of data transmission and improve the real-time performance of data processing, thereby better meeting the requirements of smart cities, autonomous driving, remote medical care, industrial Internet of Things and other application scenarios that require high real-time performance and low latency.

[0003] In the edge environment, stream processing is widely used because it can continuously process real-time data streams from data sources; however, the computing resources in the edge environment are often limited, and the system usually needs to process a large number of heterogeneous data streams, which puts higher requirements on the reliability and fault tolerance of the stream computing system. In this context, traditional fault tolerance strategies such as checkpointing, rollback, causal logging and hot backup have great limitations in resource-constrained edge computing environments. Traditional methods not only cause high recovery delay, but also may not be able to normally complete tasks due to insufficient computing resources, so it is particularly important to study fault tolerance strategies suitable for resource-limited and real-time edge environments.

[0004] In the prior art, the checkpoint-based local rollback recovery technology Clonos solves the key fault tolerance problem in the stream processing system through innovative architecture design, and its core mechanism is shown in Figure 5 As shown, a master-slave operator cooperative working mode is adopted, the master operator is responsible for real-time processing and maintains the causal log, and the standby operator keeps a warm backup state through state snapshot synchronization. The Clonos system runs through a double-buffer design (after the network transmission buffer is transmitted, it is directly used as a flight log without copying, and thereafter the network transmission buffer will be in the next buffer), which effectively solves the flight log overhead problem, and an epoch segmentation mechanism is adopted to manage the flight record log (i.e. only the flight log after the last checkpoint is saved, and the rest of the flight log is deleted), which ensures that the data stream can be accurately replayed during fault recovery; for reference Figure 6The fault recovery process of Clonos includes six key steps: after the JobManager activates the standby operator within sub-second level, the Clonos system dynamically rebuilds the network connection and retrieves the downstream stored causal log, the upstream operator replays the non-persistent data by epoch, the standby operator ensures the consistency of the results of non-deterministic operations by using the causal log, and finally the duplicate records are eliminated through the output channel log; although the Clonos system has made a major breakthrough in the field of stream processing fault tolerance, there are still key deficiencies in the edge computing scene with extremely high real-time requirements: (1) although the Clonos uses a warm standby operator, a large amount of recovery time is still required for GB-level large states; (2) a large amount of time is consumed for replaying the flight log after completing state reconstruction; (3) a large number of tuples will be accumulated for processing after fault recovery, and it is too slow to digest only by the original configuration.

[0005] In summary, there is a need for an edge stream system fault tolerance method based on parallel recovery. SUMMARY

[0006] Hereinafter, a brief summary of the present application is given in order to provide a basic understanding of some aspects of the present application. It should be understood that this summary is not an exhaustive overview of the present application. It is not intended to identify key or important parts of the present application nor is it intended to limit the scope of the present application. Its purpose is merely to present some concepts in a simplified form as a prelude to the more detailed description that is discussed later.

[0007] In view of this, in order to solve the problem of slow fault recovery speed of the traditional stream system fault tolerance method in the prior art, the present application provides an edge stream system fault tolerance method based on parallel recovery.

[0008] The technical scheme is as follows: an edge stream system fault tolerance method based on parallel recovery, comprising the following steps:

[0009] S1. replacing the failed operator with the corresponding warm standby copy operator of the added operator to complete parallel state reconstruction;

[0010] S2. according to the state migration based on fine granularity, using the warm standby copy operator as a container operator to migrate the operator of step S1 to the container operator, and completing state recollection after the conversion of the container operator;

[0011] S3. obtaining the parallel state recovery framework of the stream system through the process of parallel state reconstruction and state recollection, and completing parallel recovery.

[0012] Further, in the S1, each operator additionally starts a warm standby operator during job deployment, i.e. by adding a standby execution on the execution graph, the operator enters a blocked state after job deployment, and is unblocked after being woken up by the jobManager, and performs a recovery task, and after being woken up by the jobManager, the operator performs the recovery task;

[0013] When a certain operator in the stream system encounters a failure, all warm standby operators of the same parallel instance participate in parallel recovery, the jobManager wakes up all warm standby operators through RPC, and each warm standby operator is responsible for reconstructing part of the state of the failed operator, and after being woken up by the jobManager, the warm standby operator modifies its data partition to be the same as the data partition allocated to the part of the state to be reconstructed, and obtains the corresponding state, and reconstructs the part of the state to be reconstructed, until the same state as at the beginning of the failure is restored, the warm standby operator establishes a connection with the upstream and downstream, and turns into a running state, and each warm standby operator temporarily replaces part of the data partition of the failed operator.

[0014] Further, in the S2, the jobManager regards the warm standby operator corresponding to the source operator in the S1 as the final container operator, and notifies all warm standby operators through a stream system call to migrate the state to the container operator, and the migration process includes the following steps:

[0015] S21. The jobManager injects control information barrier into the data channel of the source operator, and all source operators receiving the control information continue to send the control information to all downstream operators through the data channel;

[0016] S22. For the operator receiving the control information, if the downstream operator of the current operator needs to be scaled, the data connection with the container operator is established after receiving the control information, and the routing of the data partition to be migrated is changed to the container operator;

[0017] S23. For the operator receiving the control information, if the current operator itself needs to be migrated, it enters an aligning state after receiving the control information from a certain upstream operator, and starts to receive the key of the data partition of the corresponding upstream operator at this time, and if the local key does not exist, it actively sends a request to the operator holding the corresponding key;

[0018] S24. After receiving the control information of all upstream channels, the downstream operator enters the aligned state, no longer receives the old data partition, and starts to migrate the state to the new operator, i.e., sends the state to be migrated to the container operator in batches according to the set granularity;

[0019] S25. After the state migration is completed, the container operator sends a confirmation signal to the control node jobManager, and when the control node jobManager receives the confirmation signal from all container operators, the migration phase ends;

[0020] S26. After the migration phase ends, the container operator will be converted into a master operator, and a new replica operator is started as a replica operator of itself, and other standby replica operators are converted from the running state to the standby state to continue to block and release the consumed resources, and when the control node jobManager confirms that all container operators are converted, the state collection phase ends.

[0021] Further, in S2, when performing fine-grained state management, the overall key value key range is divided into a fixed number of key groups, each key group is assigned to an operator, and when the job performs checkpointing, the key value state of each operator is serialized into a distributed file system in a key group grouping sequence. For operator state, it will be serialized separately and stored in a distributed file system.

[0022] Further, in S2, when performing parallel recovery based on detachable state, when a certain operator fails, the control node first splits the to-be-reconstructed state of the failed operator into two parts, checkpoint and flight log, according to the time dimension, and assigns them to two standby replica operators. Then, through a stream system call, the standby replica operators are notified, and the two standby replica operators are responsible for recovering the checkpoint and flight log states of the failed operator, respectively. After being awakened by the control node, the standby replica operator immediately enters the running state, and uses the checkpoint in the distributed file system to reconstruct the checkpoint state, and uses the flight log replayed by the upstream operator to reconstruct the flight log state. After the two-part state is reconstructed, the state collection phase is completed to merge the two-part state, and parallel recovery is achieved;

[0023] The detachable state is defined as follows:

[0024] Set the processing logic of a certain operator to operation ⊕, which is formally represented as e -> SR -> (e’, SR’), where e- represents, SR - represents, R’ = R∪{e}, and SR represents the state generated after processing the same tuple set R. Under a certain splitting method, the state is merged through the ⊕ operation, i.e., x⊕y, where x and y represent the split states;

[0025] If an operation is commutative, for any elements x, y, x⊕y = y⊕x is satisfied.

[0026] If an operation is associative, for any elements x, y, z, x⊕(y⊕z) = (x⊕y)⊕z is satisfied.

[0027] When the operation is commutative and associative, it is called a spatially splittable state, and if only associative, it is called a temporally splittable state.

[0028] When the state R of an operator is tup1⊕tup2⊕tup3⊕tup4⊕tup5⊕tup6⊕tup7, a checkpoint is created after processing tup3, and a failure occurs after processing tup5, the state saved by the checkpoint is tup1⊕tup2⊕tup3, the flight log contains tup4 and tup5, and the tuples accumulated after the failure are tup6 and tup7.

[0029] If the current state of the operator R is a splittable state, it is represented as: R = (tup1⊕tup2⊕tup3)⊕(tup4⊕tup5⊕tup6⊕tup7), and R1 = (tup1⊕tup2⊕tup3), R2 = (tup4⊕tup5⊕tup6⊕tup7) are set, and R = R1⊕R2 is obtained.

[0030] The beneficial effects of the present application are as follows: the present application further accelerates the fault recovery speed of the stream system (uses parallel means to accelerate, uses the redundant resources of the system), that is, multiple warm standby operators are used to parallelly rebuild checkpoints, the state reconstruction speed is accelerated, multiple operators are used to parallelly consume flight logs, the recovery speed is accelerated, and multiple reconstructed operators are used to parallelly process accumulated tuples, the running speed is accelerated.

[0031] The parallel state recovery framework of the stream system of the present application, specifically mainly refers to the parallel state recovery based on a warm standby operator stream system and the state collection based on state migration, especially the process of warm standby-parallel reconstruction-state recollection; the present application combines checkpoint mechanisms, standby operators, causal log recording and fine-grained state migration mechanisms, realizes efficient fault recovery through parallel recovery and dynamic resource scheduling, the core concept of the present application is to split the state of a failed operator into multiple parts, use warm standby copy operators to parallelly rebuild, and then merge the state through fine-grained migration, thereby significantly reducing the recovery delay.

[0032] The fault recovery effect of the application is verified by overhead experiment and failure experiment, wherein the overhead experiment measures the overhead of the method (PRSP) in the application in terms of throughput and delay under normal operation, and the failure experiment studies the fault recovery efficiency of PRSP; in both cases, Flink native method and clonos are compared, the Flink native method is the basic engine introduced in the application, and the clonos is the current more advanced engine; reference Figure 7 In the overhead experiment, the performance of PRSP under normal operation (i.e. no fault) is observed, and the runtime management overhead is quantified, a complete Nexmark benchmark query is performed, the parallelism of each operator is set to 20, the average delay of different applications is tested in the overhead experiment, the test applications are DEBS, NEXMARK and WORDCOUNT respectively, from the overhead experiment results, in the case of relatively large throughput, the difference between the application and the Flink native method is about 5%, and in the case of relatively small throughput (not more than 10kb / s), the difference is within 1%, the throughput of the parallel recovery method of the application and the clonos method using causal log for local recovery is basically the same in various cases; then the average delay of different applications is tested, the test applications are DEBS, NEXMARK and WORDCOUNT respectively, from the experimental results, the difference between the application and the Flink native method is very small, within 5%. The average delay of the parallel recovery method and the clonos method using causal log for local recovery is basically the same; reference Figure 8 and Figure 9 For the failure experiment, the detailed throughput and delay indicators of two queries (Nexmark and DEBS) that are most worth paying attention to are selected; in addition, reference Figure 10 The recovery time of PRSP and Flink under multiple concurrent failure scenarios is compared using simulated workloads, the recovery time is defined as the time from the occurrence of failure to the recovery of the system to observe the delay within 10% of the delay before failure, this indicator is used to evaluate the ability of the mechanism to quickly recover from failure;

[0033] The verification experiment shows that on benchmark tests such as Nexmark and DEBS, the recovery time of PRSP is at least one order of magnitude faster than Flink (native method), and 33% faster than Clonos, while ensuring exactly-once semantics, PRSP is based on Apache Flink implementation, and partitions the state along the key range and time dimension, optimizing resource utilization, the application provides a lightweight and highly available solution for edge stream processing. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0035] Figure 1 A flowchart of a fault-tolerant method for an edge flow system based on parallel recovery;

[0036] Figure 2 A schematic diagram of the system architecture of the present application;

[0037] Figure 3 A schematic diagram of the system fault recovery process of the present application;

[0038] Figure 4 A schematic diagram of the fault recovery process under the state of splittable optimization;

[0039] Figure 5 A schematic diagram of the architecture of the Clonos system;

[0040] Figure 6 A schematic diagram of the fault recovery process of the Clonos system;

[0041] Figure 7 A schematic diagram of the comparison of the average processing time delay of the system of the present application and other systems;

[0042] Figure 8 A schematic diagram of the comparison of the average throughput of the system of the present application and other systems;

[0043] Figure 9 A schematic diagram of the comparison of the recovery time corresponding to the first group of data of the system of the present application and other systems;

[0044] Figure 10 A schematic diagram of the comparison of the recovery time corresponding to the second group of data of the system of the present application and other systems;

[0045] Figure 11 A schematic diagram of the comparison of the average processing time delay change in one fault recovery of the system of the present application and other systems. DETAILED DESCRIPTION

[0046] In order to make the technical solutions and advantages of the embodiments of the present application clearer, the following further describes the exemplary embodiments of the present application with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. It should be noted that the embodiments and features in the embodiments can be combined with each other without conflict.

[0047] REFERENCE Figures 1-10To elaborate the embodiment, the edge flow system fault-tolerant method based on parallel recovery specifically comprises the following steps:

[0048] S1. Complete parallel state reconstruction by replacing the failed operator with the corresponding warm standby operator added by the operator;

[0049] S2. According to the state migration based on fine granularity, migrate the operator in step S1 to the container operator by using the warm standby operator as the container operator, and complete state re-collection after the container operator is converted;

[0050] S3. Obtain the parallel state recovery framework of the flow system through the parallel state reconstruction and state re-collection process, and complete parallel recovery.

[0051] Specifically, the operator is the smallest unit of processing logic of a job in a distributed flow processing system, multiple jobs can run on a flow system, each job runs on multiple physical nodes, a job is divided into multiple tasks (i.e. operators) according to the processing logic, and one or more tasks can run on one physical node;

[0052] State: connection information of the operator and key-value pair data stored in the memory of the operator;

[0053] Checkpoint: The flow system will record the state of all operators regularly, and this recording action can be called checkpointing, and the persistent file containing all operator states is also called checkpoint.

[0054] Further, in S1, a warm standby operator is additionally started for each operator when the job is deployed, i.e. standby execution is realized by adding a warm standby operator on the execution graph, the operator enters a blocked state after the job is deployed, only consumes a small amount of system resources, and is unblocked after being awakened by the control node jobManager to perform the recovery task, the operator performs the recovery task after being awakened by the control node jobManager, and the time overhead of redeployment is saved;

[0055] When an operator in the stream system encounters a failure, all warm standby operators of the same parallel instance participate in parallel recovery. The control node jobManager wakes up all warm standby operators through remote procedure call (RPC), and each warm standby operator is responsible for reconstructing part of the state of the failed operator. After being woken up by the control node jobManager, the warm standby operator modifies its data partition to the same as the data partition assigned to the part of the state it is responsible for reconstructing, and obtains the corresponding state (such as checkpoint and causal log). The warm standby operator then reconstructs the part of the state it is responsible for, until it recovers to the same state as at the beginning of the failure. The warm standby operator establishes a connection with the upstream and downstream, and enters the running state. Each warm standby operator temporarily replaces part of the data partition of the failed operator.

[0056] Specifically, with the development of the field of stream processing, more and more selectable stream processing systems have emerged. However, Flink occupies a leading position in real-time processing among stream processing engines, and is known for its high-performance and low-latency stream processing capabilities. The present application is based on the native Flink and implements a parallel recovery framework based on operators and fine-grained migration, thereby solving the problem of slow recovery.

[0057] In the existing local recovery approach, checkpointing and causal logging are usually used for failure recovery. This method has greatly reduced the time overhead compared to the original global failure recovery strategy of Flink. However, it still cannot meet the needs of edge environments with high real-time requirements. Therefore, the present application introduces a parallel recovery strategy to use the idle resources of multiple nodes for parallel state reconstruction, thereby further improving the efficiency of state reconstruction.

[0058] In the traditional failure recovery approach, existing operators or new operators are usually used for state reconstruction. However, both of these implementations have many defects. First, in a large-scale stream processing system, the time overhead of deploying a new operator is huge. Although this time is considered as a fixed constant time overhead, this part of the overhead cannot be ignored and will greatly reduce the efficiency of state reconstruction. Second, although using existing operators for state reconstruction eliminates the overhead of deploying new operators, the selected operators still need to perform a series of switching operations, which is also very time-consuming. Therefore, a parallel state reconstruction approach based on operators is introduced. In traditional stream processing systems, one logical operator in a logical execution flow graph corresponds to one physical operator in a physical execution flow graph. Referring to Figure 2 After the addition of warm standby operators in the present application, one logical operator in a logical execution flow graph will correspond to two physical operators in a physical execution flow graph, one of which is a main operator current execution and the other is a replica operator. The warm standby operator is deployed on different taskManagers as much as possible to avoid simultaneous failure with the main operator, and to make use of the idle resources of multiple nodes as much as possible.

[0059] Further, in S2, the control node jobManager takes the warm standby operator corresponding to the operator in S1 as the final container operator, and notifies all warm standby operators of the state migration to the container operator through a system call of the stream. The migration process includes the following steps:

[0060] S21. The control node jobManager injects control information barrier into the data channel of the source operator, and all source operators receiving the control information continue to send the control information to all downstream operators through the data channel.

[0061] S22. For the operator receiving the control information, if the downstream operator of the current operator needs to be scaled, the data connection with the container operator is established after receiving the control information, and the routing of the data partition that needs to be migrated is changed to the container operator.

[0062] S23. For the operator receiving the control information, if the current operator itself needs to be migrated, it enters the aligning state when receiving the control information from a certain upstream operator, and starts to receive the key of the data partition corresponding to the upstream operator. If the local key does not exist, the operator actively sends a request to the operator holding the corresponding key.

[0063] S24. After the downstream operator receives the control information from all upstream channels, it enters the aligned state and no longer receives the old data partition. At the same time, it starts to migrate the state to the new operator, i.e., sends the state to be migrated to the container operator in batches according to the set granularity.

[0064] S25. After the state migration is completed, the container operator sends a confirmation signal to the control node jobManager. When the control node jobManager receives the confirmation signal from all container operators, the migration phase ends.

[0065] S26. After the migration phase ends, the container operator will be converted into a master operator and no longer serve as a warm standby operator. A new standby operator is started as its own standby operator. Other warm standby operators are converted from the running state to the standby state and continue to block, releasing the consumed resources. When the control node jobManager confirms that all container operators have been converted, the state collection phase ends.

[0066] Specifically, after step S1, the state of the fault operator is reconstructed on multiple warm standby operators, which is not conducive to unified management, and the resource overhead of the warm standby operator after being converted to a running state is greatly increased, and there is a certain resource waste when multiple warm standby operators run simultaneously to replace the data partition of the fault operator; therefore, the states on multiple warm standby operators need to be collected into one operator (i.e., one elastic scaling is performed) to solve the data partition and resource waste problem, and the application introduces state recollection based on a fine-grained migration mechanism.

[0067] In the current more advanced elastic scaling mechanism of the stream system, there is a no-downtime elastic scaling mechanism, but the current elastic scaling mechanism is usually for all data partitions, not for specified partial data partitions, so an improved partial elastic scaling based on fine-grained migration is introduced.

[0068] Reference Figure 3 Based on the parallel reconstruction and recollection of the states of steps S1-S2, the application realizes a parallel recovery framework of the stream system, when a certain operator fails, the control node notifies each warm standby operator through a system call, each warm standby operator is responsible for recovering part of the state of the fault operator, the warm standby operator is immediately converted to a running state after being awakened by the control node, and obtains the persistent storage required for reconstructing the responsible state, the warm standby operator reconstructs the part of the state it is responsible for, and establishes a connection with the upstream and downstream operators according to the operator state, to replace part of the function of the fault operator.

[0069] After completing the state reconstruction in the checkpoint, the upstream operator replays the causal log and the flight log to recover the state lost from the last checkpoint to the failure time. In the process of parallel reconstruction, the function of the original fault operator is replaced by multiple replica operators,

[0070] After each replica operator recovers to the state at the time of failure, a fine-grained state migration is performed to smoothly collect the reconstructed states in each replica operator into the replica operator of the fault operator. The data partition of the container operator will be gradually modified to be the same as the data partition of the fault operator, until completely replacing the fault operator.

[0071] After the state reconstruction and recollection are completed, the control node will notify each replica operator to convert to a blocking state, the container operator converts to a master operator, and replaces the function of the original operator, and the parallel recovery is completed.

[0072] Further, in the fine-grained state management (sending the state to be migrated to the container operator in batches according to the set granularity), the overall key-value key range is divided into a fixed number of key groups, each key group is assigned to an operator, and when the job performs checkpointing, the key-value state of each operator is serialized and stored in the distributed file system in the form of key group grouping, instead of being serialized and stored according to the operator granularity in the traditional method. For the operator state, it will be serialized and stored in the distributed file system separately, instead of being stored together with the key-value state.

[0073] Specifically, the application designs splitting based on fine-grained state management, introduces fine-grained state management, that is, state management in the unit of key group; in the traditional stream processing system, when the job performs checkpointing, the state is usually serialized in the unit of operator and stored in the distributed file system (HDFS), and the checkpointing storage content of the traditional stream system is divided into two parts, operator state and keyed state. The operator state mainly includes information such as the upstream and downstream of the operator and the held resources, and the keyed state is the state stored by the operator in a broad sense, wherein the keyed state responsible for each operator is a continuous part of the overall keyed range, the keyed range of each operator is disjointed, and the union of all operator keyed ranges is the overall keyed range, and the keyed range of each operator is managed as a whole.

[0074] Further, in the parallel recovery based on the splittable state, when an operator fails, the control node first splits the state to be reconstructed of the failed operator into two parts, checkpoint and flight log, according to the time dimension, and assigns them to two warm standby operators, and then notifies each warm standby operator through the stream system call. The two warm standby operators are responsible for recovering the checkpoint and flight log states of the failed operator, respectively. After being awakened by the control node, the warm standby operator immediately enters the running state, and uses the checkpoint in the distributed file system to reconstruct the checkpoint state, and uses the flight log replayed by the upstream operator to reconstruct the flight log state. After the two parts of the state are reconstructed, the state is collected again to complete the merging of the two parts of the state, and the parallel recovery is realized.

[0075] The splittable state is defined as follows:

[0076] The processing logic of an operator is set to operation, which is formally represented as e -> SR -> (e', SR'), wherein e- represents, SR - represents, R' = R∪{e}, and SR represents the state generated after processing the tuples with the same set R. Under a certain splitting mode, the state is merged by the operation, that is, x, y at this time x, y represent the split state;

[0077] If the operation is satisfied for any element x, y, x⊕y = y⊕x, the operation is called commutative;

[0078] If the operation is satisfied for any element x, y, z, x⊕(y⊕z) = (x⊕y)⊕z, the operation is called associative;

[0079] When the operation is commutative and associative, it is called a spatially decomposable state; if only associative, it is called a temporally decomposable state;

[0080] When the state R of an operator is tup1⊕tup2⊕tup3⊕tup4⊕tup5⊕tup6⊕tup7, a checkpoint is created after processing tup3, and a failure occurs after processing tup5, the state saved by the checkpoint is tup1⊕tup2⊕tup3, the flight log contains tup4 and tup5, and the tuples accumulated after the failure are tup6 and tup7;

[0081] If the current state of the operator R is a decomposable state, it is represented as: R = (tup1⊕tup2⊕tup3)⊕(tup4⊕tup5⊕tup6⊕tup7), set R1 = (tup1⊕tup2⊕tup3), R2 = (tup4⊕tup5⊕tup6⊕tup7), and R = R1⊕R2 is obtained.

[0082] Specifically, for a fault operator in a stream processing system, its state can be decomposed into two parts in the time dimension: a checkpoint part and a flight log part, both of which are reconstructed in parallel by a warm standby replica operator, and finally merged by state collection;

[0083] According to the definition of the decomposable state, in the fault recovery of the stream system, R1 and R2 can be recovered in parallel, and finally merged into the complete state, that is, if the state is decomposable, the checkpoint and the tuples in the flight log or the real-time tuples in the data stream can be processed in parallel in the actual recovery process, without waiting for the checkpoint to be reconstructed before starting processing, that is, the present application can establish a temporary state during the reconstruction of the checkpoint, and then merge the two states to obtain the final state after the checkpoint of the operator is reconstructed; In this case, real-time tuples can be processed without waiting for the checkpoint of the operator to be reconstructed, greatly reducing the processing delay of tuples during fault recovery, and this method allows the checkpoint and the flight log to be processed in parallel, further improving the parallel degree of recovery, and better utilizing the idle resources of the system to achieve faster recovery speed;

[0084] Reference Figure 2Each operator has multiple backup operators, the state of the master operator is stored in multiple backup operators, and the output queue is recorded in the flight log and truncated periodically for fault recovery;

[0085] Reference Figure 3 After the master operator encounters a failure, the backup operator is activated first, then the network connection upstream and downstream is reconfigured to connect with the backup operator, then the backup operator uses the stored backup to restore the state, then the flight log is played back, and finally the state is collected again to complete the fault recovery;

[0086] Reference Figure 4 After the master operator encounters a failure, the backup operator is activated first, then the network connection upstream and downstream is reconfigured to connect with the backup operator, and Figure 3 The difference is that the backup operator uses the stored backup to restore the state and play back the flight log in two steps in parallel, and the last step is to collect the state again to complete the fault recovery;

[0087] Reference Figure 5 The master and backup operators work in cooperation, the master operator is responsible for real-time processing and maintaining the causal log, and the backup operator keeps the warm backup state through state snapshot synchronization; When the system runs, the double buffer design (after the network transmission buffer is transmitted, it is directly used as the flight log without copying, and thereafter the network transmission buffer will be in the next buffer) effectively solves the flight log overhead problem, and the epoch segmentation mechanism is adopted to manage the flight record log, ensuring accurate data flow playback during fault recovery;

[0088] Reference Figure 6 It includes six key steps: the JobManager activates the backup operator, the system dynamically rebuilds the network connection, retrieves the stored causal log downstream, the upstream operator replays the unpersisted data by epoch, the backup operator uses the causal log to ensure the consistency of non-deterministic operation results, and finally the output channel log is used to eliminate duplicate records;

[0089] Reference Figure 7 The average processing delay of the present application is slightly better than that of the Clonos system, and the difference compared with the non-fault-tolerant flink is also small (proving that the fault-tolerant overhead is small);

[0090] Reference Figure 8 The average throughput of the present application is slightly better than that of the Clonos system, and the difference compared with the non-fault-tolerant flink is also small (proving that the fault-tolerant overhead is small);

[0091] Reference Figure 9The recovery time of the present application under the first group of data is obviously superior to the Clonos system, and is more than one order of magnitude different from the time without fault tolerance (flink) (much superior to flink).

[0092] With reference to Figure 10 The recovery time of the present application under the second group of data is still obviously superior to the Clonos system, and is more than one order of magnitude different from the time without fault tolerance (flink) (much superior to flink).

[0093] With reference to Figure 11 The fault recovery speed of the present application is slightly superior to the Clonos system, and is obviously superior to the time without fault tolerance (flink).

[0094] Although the present application has been described in terms of limited number of embodiments, those skilled in the art will appreciate that other embodiments are contemplated within the scope of the present application described herein. In addition, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes and can not have been selected to expressly convey the scope of the present application. Accordingly, numerous modifications and alterations are possible in the embodiments of the present application without departing from the scope and spirit of the appended claims. The disclosure of the present application is illustrative only and not restrictive of the scope of the present application, which is defined by the appended claims.

Claims

1. A fault-tolerant method for edge streaming systems based on parallel recovery, characterized in that, Includes the following steps: S1. Parallel state reconstruction is completed by replacing the faulty operator with the warm backup replica operator corresponding to the added operator; S2. Based on fine-grained state transition, the warm backup replica operator is used as the container operator. The operator in step S1 is transferred to the container operator. After the container operator is transformed, the state is collected again. S3. Through the process of parallel state reconstruction and state recollection, the parallel state recovery framework of the flow system is obtained, and parallel recovery is completed.

2. The fault-tolerant method for edge streaming systems based on parallel recovery according to claim 1, characterized in that, In S1, when the job is deployed, each operator additionally starts a warm standby replica operator, which is achieved by adding a warm standby replica operator standby execution to the execution graph. After the job is deployed, the operator enters a blocked state until it is awakened by the control node jobManager and then unblocked to execute the recovery task. After being awakened by the control node jobManager, the operator executes the recovery task. When an operator in the streaming system encounters a failure, all warm backup replica operators of the same parallel instance participate in parallel recovery. The control node jobManager wakes up all warm backup replica operators via remote RPC. Each warm backup replica operator is responsible for rebuilding part of the state of the failed operator. After being woken up by the control node jobManager, the warm backup replica operator modifies its own data partition to be the same as the data partition it is responsible for rebuilding, obtains the corresponding state, and rebuilds the part of the state it is responsible for until it is restored to the same state as when the failure started. The warm backup replica operator establishes connections with upstream and downstream, and enters the running state. Each warm backup replica operator temporarily replaces part of the data partition of the failed operator.

3. The fault-tolerant method for edge streaming systems based on parallel recovery according to claim 2, characterized in that, In step S2, the control node jobManager takes the warm backup replica operator corresponding to the source operator (i.e., the operator in step S1) as the final container operator, and notifies all warm backup replica operators through a streaming system call to migrate the state to the container operator. The migration process includes the following steps: S21. Control information barrier is injected into the data channel of the source operator through the control node jobManager. All source operators that receive the control information continue to send the control information to all downstream operators through the data channel. S22. For an operator that receives control information, if the downstream operator of the current operator needs to be scaled, after receiving the control information, establish a data connection with the container operator and change the route of the data partition that needs to be migrated to the container operator. S23. For operators that receive control information, if the current operator itself needs to perform state transition, it enters the alignment state when it receives control information from an upstream operator. At this time, it starts to receive the key of the data partition of the corresponding upstream operator. If the corresponding key does not exist locally, it actively sends a request to the operator that holds the corresponding key. S24. After receiving control information from all upstream channels, the downstream operator enters the aligned state and no longer receives old data partitions. At the same time, it begins to migrate the state to the new operator, that is, it sends the state to be migrated to the container operator in batches according to the set granularity. S25. After the state transition is completed, the container operator sends an acknowledgment signal to the control node jobManager. When the control node jobManager receives acknowledgment signals from all container operators, the transition phase ends. S26. After the migration phase ends, the container operator will be converted into the primary operator and start a new replica operator as its own replica operator. Other warm standby replica operators will be converted from the running state to the standby replica state and continue to be blocked, releasing the consumed resources. When the control node jobManager confirms that all container operators have been converted, the state collection phase ends.

4. The fault-tolerant method for edge streaming systems based on parallel recovery according to claim 3, characterized in that, In S2, when performing fine-grained state management, the overall key value range is divided into a fixed number of key groups. Each key group is assigned to an operator. When the job performs a checkpoint, the key value state of each operator is serialized and stored in the distributed file system by key group. For operator states, they are serialized and stored in the distributed file system separately.

5. The fault-tolerant method for edge streaming systems based on parallel recovery according to claim 4, characterized in that, In S2, during parallel recovery based on divisible states, when an operator fails, the control node first splits the state to be rebuilt of the failed operator into two parts, checkpoint and flight log, according to the time dimension, and assigns them to two warm backup replica operators. Then, it notifies each warm backup replica operator through a stream system call. The two warm backup replica operators are responsible for restoring the checkpoint and flight log states of the failed operator, respectively. After being woken up by the control node, the warm backup replica operators immediately switch to the running state and rebuild the checkpoint state using the checkpoint in the distributed file system and rebuild the flight log state using the flight log replayed by the upstream operator. After the two states are rebuilt, the two states are merged in the state recollection phase to achieve parallel recovery. The specific definition of a separable state is as follows: Let the processing logic of a certain operator be the operation ⊕, formally represented as e -> SR -> (e', SR'), where e - represents, SR - represents, R' = R∪{e}, and SR represents the state generated after processing the set R with the same tuples. Under a certain splitting method, the states are merged by the ⊕ operation, i.e., x⊕y, where x and y represent the split states; If the operation ⊕ satisfies x⊕y = y⊕x for any elements x and y, then the operation ⊕ is said to be commutative. If the operation ⊕ satisfies x⊕(y⊕z) = (x⊕y)⊕z for any elements x, y, z, then the operation ⊕ is said to be associative. When the operation ⊕ has both commutativity and associativity, it is called a spatially separable state; if it only has associativity, it is called a temporally separable state. When the state R of a certain operator is tup1⊕tup2⊕tup3⊕tup4⊕tup5⊕tup6⊕tup7, it creates a checkpoint after processing tup3, and fails after processing tup5. The state saved by the checkpoint is tup1⊕tup2⊕tup3. The flight log contains tup4 and tup5, and the tuples accumulated after the failure are tup6 and tup7. If the current operator's state R is a separable state, it is represented as: R = (tup1⊕tup2⊕tup3)⊕(tup4⊕tup5⊕tup6⊕tup7). Let R1 = (tup1⊕tup2⊕tup3) and R2 = (tup4⊕tup5⊕tup6⊕tup7), then we get R = R1⊕R2.

Images