An Iterator Based on Optimistic Fault Tolerance

Abstract

The invention discloses an iterator based on an optimistic fault-tolerant method, which belongs to the technical field of distributed iterative computing in a big data environment. The iterator includes an incremental iterator and a batch iterator, and comprehensively considers iteration tasks of different sizes and different faults. For the iterative calculation task of rate, a compensation function is introduced, and the system uses this function to re-initialize the lost partitions. When a failure occurs, the system pauses the current iteration, ignores the failed tasks, and redistributes the lost computations to newly acquired nodes, calling the compensation function on the partition to restore a consistent state and resume execution. For situations where the fault frequency is low, the calculation delay is greatly reduced and the iterative processing efficiency is improved. For situations with high failure frequency, this iterator can ensure that the iteration processing efficiency is not lower than that of the iterator before optimization. This optimistic fault-tolerant iterator does not need to add additional operations to the task, effectively reducing the fault-tolerant overhead.

Description

technical field [0001] The invention relates to the technical field of distributed iterative computing in a big data environment, in particular to an iterator based on an optimistic fault-tolerant method. Background technique [0002] With the rapid development of Internet technology, iterative processing and analysis of massive data has great application value. With the continuous increase of computing nodes in distributed systems, the failure rate also increases. The design of iterators and fault-tolerant methods is one of the main bottlenecks restricting computing efficiency, and it is a key technology that cannot be ignored. [0003] In the prior art, when the iterator processes batch tasks, if a node fails or a task fails, the system will restart the iterative task according to the feature that the file can be accessed repeatedly. When processing stream computing tasks, fault tolerance is implemented based on distributed snapshots and partially resendable data sources...

AI Technical Summary

Problems solved by technology

Therefore, when the existing iterator has a small amount of tasks or a low probability of task failure, the checkpoint will bring a large additional overhead, which affects the real iterative computing efficiency.

In addition, only pessimistic fault-tolerant strategies are used in existing iterators, and there is no fault-tolerant scheme that comprehensively considers the size of iteration tasks and task failure rate

Images