Iterator based on optimistic fault tolerant method

Abstract

The invention discloses an iterator based on an optimistic fault tolerant method. The invention belongs to the technical field of distributed iterative computation in a big data environment. The iterator comprises an incremental iterator and a batch iterator, comprehensively considers iterative tasks of different sizes and iterative computation tasks of different failure rates, introduces a compensation function, and enables a system to use the function to re-initialize lost partitions. And when a fault occurs, the system suspends the current iteration, ignores the failed task, reallocates thelost calculation to the newly obtained node, and calls the compensation function on the partition to recover the consistent state and recover execution. And for the condition of relatively low faultfrequency, the calculation delay is greatly reduced, and the iterative processing efficiency is improved. For the condition of higher fault frequency, the iterator can ensure that the iterative processing efficiency is not lower than that of the iterator before optimization. The iterator based on an optimistic fault tolerant method does not need to add additional task operation, thus effectively reducing the fault-tolerant overhead.

Description

technical field [0001] The invention relates to the technical field of distributed iterative computing in a big data environment, in particular to an iterator based on an optimistic fault-tolerant method. Background technique [0002] With the rapid development of Internet technology, iterative processing and analysis of massive data has great application value. With the continuous increase of computing nodes in distributed systems, the failure rate also increases. The design of iterators and fault-tolerant methods is one of the main bottlenecks restricting computing efficiency, and it is a key technology that cannot be ignored. [0003] In the prior art, when the iterator processes batch tasks, if a node fails or a task fails, the system will restart the iterative task according to the feature that the file can be accessed repeatedly. When processing stream computing tasks, fault tolerance is implemented based on distributed snapshots and partially resendable data sources...

AI Technical Summary

Problems solved by technology

Therefore, when the existing iterator has a small amount of tasks or a low probability of task failure, the checkpoint will bring a large additional overhead, which affects the real iterative computing efficiency.

In addition, only pessimistic fault-tolerant strategies are used in existing iterators, and there is no fault-tolerant scheme that comprehensively considers the size of iteration tasks and task failure rate

Images