Fault tolerance method based on partial active backup of key task, and stream processing platform

A stream processing and task technology, which is used in electrical digital data processing, redundant operation in data error detection, and response error generation. It can solve problems such as high computing resource overhead and low resource utilization, and achieve shorten Average failure recovery time, fast failure recovery, and the effect of improving resource utilization

Active Publication Date: 2020-03-17
BEIJING SIMULATION CENT
View PDF4 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, although active backup can achieve rapid failure recovery in case of failure, its high computing resource overhead is often unacceptable in the current large-scale computing scenarios
In addition, storage nodes often have low utilization of CPU computing resources.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Fault tolerance method based on partial active backup of key task, and stream processing platform
  • Fault tolerance method based on partial active backup of key task, and stream processing platform
  • Fault tolerance method based on partial active backup of key task, and stream processing platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] In order to make the technical solutions and advantages in the embodiments of the present application clearer, the exemplary embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. Apparently, the described embodiments are only part of the embodiments of the present application, and Not an exhaustive list of all embodiments. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0032] When a fault occurs in the Flink stream processing platform, the entire application topology needs to be restarted, and the global consistent state of each operator is pulled back, and the source operator replays the corresponding data according to the pulled state. The global fault recovery time is relatively short. long. Active backup is the fault tolerance method with the fastest fault recovery time in r...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a fault tolerance method based on partial active backup of a key task and a stream processing platform, and the method comprises the steps: carrying out the backup of the position of the key task at least once; processing tuples generated by the same upstream instance by each backed-up key task position; and sending processing results of non-fault positions in all the key task positions after backup to the same downstream. According to the technical scheme, the average fault recovery time under the fault condition can be effectively shortened. When a fault occurs on an active backup vertex, rapid fault recovery can be realized in a second level. In addition, according to the scheme, the computing resource utilization rate on the state backup nodes can be fully utilized, and the resource utilization rate of the nodes can be effectively increased to 50% or above.

Description

technical field [0001] The application relates to the field of fault-tolerant backup of streaming platforms, in particular to a fault-tolerant method and a streaming processing platform based on active backup of key mission parts. Background technique [0002] As a paradigm of memory-based computing, stream computing mostly runs in a distributed environment, so it is extremely vulnerable to system failures. In a stream processing system, once the system detects the occurrence of a failure, the system must respond quickly and resume the normal processing of the continuously arriving stream data as soon as possible, otherwise it may not only cause data loss or even produce wrong calculation results. In traditional distributed stream processing systems, two fault-tolerant mechanisms, active backup and passive backup, are widely used. Active backup can complete instant switching from the primary instance to its backup instance in the event of a failure, ensuring the shortest fa...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/14
CPCG06F11/1448
Inventor 庆骁
Owner BEIJING SIMULATION CENT
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap