Failure recovery system and method of real-time streaming data processing based on memory data grid

A data processing and memory data technology, applied in digital transmission systems, data exchange networks, transmission systems, etc., can solve problems such as reducing failure recovery speed, and achieve the effect of reducing network delay, light snapshot mechanism, and reducing delay

Active Publication Date: 2016-08-17
INST OF SOFTWARE - CHINESE ACAD OF SCI
View PDF4 Cites 20 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

But it saves records on disk, which greatly reduces the speed of failure recovery

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Failure recovery system and method of real-time streaming data processing based on memory data grid
  • Failure recovery system and method of real-time streaming data processing based on memory data grid
  • Failure recovery system and method of real-time streaming data processing based on memory data grid

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0029] The failure recovery method of the real-time streaming data processing framework based on the memory data grid proposed by the present invention, on the Hazelcast memory data grid platform, performs operations such as converting and aggregating the vehicle information sent from the taxi equipment, through asynchronous snapshot Save the operating status information of the vehicle record and the micro-batch processing of the data window to achieve a lightweight, efficient, "exactly once" and as low-latency failure recovery method as possible.

[0030] The application framework of this example is as follows figure 1 shown. The Hazelcast used in the example of the present invention is an in-memory data grid that provides data backup, data partition and various distributed data structures, wherein JobContainer1 and JobContainer2 are distri...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

This invention relates to a failure recovery system and method of real-time streaming data processing based on a memory data grid. The system comprises a scheduler, a status server, a task queue and a task operation center. Since records and status information are separately stored, the status information stored in a snapshot is reduced; the delay of data processing is reduced, and simultaneously the snapshot mechanism is easier; the snapshot persistence of micro-batch is achieved by a data window; and the semantic guarantee of "just one time" is ensured through a storage structure of double-buffer; the network delay is reduced to some extent without using a third-party cluster to manage the data.

Description

technical field [0001] The present invention relates to a failure recovery system and method for real-time stream data processing, in particular to a light-weight, high-efficiency, "exactly once" failure recovery method based on a distributed stream processing framework, which belongs to the field of software technology. Background technique [0002] With the rapid development of information technology such as the Internet, the amount of global data has increased sharply. The mining of the hidden value behind these big data has become the mainstream of today's era. The batch calculation that appeared in the early days used the strategy of storing first and then computing. This method is accurate but unable to meet the requirements of real-time data. Stream computing has become the mainstream application to solve real-time data processing. The streaming processing framework is a data processing framework for processing streaming data with time series characteristics. A good ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L12/24
CPCH04L41/0661
Inventor 黄涛钟华魏峻王伟支孟轩郑莹莹
Owner INST OF SOFTWARE - CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap