A highly reliable distributed data flow real-time statistical method and system

Abstract

The invention discloses a high-reliability distributed data stream real-time statistical method and system. Three technologies are included, wherein the first technology is a distributed data stream calculation model based on a MapReduce programming model, the second technology is a concurrent data transmission mechanism with serial numbers, and the third technology is a distributed task management and scheduling mechanism based on states and signals. According to the first technology, the ideology of a MapReducer model is expanded to a cluster concept, each Map or Reducer calculation unit is a node in a distributed cluster and is called a Mapper or a Reducer, all the Mapper nodes form a Mapper cluster, and all the Reducer nodes form a Reducer cluster. Through the first technology, throughput is realized by guaranteeing the expandability of the distributed system; through the second technology and the third technology, data reliability and task availability are realized, and therefore reliable semantics are guaranteed.

Description

technical field [0001] The invention relates to a highly reliable distributed data flow real-time statistical method and system, belonging to the field of big data technology. Background technique [0002] When it comes to statistical processing of data streams, most of the current solutions in the industry are based on distributed memory computing. This is because the concurrency of distributed systems can well deal with large-scale data streams, using memory computing instead of traditional The local or distributed file system is because the system needs to process the data flowing into the system as soon as possible, because in the data flow computing scenario, the data flows at a high frequency and is time-sensitive. When a packet of data flows past, it cannot be retrieved again. package data. Under such a distributed memory computing architecture, there are many problems that need to be solved, from the compromise of consistency, availability, and partition tolerance, ...

AI Technical Summary

Problems solved by technology

Existing products such as s4, storm, and spark have been recognized and applied by the industry to a certain extent, but their performance is not satisfactory in more demanding scenarios

[0003] The shortcomings of S4 mainly include two shortcomings. The first is reliability. S4 can only guarantee at-most-once semantics. When the processing node is down, the task can be transferred, but all the data in the memory will be lost.

In addition, S4 cannot dynamically expand nodes, which is unacceptable for a distributed system

[0004] There are two problems with Storm: one is the single point of failure brought about by the weakly centralized structure. After the nimbus node or ui node goes down, there will be problems in the execution of statistical tasks; the other is semantic guarantee, that is, reliability. The trident mechanism is provided to guarantee the exactly-once semantics, but this mechanism will seriously affect the processing performance

[0005] There are also two problems with Spark Streaming: first, the processing delay is high, and second-level delays cannot meet certain application scenarios that require high real-time performance; second, in terms of semantic guarantee, it is necessary to use specific data sources to ensure that lost data

Images