High availability stream processing system and method

Abstract

The invention discloses a high availability stream processing system and method. The system comprises an upstream application system, a receiving device, a message-oriented middleware device, a stream processing topology device and a downstream application system, wherein the upstream application system is used for generating original data in real time according to the progress of an upstream business application and sending to the receiving device one by one; the receiving device is used for forwarding the original data to a message queue of the message-oriented middleware device; the message-oriented middleware device is used for caching the data for the stream processing topology device in a form of the message queue; the stream processing topology device is used for processing data stream of the original data and generating a data result; and the downstream application system is corresponding to a business scene and is used for carrying out follow-up processing on the data result.

Description

technical field [0001] The invention relates to the field of information processing, in particular to a highly available stream processing system and method. Background technique [0002] The term "streaming data" first appeared in the field of communication. With the development of information network technology, its definition and scope of application are constantly expanding. In the fields of data analysis, risk monitoring, and network security, the data sequences generated by various business systems arrive quickly, are generated continuously, and grow infinitely, which we also call streaming data. However, the traditional centralized processing IT architecture faces several difficulties when processing such data. First, enterprise users have new requirements for real-time analysis and processing of streaming data, that is, the total time spent on data from upstream application system business generation to downstream application system processing is no more than a seco...

AI Technical Summary

Problems solved by technology

The present invention designs and implements a topological distributed flow data computing framework, overcomes the problem of low efficiency of disk I / O during data transmission and processing, and effectively ensures real-time data by using local memory at each data flow processing node. Processing performance; the present invention improves the problem that traditional data processing methods are difficult to be compatible with various data analysis and processing scenarios, and loosely couples the basic framework module with the specific business logic module to provide solid underlying support for complex and diverse business models; The invention overcomes the problem of excessive single-point load of the traditional data processing system, adopts a decentralized architecture design, designs multiple nodes to perform the same business processing concurrently, and makes reasonable use of cluster computing resources; the invention overcomes the failure of distributed work nodes Due to difficulties such as unstable cluster work and data loss, the Zookeeper technology is used to realize the automatic fault recovery mechanism. At the same time, the fast memory calculation of Spark is used to realize the quasi-real-time search for lost data

Images