High availability stream processing system and method

A stream processing and application system technology, applied in the field of information processing, can solve the problems of unstable cluster work, low efficiency, and difficulty in being compatible with various data analysis and processing scenarios, so as to overcome the instability of cluster work, improve stability, and ensure The effect of data real-time processing performance

Active Publication Date: 2016-09-21
INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF4 Cites 42 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The present invention designs and implements a topological distributed flow data computing framework, overcomes the problem of low efficiency of disk I/O during data transmission and processing, and effectively ensures real-time data by using local memory at each data flow processing node. Processing performance; the present invention improves the problem that traditional data processing methods are difficult to be compatible with various data analysis and processing scenarios, and loosely couples the basic framework module with the specific business logic module to provide solid underlying support for complex and diverse

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • High availability stream processing system and method
  • High availability stream processing system and method
  • High availability stream processing system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0037] The technical means adopted by the present invention to achieve the intended invention purpose are further described below in conjunction with the drawings and preferred embodiments of the present invention.

[0038] figure 1 It is a schematic structural diagram of a highly available stream processing system according to an embodiment of the present invention. Such as figure 1 As shown, the system includes: an upstream application system 101, a receiving device 102, a message middleware device 103, a stream processing topology device 104, a downstream application system 105, a heartbeat detection device 106, a system guard device 107, and a system information persistence device 108 , a massive log management device 109 and a log reconciliation device 110 . Among them, the upstream application system 101 and the downstream application system 105 are external devices connected with this system. In order to ensure that the core device of the system, that is, the stream ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a high availability stream processing system and method. The system comprises an upstream application system, a receiving device, a message-oriented middleware device, a stream processing topology device and a downstream application system, wherein the upstream application system is used for generating original data in real time according to the progress of an upstream business application and sending to the receiving device one by one; the receiving device is used for forwarding the original data to a message queue of the message-oriented middleware device; the message-oriented middleware device is used for caching the data for the stream processing topology device in a form of the message queue; the stream processing topology device is used for processing data stream of the original data and generating a data result; and the downstream application system is corresponding to a business scene and is used for carrying out follow-up processing on the data result.

Description

technical field [0001] The invention relates to the field of information processing, in particular to a highly available stream processing system and method. Background technique [0002] The term "streaming data" first appeared in the field of communication. With the development of information network technology, its definition and scope of application are constantly expanding. In the fields of data analysis, risk monitoring, and network security, the data sequences generated by various business systems arrive quickly, are generated continuously, and grow infinitely, which we also call streaming data. However, the traditional centralized processing IT architecture faces several difficulties when processing such data. First, enterprise users have new requirements for real-time analysis and processing of streaming data, that is, the total time spent on data from upstream application system business generation to downstream application system processing is no more than a seco...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L12/24H04L12/26
CPCH04L41/0654H04L41/069H04L43/10
Inventor 袁一沈贇陶玮张学舟
Owner INDUSTRIAL AND COMMERCIAL BANK OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap