A data stream multidirectional processing system based on Flink

A technology for processing systems and data streams, which is used in electrical digital data processing, special data processing applications, digital data information retrieval, etc. Added extended effects

Inactive Publication Date: 2019-05-03
BEIJING INST OF COMP TECH & APPL
View PDF4 Cites 12 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The throughput and fault tolerance of the traditional streaming data processing framework have inherent defects, and are no longer suitable for the rapidly expanding business needs of industries such as the Internet

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data stream multidirectional processing system based on Flink
  • A data stream multidirectional processing system based on Flink

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] In order to make the purpose, content, and advantages of the present invention clearer, the specific implementation manners of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

[0029] The present invention realizes a set of distributed data flow processing system, which has the characteristics of distributed, large throughput and low delay, and can calculate and process related business data accurately and in real time, thereby accelerating the computing power of the system.

[0030] The system is divided into three modules: data cache module, data multi-directional processing module, and data storage module. The data caching module is used for caching data collected from different sources and forwarding to the data multi-directional processing module. The data multidirectional processing module is used to receive data from the data cache module, perform multi-dimensional processing and analy...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a data stream multidirectional processing system based on Flink, and relates to the technical field of real-time data processing. The invention provides a Kafka / Flink / Elasticsearch-based data stream processing system for processing multiple services by a single consumer, which can efficiently and accurately process large-scale data in real time. By utilizing high expansibility and high reliability of Kafka, data of a plurality of data sources can be accurately collected and summarized, and newly-added expansion is easy to realize;and the Kafka can perform persistence operation to persist the message to a disk, so that the probability of data loss is greatly reduced. Through the efficient combination of the flink and the Flink, the distributed type can be upgraded to a form of multi-consumption and multi-direction service data processing, the data processing capability of the flink serving as a consumer is greatly expanded, and meanwhile, the rapidity of calculation and storage is also ensured. The flink-based single consumer data stream processing system not only has excellent performance on a single node, but also can present amazing analysis efficiency ina distributed manner, so that the directional processing and analysis range of the traditional algorithm and the overall capacity of quick storage are expanded.

Description

technical field [0001] The invention relates to the technical field of real-time data processing, in particular to a Flink-based multi-directional data flow processing system. Background technique [0002] With the advent of the cloud era, big data has also attracted more and more attention. Big data requires special techniques to efficiently handle large volumes of data that tolerate elapsed time. Technologies applicable to big data, including massively parallel processing (MPP) databases, data mining grids, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action streaming data in consumer-scale websites. Flink is a distributed processing engine for streaming and batch data. ElasticSearch is a Lucene-based search server designed for cloud computing, capable of real-time search, stable, reliable, fast, and eas...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/332G06F9/54
Inventor 李志强石波胡佳谢小明丁卫星徐晶
Owner BEIJING INST OF COMP TECH & APPL
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap