Multi-source heterogeneous data real-time processing system and method based on Flink stream computing technology

A multi-source heterogeneous data, real-time processing technology, applied in the direction of electrical digital data processing, special data processing applications, computing, etc., can solve the problem that log data does not convert time series data, does not provide data interactive chart display interface, and does not appear multi-source Heterogeneous data and other issues to achieve the effect of improving efficiency

Pending Publication Date: 2019-09-17
CHINA TELECOM SHANGHAI IDEAL INFORMATION IND GRP
View PDF8 Cites 35 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] 1. It does not provide a visual graphical interface that can build data cleaning and segmentation rules, but implements it through configuration files
[0007] 2. The log data is not converted into structured time series data, so it is impossible to group and calcu

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-source heterogeneous data real-time processing system and method based on Flink stream computing technology
  • Multi-source heterogeneous data real-time processing system and method based on Flink stream computing technology
  • Multi-source heterogeneous data real-time processing system and method based on Flink stream computing technology

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0037] The implementation of the present invention is described below through specific examples and in conjunction with the accompanying drawings, and those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific examples, and various modifications and changes can be made to the details in this specification based on different viewpoints and applications without departing from the spirit of the present invention.

[0038] Before introducing the present invention, first introduce several open source components involved in the present invention:

[0039] 1. Elasticsearch

[0040] As the storage component of the ultimate log, it can realize distributed storage, real-time search and massive data analysis, and can persist the log to disk. Moreover, an open source RESTful API interface is provided to reali...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a multi-source heterogeneous data real-time processing system and method based on a Flink stream computing technology. The system comprises a data acquisition side which acquires heterogeneous data dispersed in a plurality of system components in a log mode and/or an SDK mode and/or an MQ mode, and sends the data to Kafka in a continuous flow mode after preliminary processing; a task management platform side which configures a data source type, configures cleaning and segmentation rules of heterogeneous data and dimensions and indexes of a configuration data set, starts a data real-time processing task based on an Flink stream computing technology after all configurations are completed, and defines and stores the data according to the data set after real-time computing of the data; and a data presentation and output side which obtains the result output of the data set. The system can analyze the data with multiple sources and different structures output in the existing service system to find the correlation between the log event and the service so as to help the operation and maintenance personnel improve the efficiency and provide supplementation for the existing service analysis system.

Description

technical field [0001] The present invention relates to a multi-source heterogeneous data real-time processing system and method, in particular to a multi-source heterogeneous data real-time processing system and method based on Flink stream computing technology. Background technique [0002] In the Internet+ era, in order to meet the needs of rapid business development and elastic scaling, the IT system architecture of enterprises is evolving towards Docker container clusters and microservices. This architecture improves resource utilization and brings greater flexibility. , Support high concurrency scenarios. [0003] However, with the expansion of business scale and the increase in the complexity of calling relationships between services, the amount of log output is increasing. When facing failures and performance problems, it is more difficult to analyze. Therefore, how to analyze the large amount of data output by the system It is an urgent problem to be solved to anal...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/2455G06F16/18
CPCG06F16/24568G06F16/1815
Inventor 肖荣马思峻陆晋军郑荣丁富强姚磊孙海
Owner CHINA TELECOM SHANGHAI IDEAL INFORMATION IND GRP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap