Stream data processing method, system, apparatus, and computer-readable storage medium

A processing method and technology of a processing system, applied in the field of stream data processing and computer-readable storage media, can solve problems such as insufficient functions, single Flink function, and complex use of FlinkCEP, and achieve improved accuracy, tight integration, and rapid deployment. Effect

Active Publication Date: 2019-01-22
HANGZHOU ANHENG INFORMATION TECH CO LTD
View PDF7 Cites 31 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Pure Flink has a single function, Flink SQL has limited functions in practical applications, and Flink CEP uses relatively complex functions and is not complete enough, which is not convenient for quickly carrying out related businesses based on streaming data such as system logs, file transfers, and device status monitoring.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Stream data processing method, system, apparatus, and computer-readable storage medium
  • Stream data processing method, system, apparatus, and computer-readable storage medium
  • Stream data processing method, system, apparatus, and computer-readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0050]Combine the following figure 1 , figure 1 A flowchart of a method for processing stream data provided in the embodiment of the present application, which specifically includes the following steps:

[0051] S101: Obtain stream data from the message queue;

[0052] The purpose of this step is to use the stream data output by the message queue as the input data of the Flink framework. The purpose is to first perform some processing on the initial stream data with the help of the data processing capability of the message queue, so as to share part of the processing pressure of the Flink framework and reduce the Some data processing difficulties.

[0053] Messages are sent to a queue, a message queue (MQ) is a container that holds messages during their transmission, a message queue manager acts as a middleman in relaying messages from its source to its destination, and a queue The main purpose of the message queue is to provide routing and guarantee the delivery of the mes...

Embodiment 2

[0072] Combine the following figure 2 , figure 2 A flowchart of another method for processing stream data provided in the embodiment of the present application. On the basis of Embodiment 1, this embodiment takes stream data such as logs as an example, and provides a method using Kafka as an example through S201 and S202. The method of how to obtain stream data in message queues. S204 provides a method for receiving SQL commands, and S205 provides a method for outputting analysis results to ElasticSearch and implementing reports and information feedback with ElasticSearch. Need to explain What’s more, these three parts can form corresponding embodiments based on Embodiment 1 alone. This embodiment is only one of the embodiments in which there are three preferred modes at the same time. The specific embodiment steps are as follows:

[0073] S201: Use Kafka to process initial stream data in a topical manner to obtain stream data;

[0074] The initial log data is processed wi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

A stream data processing method method includes: getting stream data from a message queue, that is, the output data of the message queue is the input data of Flink, processing initial flow data once with the help of a message queue, and then parsing the received stream data into key-value pair events one by one by using preset data parsing rules, in order to preserve the data and their corresponding relationships with the widely used relational databases, which is convenient for data processing based on SQL, At the same time, the data analysis engine is utilized to analyze the data from many aspects and angles, so as to improve the accuracy of the analysis result. On the basis of the Flink framework which applies the data stream processing method provided by the application, the traditional stream data service can be rapidly expanded, and the combination with the existing tools is closer, and the efficiency is higher. At that same time, the application also disclose a stream data processing system, a device and a computer-readable storage medium, which have the beneficial effect mentioned above.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular, to a stream data processing method, system, device, and computer-readable storage medium. Background technique [0002] In the field of data processing, data can be divided into batch data and stream data in a certain way. Compared with the static and bounded batch data stored in a fixed location, stream data has a steady stream of characteristics. The processing process is obviously different from batch data, and the processing is more difficult. [0003] There are a variety of frameworks that can be used to process streaming data, such as the Storm and Samza stream processing frameworks that only process streaming data, and the Flink framework that can not only process streaming data but also small-scale batch data processing. This is because Flink Able to execute at scale, has very high throughput, provides correct results even in the presence of out-of-or...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2455G06F16/242
Inventor 朱彦安范渊龙文洁
Owner HANGZHOU ANHENG INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap