Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method and system based on stream processing framework

A data processing and stream processing technology, applied in the computer field, can solve the problems that the stream processing framework consumes a lot of time and resources, and cannot guarantee the real-time performance of data processing.

Active Publication Date: 2022-06-24
HUNDSUN TECH
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

As a mainstream streaming computing framework, Flink also provides rich window aggregation functions, which can realize monitoring tasks in most business scenarios, such as counting the maximum CPU usage of a specified host within 5 minutes; but in some For changes in monitoring tasks in business scenarios, it is necessary to readjust the task logic under the Flink architecture, resulting in the stream processing framework consuming a lot of time and resources, and unable to guarantee the real-time performance of data processing for monitoring tasks

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and system based on stream processing framework
  • Data processing method and system based on stream processing framework
  • Data processing method and system based on stream processing framework

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] In the following description, numerous specific details are set forth in order to provide a thorough understanding of this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can make similar promotions without departing from the connotation of this specification, so this specification is not limited by the specific implementation disclosed below.

[0034] The terminology used in one or more embodiments of this specification is for the purpose of describing a particular embodiment only and is not intended to limit the one or more embodiments of this specification. As used in the specification or embodiments and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and / or" as used in this specification in one or more embodiments re...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a data processing method and device based on a stream processing framework. The data processing method based on the stream processing framework comprises the steps that task data streams of the stream processing framework are received; a monitoring task flow corresponding to the task data flow is determined, and the monitoring task flow carries at least one monitoring task factor; and determining the target data according to the parameter information in the task data flow based on the at least one monitoring task factor, so that monitoring processing can be performed on the task data flow according to the monitoring task flow only by determining the monitoring task flow corresponding to the task data flow in real time on the premise of normal processing of the task data flow. The stream processing framework does not need to consume a large amount of time and resources, and the real-time performance of monitoring task processing of the task data stream can be guaranteed.

Description

technical field [0001] The embodiments of the present specification relate to the field of computer technologies, and in particular, to a data processing method based on a stream processing framework. Background technique [0002] Flink is an emerging framework in the field of big data that uses stream processing to simulate batch processing. One of the usage scenarios is to build real-time data channels, move and transform data between different storages, and can also be extended to real-time monitoring, real-time Application scenarios such as reports and real-time warehouses. As a mainstream streaming computing framework, Flink also provides rich window aggregation functions, which can implement monitoring tasks in most business scenarios, such as counting the maximum CPU usage of a specified host within 5 minutes; but in some cases For changes in monitoring tasks in business scenarios, the task logic under the Flink architecture needs to be readjusted, resulting in a str...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/30G06F9/445G06F16/22G06F16/2455
CPCG06F11/3006G06F11/3051G06F9/4451G06F16/2282G06F16/24556
Inventor 李森秦元张绍震钱山刘芃
Owner HUNDSUN TECH