Unlock instant, AI-driven research and patent intelligence for your innovation.

A real-time data processing method and device

A real-time data processing and data technology, applied in the field of data processing, can solve the problems of large space and time overhead, no obvious distinction between rolling windows and sliding windows, and poor aggregation performance of huge sliding windows, so as to improve efficiency and aggregation ability strong effect

Active Publication Date: 2021-09-03
北京长亭未来科技有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Apache Flink is a stream processing engine with the following features: support task management, allow delivery, stop statistical tasks, real stream processing, data does not need to be placed on the disk, pure memory computing, real-time at the second or millisecond level, and support window-based Statistics, supports the use of SQL syntax to customize statistical logic, and supports cluster deployment. Flink has become the most mature real-time stream processing solution in the open source community; but Flink is not without disadvantages: Flink has poor aggregation performance for huge sliding windows, and Flink does not There is no obvious distinction between rolling windows and sliding windows, and its aggregation operation is independent for each window. Suppose a statistical task needs to calculate the total number of all elements in 1 hour every 1 second, then Flink needs to maintain 3600 windows at the same time , and calculate the number of elements in the oldest window every second, the resulting space and time overhead is very large

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A real-time data processing method and device
  • A real-time data processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0016] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiment of the application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the application. Obviously, the described embodiment is only It is an embodiment of a part of the application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0017] It should be noted that the terms "first" and "second" in the specification and claims of the present application and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the present invention discloses a real-time data processing method and device. The method includes: dividing the first data packet into N groups of sub-data packets according to preset rules; the aggregation result of the first data group; calculate the aggregation result of the second data group according to the aggregation result of the first data group and the aggregation result of each sub-data group. The technical solution disclosed by the invention can effectively solve the problem of poor aggregation performance of Flink under a huge sliding window.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a real-time data processing method and device. Background technique [0002] With the vigorous development of the Internet, enterprises have higher and higher requirements for real-time business, the concept of stream processing has emerged as the times require, and the open source community has successively provided corresponding solutions. [0003] Apache Flink is a stream processing engine with the following features: support task management, allow delivery, stop statistical tasks, real stream processing, data does not need to be placed on the disk, pure memory computing, real-time at the second or millisecond level, and support window-based Statistics, supports the use of SQL syntax to customize statistical logic, and supports cluster deployment. Flink has become the most mature real-time stream processing solution in the open source community; but Flink is not withou...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/2453G06F16/2455
CPCG06F16/24534G06F16/2455
Inventor 张盛宇刘超刘金钊姚战伟李昌志张嘉欢
Owner 北京长亭未来科技有限公司