A Dynamic Gradual Backpressure Method for Real-time Stream Computing

A real-time streaming and dynamic technology, applied in the information field, can solve problems such as the uneven back pressure process, affecting the overall delay, and the system is prone to load oscillations

Active Publication Date: 2020-05-05
ZHEJIANG UNIV OF TECH
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] In order to overcome the shortcomings of the existing real-time stream computing method that a single task affects the overall delay, the back pressure process is not smooth, and the system is prone to load oscillations, the present invention proposes a method to reduce the impact of a single task on the overall delay in the topology. A dynamic step-by-step back-pressure method for real-time flow computing that is smooth in the pressure process and does not cause load oscillations in the system. This method adjusts the rate at which data is sent from the upstream to the task node according to its current load.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Dynamic Gradual Backpressure Method for Real-time Stream Computing
  • A Dynamic Gradual Backpressure Method for Real-time Stream Computing
  • A Dynamic Gradual Backpressure Method for Real-time Stream Computing

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] In order to make the above-mentioned features and processes of the present invention more comprehensible, the following specific embodiments are described in detail with reference to the accompanying drawings.

[0031] figure 1 It is a schematic diagram of the topology task (Task) component in the embodiment of the present invention. Such as figure 1 As shown, each current component C (Spout or Bolt) in the topology, after inputting the data flow i, triggers the related operation fv, generates a new data flow t and sends it to the downstream component, and changes the state from S changes to S' (if the state remains unchanged, then S=S'), expressed by the formula is (S', t) = fv(S, i).

[0032] In this embodiment, the message queue for communication between Tasks is implemented by Zookeeper, figure 2 is the Zookeeper message queue in the embodiment of the present invention. Such as figure 2 As shown, the message queue provides a first-in-first-out (FIFO) queue of...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a dynamically stepwise back pressing method for live streaming computation. A data sending speed of upstream is adjusted according to the current load condition of a task node (Task), when the Task is overloaded and thus delay is increased, upstream Bolt reduces the sending speed to downstream, and more resources are used for processing the data in current processing, so as to avoid increase of the delay caused by blockage, and data overtime and retransmission. When the load of the Task is reduced to a minimum threshold, and continues for sensitivity seconds, the Task sends a back pressing canceling signal to the upstream, after receiving the back pressing canceling signal, the upstream Task firstly checks whether in a back pressing state, if yes, the last sending speed is recovered, and only the component is recovered to the initial speed can the back pressing state be eliminated. According to the method provided by the invention, the influence of the single task on the whole delay in topology is reduced, the back pressing process is smooth, and the system is prevented from generating load vibration.

Description

technical field [0001] The invention relates to the field of information technology, in particular to a dynamic step-by-step backpressure method oriented to real-time stream computing. Background technique [0002] From social network information (to provide hot topics or real-time search) to advertising processing data engines, real-time stream computing is widely used in today's industry, such as Apache Storm, Twitter's Heron, Apache Flink, SparkStreaming, Samza, etc. In these systems, the generation of data is completely determined by the data source, and the dynamic change and inconsistency of the data source cause the rate of the data flow to present a bursty characteristic, and the bursty characteristic of the data flow often leads to the occurrence of overload , There are several reasons for overloading: network congestion, high resource utilization, interference, heterogeneity, IO high-frequency blocking, etc. Therefore, in real-time stream computing, overload is co...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): H04L12/803H04L12/24G06F9/52
CPCG06F9/526H04L41/083H04L47/125
Inventor 陆佳炜陈烘马俊肖刚张元鸣高飞李杰卢成炳
Owner ZHEJIANG UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap