Distributed data stream processing method and device

A processing method and technology of data flow, applied in the field of data processing, can solve problems such as inability to load balance processing of data flow

Active Publication Date: 2015-04-29
HUAWEI TECH CO LTD
View PDF3 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] Embodiments of the present invention provide a distributed data stream processing method and device to solve the problem that distributed stream processing technology cannot perform load balancing processing on data streams with the same key value

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data stream processing method and device
  • Distributed data stream processing method and device
  • Distributed data stream processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0072] The embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0073] Such as figure 1 As shown, it is a flowchart of a method for performing distributed data stream processing in Embodiment 1 of the present invention, including the following steps:

[0074] S101: The first node determines that the second node is the next-hop working node for processing the data flow corresponding to the key value according to the key value of the data stream assigned to the next-hop working node as needed;

[0075] S102: After the first node determines that the second node satisfies the set load migration condition, it migrates the data flow that needs to be allocated to the next-hop working node from the second node to the third node for processing , and instruct the second node and the third node to synchronize the state information of the data flow corresponding to the key value;

[0076] Wherein, the second node...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention relates to the field of data processing, and particularly to a distributed method and device for processing a data stream, which are used for solving the problem that the distributed stream processing technology cannot conduct load balancing processing on data streams having the same key value. The method in the embodiments of the present invention comprises: according to a key value of a data stream which needs to be allocated to a next-hop work node, a first node determining a second node as the next-hop work node for processing a data stream corresponding to the key value, and after it is determined that the second node satisfies a set load migration condition, migrating the data stream, which needs to be allocated to the next-hop work node, to a third node for processing from the second node, and indicating that the second node and the third node are synchronizing state information about the data stream corresponding to the key value. By means of the method, after it is determined that a downstream second node satisfies a set load migration condition, a first node can migrate a data stream corresponding to the key value, which is mainly processed by the second node, to a third node for processing with a small accumulated load at present, so that load balancing processing can be conducted on data streams with the same key value.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a distributed data flow processing method and device. Background technique [0002] Data stream processing technology is widely used in many fields, such as financial management, network monitoring, communication data management, Web applications, sensor network data processing, etc. There is a typical feature in these applications: the data flow processing volume is extremely high, and it has a very high burstiness. When the data arrival speed exceeds the processing capacity of the system, the system will be overloaded and the performance will drop. Therefore, load management has become the focus and focus of research in data stream processing. [0003] Data stream processing technology includes centralized data stream processing technology and distributed data stream processing technology; in centralized data stream processing technology, when the system is detected to be overlo...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/10H04L67/1001H04L47/10
Inventor 何诚李柏晴黄群刘勤
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap