Distributed flow processing system fault tolerance method, nodes and system

A stream processing system and distributed technology, applied in the computer field, can solve the problems of low data processing efficiency, affecting data processing efficiency, and low data processing efficiency of the whole network.

Active Publication Date: 2014-09-10
HUAWEI TECH CO LTD
View PDF4 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] For the method (1) above, when the stream processing network of the system is rebuilt, the system needs to roll back the entire network, and the source node resends the data, and each other working node receives and processes the source node or upstream working node again The resent data causes the data processing efficiency of the entire network to be dragged down, which affects the data processing efficiency and wastes node resources
For the above method (2), each working node needs to back up the processed data, and the storage overhead is large
In addition, due to the frequent interaction between upstream worker nodes and downstream worker nodes, the data processing efficiency is low

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed flow processing system fault tolerance method, nodes and system
  • Distributed flow processing system fault tolerance method, nodes and system
  • Distributed flow processing system fault tolerance method, nodes and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0159] In combination with the foregoing, embodiments of the present invention provide a fault-tolerant method for a distributed stream processing system. Take the perspective of executing this method on a worker node other than the source node in a distributed stream processing system as an example, see Figure 4 , the method flow provided by the embodiment of the present invention includes:

[0160] 401: Obtain the node information of the failed node in the distributed stream processing system, and receive the retransmission data sent by the upstream node. The retransmission data is initiated by the source node, and the upstream node is the source node or the upstream working node;

[0161] 402: Determine whether it is related to the failed node according to the node information of the failed node;

[0162] 403: If it is related to the failure node, process the received retransmission data and send it to the downstream node.

[0163] Preferably, the node information of the...

Embodiment 2

[0220] combine figure 1 The schematic diagram of the implementation environment structure shown, figure 2 Referring to the schematic structural diagram of the distributed stream processing system and the content of the first embodiment above, the embodiment of the present invention provides a fault-tolerant method for the distributed stream processing system. For the convenience of description, in the embodiment of the present invention, after a working node in the distributed stream processing system fails, when the source node sends retransmission data, the incoming data is grouped and numbered according to the time interval or data size of the received data, to obtain Each data corresponds to a group identifier, and the group identifier is added to the signal packet and sent to the downstream node, so that the downstream node stores the state information according to the group identifier carried in the signal packet after receiving the signal packet. As an example, this em...

Embodiment 3

[0260] to combine figure 1 Schematic diagram of the structure of the implementation environment shown, figure 2 Referring to the schematic structural diagram of the distributed stream processing system and the contents of the first and second embodiments above, the embodiment of the present invention provides a fault-tolerant method for a distributed stream processing system. For the sake of illustration, in the embodiment of the present invention, after a working node in the distributed stream processing system fails, when the source node sends the retransmission data, the retransmission data is grouped and numbered and added to the retransmission data, so that the downstream node After receiving the data carrying the group number, the method provided in this embodiment may be explained in detail by taking the state information storage according to the group number carried in the data as an example. see Figure 7 , the method flow provided by the embodiment of the present ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a distributed flow processing system fault tolerance method, nodes and a system, and belongs to the technical field of computers. The method comprises that node information of a failure node is acquired, retransmission data transmitted by an upstream node are received, and the retransmission data are initially transmitted by a source node; correlation with the failure node is confirmed according to node information of the failure node; and if correlation with the failure node is confirmed, the received retransmission data are processed and transmitted to a downstream node. The retransmission data transmitted by the upstream node are received, correlation with the failure node is confirmed and the received retransmission data are processed so that the retransmission data transmitted by the upstream node are processed only when correlation with the failure node is confirmed, data processing efficiency of other node is not influenced, large amount of backup data do not need to be locally stored, and thus local resource is saved. Besides, frequent interaction with other upstream or downstream nodes is not needed so that data processing efficiency is enhanced.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a fault-tolerant method, node and system of a distributed stream processing system. Background technique [0002] With the continuous expansion of distributed computing technology in various application fields, distributed stream processing systems are widely used in many fields, such as financial management, network monitoring, communication data management, WEB applications, sensor network data processing, etc. A distributed stream processing system refers to an Internet software system that uses a distributed system to process data stream services, and the fault tolerance of a distributed stream processing system refers to the ability to provide correct services to the external environment in the event of internal failures in the system. The fault tolerance method of the distributed stream processing system is the main means to enhance the reliability and availability of the...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L12/24H04L29/08H04L45/28H04L45/24H04L45/42
CPCH04L43/10H04L41/0654H04L45/22H04L45/28H04L45/42H04L41/0677H04L43/091H04L41/069
Inventor 潘璐伽何诚
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap