Fingerprint-based state detection method and device for distributed processing system
A distributed processing and fingerprint technology, applied in transmission systems, digital transmission systems, data exchange networks, etc., can solve problems such as unfriendly network cards, consumption of memory and CPU resources, and large delays.
Active Publication Date: 2020-07-17
EMC IP HLDG CO LLC
View PDF4 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0031] In addition, additional memory and CPU resources are required to send each report traffic, so more interrupts and dense small packets are not friendly to the network card
[0032] In addition, each node in the path from upstream to downstream incurs a small amount of delay, especially due to memory copying and synchronous / asynchronous processing from the application data processing layer to the NIC buffer, etc.
With tens of thousands of nodes running, the cumulative latency can be quite large, since end-to-end latency is on the order of milliseconds (ms) for real-time streaming systems
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0064] The principles of the disclosure will be described below with reference to several example embodiments illustrated in the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the drawings, it should be understood that these embodiments are only described to enable those skilled in the art to better understand and implement the present disclosure, but not to limit the scope of the present disclosure in any way.
[0065] image 3 A flowchart of an example of a fingerprint-based state detection method 300 in a distributed processing system according to an embodiment of the present disclosure is shown.
[0066] Such as image 3 As shown, method 300 includes:
[0067] Step S301: At the root node, generate and send an initial fingerprint based on the output message identifier to be transmitted;
[0068] Wherein, step S301 may also include sending the RootMsg identifier R to the task tracking point as the initial state value, and ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
Embodiments of the present disclosure provide a fingerprint-based state detection method and device in a distributed processing system. The method includes: at the root node, generating and sending an initial fingerprint based on an output message identifier to be transmitted; at an intermediate node, generating and sending an initial fingerprint based on the received input message identifier, the output message identifier to be transmitted, and the received fingerprint sending the updated fingerprint; at the leaf node, sending the final updated fingerprint to a task tracking point; and at the task tracking point, obtaining state information based on the final updated fingerprint. Since this method does not generate traffic from each node, but usually only generates tracking traffic from leaf nodes, the method can greatly reduce the tracking overhead while ensuring the reliability of data processing.
Description
technical field [0001] Embodiments of the present disclosure relate to the field of distributed data processing, and more specifically relate to a fingerprint-based state detection method and device used in a distributed processing system. Background technique [0002] Distributed data processing systems, especially real-time streaming systems, are receiving widespread attention. Modern real-time streaming systems such as Pivotal SpringXD, Storm, Spark Streaming, Samza are being widely used in e-commerce, ETL, big data analysis, etc. It is important to provide a processing capability that is reliable so as to ensure successful processing of each data item (or message) even in the event of any failure at the node or in the network. For such distributed systems, especially for large systems with tens of thousands of nodes or intermediate connections, the key challenge is how to detect failures in an efficient manner, at the lowest cost and with minimal impact on performance ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): H04L1/00H04L12/26H04L29/08
CPCH04L1/0063H04L43/0823H04L67/10H04L43/0817H04L43/0876H04L43/10
Inventor 赵军平张峰豪曹逾陶隽孙宇熙
Owner EMC IP HLDG CO LLC