Distributed data processing system and distributed data processing method

Abstract

The embodiment of the invention provides a distributed data processing system and method, and relates to the technical field of data processing, a reference number is sent to a storage node through amanagement node, and the storage node sets its own node maintenance number according to the reference number; when the current data block is stored, the current node maintenance number is stored intoa currently stored data block; based on preset step size value, own node maintenance number is updated; the management node is used for determining a coverage number, sending the coverage number to astorage node; the storage nodes are used for storing the data blocks according to the coverage numbers and the storage numbers of the stored data blocks, deleting a specified data block from the stored data blocks. When the data blocks are stored, the data blocks are stored according to the node maintenance numbers, then the data blocks are deleted according to the numbers, the memory is cleaned.The problem that in the prior art, due to the fact that the time of the management nodes and the time of the storage nodes are not unified, the storage data are deleted mistakenly is solved, and the situation that the storage data in the storage nodes are deleted mistakenly is reduced.

Description

technical field [0001] The present application relates to the technical field of data processing, in particular to a distributed data processing system and a distributed data processing method. Background technique [0002] At present, big data application systems have been popularized and widely used in many fields, generating various real-time streaming big data, such as urban traffic management, network monitoring management, financial transaction management, mobile social network analysis, etc. The object storage system can be used to save a large amount of video data. Considering the storage space and other factors, the previous data needs to be cleared every certain storage cycle to release the storage space so that the latest data can be successfully saved. [0003] In the prior art, the distributed data processing system unifies the time of each storage node according to the computer system time, and the management node issues a delete command to each storage node to...

AI Technical Summary

Problems solved by technology

[0003] In the prior art, the distributed data processing system unifies the time of each storage node according to the computer system time, and the management node issues a delete command to each storage node to delete the data before the specified time, but because the time of the management node and the storage node may not be the same Uniformity will cause data deletion by mistake. For example, the current time is the 10th, and the time of the storage node A is displayed as the 5th, and the time when the storage node A records data is the 5th. If the management node determines to overwrite the data before the 6th, the storage node A will delete all the data stored in storage node A according to the data issued by the management node to cover the data before the 6th

Images