Distributed data processing system, distributed data processing method

Abstract

The distributed data processing system and method provided by the embodiments of the present application relate to the technical field of data processing. A management node sends a reference number to a storage node, and the storage node sets its own node maintenance number according to the reference number; when storing a current data block , store the current node maintenance number in the currently stored data block, and then update its own node maintenance number according to the preset step value. The management node is used to determine the coverage number, and sends the coverage number to the storage node. The storage node uses the Overwrite the number and the storage number of the stored data block, delete the specified data block from the stored data block, when storing the data block, store the data block according to the node maintenance number, and then delete the data block according to the number, clear the memory, and solve the problem. The problem of accidental deletion of stored data in the prior art due to inconsistent time between the management node and the storage node is solved, and the accidental deletion of stored data in the storage node is reduced.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular, to a distributed data processing system and a distributed data processing method. Background technique [0002] At present, big data application systems have been popularized and widely used in many fields, producing various real-time streaming big data, such as urban traffic management, network monitoring management, financial transaction management, mobile social network analysis, etc. The object storage system can be used to save a large amount of video data. Considering the storage space and other factors, the previous data needs to be cleaned up every certain storage period to release the storage space, so that the latest data can be successfully saved. [0003] In the prior art, the distributed data processing system unifies the time of each storage node according to the computer system time, and the management node issues a deletion instruction to each ...

AI Technical Summary

Problems solved by technology

[0003] In the prior art, the distributed data processing system unifies the time of each storage node according to the computer system time, and the management node issues a delete command to each storage node to delete the data before the specified time, but because the time of the management node and the storage node may not be the same Uniformity will cause data deletion by mistake. For example, the current time is the 10th, and the time of the storage node A is displayed as the 5th, and the time when the storage node A records data is the 5th. If the management node determines to overwrite the data before the 6th, the storage node A will delete all the data stored in storage node A according to the data issued by the management node to cover the data before the 6th

Images