Method and system for controlling data downbrushing, equipment and medium

Abstract

The invention discloses a method and system for controlling data downbrushing, equipment and a storage medium, and the method comprises the steps: in response to a received write request, carrying out write operation on data packets in sequence, and judging whether the write operation of the current data packet is cache write or not; in response to the fact that the write operation of the current data packet is cache write, writing data of the data packet into a data cache layer, and judging whether the size of the data reaches the size of a distributed object or not; in response to the situation that the size of the data reaches the size of the distributed object, carrying out persistence on the data, and judging whether the number of the current to-be-confirmed batch structures exceeds a threshold value or not; and in response to the condition that the number of the current to-be-confirmed batch structures exceeds a threshold value, performing range detection on the persistent data, and performing downbrushing according to a range detection result. According to the invention, through the mechanisms of batch, asynchronous downbrushing and overall response and the adoption of a whole-strip downbrushing mechanism in the data cache layer, the two-layer cooperation guarantees the rapid persistence and safe response of the data.

Description

technical field [0001] The present invention relates to the field of distributed file systems, and more specifically refers to a method, system, computer equipment and readable medium for controlling data downloading. Background technique [0002] A distributed file system means that the physical storage resources managed by the file system are not necessarily directly connected to the local node, but are connected to the node through a computer network. For the distributed file system (object storage) in the writing process using the HDFS (Hadoop Distributed File System, distributed file system) protocol, how to ensure data security and performance is very important. In a general distributed file system, multi-buffer memory caches are used to ensure data security in the fault domain. However, it is impossible to rely on memory caches to maintain data security in scenarios where multiple nodes fail at the same time, such as systemic power outages. For storage services, bala...

AI Technical Summary

Problems solved by technology

In a general distributed file system, multi-buffer memory caches are used to ensure data security in the fault domain. However, it is impossible to rely on memory caches to maintain data security in scenarios where multiple nodes fail at the same time, such as systemic power outages.

Images