Distributed system data writing method and device

Abstract

The embodiment of the invention discloses a distributed system data writing method and device which are used for reducing the burden of data writing operation and the network overhead of verification so as to improve the overall performance of a distributed system. The method comprises the steps that after at least two data writing requests including a first data writing request and a second data writing request are received, a temporary partition is determined according to first to-be-written data and a first attribution partition carried by the first data writing request and second to-be-written data and a second attribution partition carried by the second data writing request; and the method also includes writing the first to-be-written data and the second to-be-written data into temporary partitions, and marking the temporary partition and the first attribution partition corresponding to the first to-be-written data written into the temporary partitions and the temporary partition and the second attribution partition corresponding to the second to-be-written data written into the temporary partitions; further, according to the temporary marks, writing the first to-be-written data back to the first attribution partition, and writing the second to-be-written data back to the second attribution partition.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a method and device for writing data in a distributed system. Background technique [0002] In the current era of information and data storm growth, in order to improve the data processing rate, more and more distributed systems are used for data storage to provide distributed storage services for massive data and high-speed read and write access capabilities. [0003] A distributed system consists of a client and several data storage nodes connected to each other. Among them, the data storage node is responsible for data storage, and each object storage device (OSD) module is deployed on a data storage node, and a fixed number of partitions are usually set in a distributed system, for example, 1024 partitions, and each partition belongs to one or more different data storage nodes. [0004] At present, the existing distributed system data writing method is...

AI Technical Summary

Problems solved by technology

However, in this data writing scheme, in the case of a large number of partitions, it is difficult to fill up the stripes of each partition and then write them into the data storage nodes to which each partition belongs, which will cause waste of storage space. , and increases the burden of data writing operations and the network overhead of verification, reducing the overall performance of the distributed system

Images