A multi-disk load balancing asynchronous read and write scheduling method and device

Abstract

Multi-disk load balancing asynchronous read-write scheduling method and device, HDFS is configured, so that the data nodes of the HDFS system report disk information to the name node, and start the metadata query service process on the name node; obtain all local disk information, and Number the disks according to the disk type in the disk information; create independent read and write work threads for each disk, so that the disks wait for the upper layer to send read and write tasks; the metadata query process receives the HDFS data requests that the upper layer users want to read, and passes through the name node The metadata query service process obtains the disk number of the HDFS data to be read; when sending a read-write task to the read-write worker thread, it also sends the disk number, and when the read-write worker thread receives the read-write task, it processes each task asynchronously according to the disk information The read and write tasks are notified to the upper layer users after completion. Reduce the performance overhead caused by multiple thread contention and multiple disk contention under high concurrency, and improve data read and write efficiency.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of databases, in particular to a multi-disk load balancing asynchronous read and write scheduling method and device. Background technique [0002] HDFS refers to a distributed file system. HDFS is usually designed as a distributed file system suitable for running on general-purpose hardware. HDFS is a highly fault-tolerant system, which is suitable for deployment on cheap machines, can provide high-throughput data access, and is very suitable for applications on large-scale data sets. HDFS is a master-slave structure. An HDFS cluster consists of a name node. The name node is a master server that manages the file namespace and regulates client access to files. There are also some data nodes, usually a node and a machine. The data nodes come from Manage the storage of the corresponding node. HDFS opens the file namespace to the outside world and allows user data to be stored in the ...

AI Technical Summary

Problems solved by technology

[0003] At present, under the HDFS system, the data nodes store the actual data on multiple disks, including ordinary mechanical disks and SSD solid state disks. During the execution of the database, the temporary data also needs to be saved on these disks of the data nodes. Most databases are divided into The local data stored in HDFS is directly read and written through the HDFS interface, and the database also reads and writes local temporary files through the interface of the local file system, resulting in synchronous reading and writing of multiple different types of disks under concurrent multiple requests, and is The reads and writes of different types of disks are mixed with each other, resulting in the performance overhead caused by multiple threads scrambling and multiple disks scrambling under high concurrency

At this stage, there is no asynchronous read and write technical solution for load balancing multiple disks under the HDFS system

Images