An Efficient Erasure Correction Distributed Storage Writing Method, System, Medium and Terminal

Abstract

The present invention provides an efficient erasure correcting distributed storage writing method, system, medium and terminal, including: sending the first request information to the metadata server through the client, and the metadata server creates The metadata corresponding to the data block, and feed back the corresponding block ID information to the client; the client initiates the second request information to the data node to request to open the data block operation handle of the data node according to the block ID information; according to the data The block operation handle writes the data to be stored into the data node, and closes the written data block; the present invention can ensure the maximum utilization rate of the socket connection by establishing a thread for managing the socket connection pool during initialization, and reduce the number of threads and the number of threads on the server side. The number of connections reduces thread context switching, improves CPU utilization, reduces network pressure, reduces system overhead, and improves service throughput.

Description

technical field [0001] The present invention relates to the field of computers, in particular to an efficient erasure correction distributed storage writing method, system, medium and terminal. Background technique [0002] In the big data environment, the volume of metadata is also very large, and the access performance of metadata is the key to the performance of the entire distributed file system. The distributed metadata management architecture distributes metadata on multiple nodes. It solves the performance bottleneck of the metadata server and other problems. [0003] However, in the traditional IO model, when the client writes IPC video files, one way is to use the mode of one thread and one socket connection, which will cause too many concurrent connections and frequent thread context switching, resulting in low CPU utilization; The other is to use x threads to process y links (x<y) to reduce the number of threads on the server side. However, when the write conc...

AI Technical Summary

Problems solved by technology

[0003] However, in the traditional IO model, when the client writes IPC video files, one way is to use the mode of one thread and one socket connection, which will cause too many concurrent connections and frequent thread context switching, resulting in low CPU utilization; The other is to use x threads to process y links (x<y) to reduce the number of threads on the server side. However, when the write concurrency is high, the fewer x threads cannot process the write business of y links in time. , causing video files to be lost during transmission,

In addition, the server generally uses seek to write video files, and multi-threaded writing to the same disk also requires lock protection, which will also bring a very large system overhead.

Images