A metadata request distribution method and device based on congestion perception

Abstract

The embodiment of the invention provides a metadata request distribution method and device based on congestion perception. The method comprises the steps of receiving a metadata request and server state information of all reachable servers, performing performance evaluation on all the reachable servers through a preset cost estimation function according to the server state information, and obtaining the performance cost of each server in all the reachable servers; selecting a server with the lowest performance cost as a target server, judging and knowing that the number of requests sent to thetarget server in the current time window is smaller than a preset threshold value, and sending a metadata request to the target server; And the client equivalently judges the congestion state of thetarget server, and adjusts the sending rate of the metadata request according to the congestion state of the target server. According to the method provided by the invention, the processing speed of the metadata request and the throughput rate of the whole system are improved, the request sending rate is adjusted by sensing the congestion condition of the server, and the maintenance cost and the time cost of the system are saved.

Description

technical field [0001] Embodiments of the present invention relate to the field of video technologies, and in particular, to a congestion-aware-based metadata request distribution method and device. Background technique [0002] In recent years, with the advent of the era of big data, data storage and utilization has gradually become a topic of common concern in many industries. In order to make better use of these real-time and uninterrupted data, it is often necessary to use a distributed file system to organize and manage data on many storage devices. Since the vast majority of requests in a distributed file system are requests for metadata, most file systems often use replicated metadata server clusters to provide metadata services for clients to share the load, while also avoiding a single Node failure. However, since the distributed file system does not provide heuristic guidance and constraints to the client, the client will often cause congestion and performance sh...

AI Technical Summary

Problems solved by technology

The problem with this type of method is that when the load of some servers reaches the upper limit, the metadata requests sent to these servers will be lost. After a long waiting timeout, the client needs to resend these lost requests. During this period, all involved Requests to the same metadata need to be blocked, which ultimately makes the client experience worse, and also brings many difficulties to the maintenance of consistency

At the same time, there are often many servers under light load, but the client does not use the heuristic algorithm to perceive this information, thus causing the unbalanced load on the server

[0004] In the existing technology, metadata distribution optimization is often performed from the server side to avoid excessive load on some servers. However, only server optimization has great limitations, and the efficiency is relatively low. At the same time, certain performance is lost, and additional The overhead does not solve the problem of server-side load fluctuation and performance degradation from the root cause

Images