A distributed storage QoS control method and system

Abstract

The application provides a distributed storage QoS control method and system. The QoS control method comprises: setting a speed limit parameter for a target file storage directory through a storage management interface of the distributed storage system, and writing the speed limit parameter into an ECTD cluster; a NAS storage node reads the speed limit parameter and transmits the speed limit parameter into a corresponding type of shared service module; and the shared service module performs corresponding speed limit processing on the target file storage directory. The QoS control system comprises a processor and a memory, and the processor realizes the QoS control method when executing a computer program stored in the memory. According to the application, the problems that the existing distributed storage QoS control strategy cannot perform individual speed limiting on a file storage directory and is prone to system speed limiting failure due to a single storage node downtime can be solved.

Description

Technical Field

[0001] This invention belongs to the field of distributed storage technology, and more specifically, relates to a distributed storage QoS control method and system. Background Technology

[0002] QoS stands for Quality of Service. Originating from network resource contention technology, it is used to solve network congestion or performance bottlenecks, resolve client-side contention for services and resources, improve service quality, allocate higher network bandwidth or IOPS for high-priority services, meet the resource needs of different users, optimize resource allocation schemes, and prevent users from contention for resources. QoS can help administrators better allocate storage resources, enabling various application clients to fully and stably utilize backend storage resources.

[0003] Existing distributed storage QoS control strategies are primarily based on iptables rules. These strategies use hashlimit to rate-limit client IP addresses and bandwidth. However, this approach only limits the bandwidth of specific clients and cannot satisfy the requirement for rate limiting of file directories. Furthermore, if one of the storage nodes in the backend distributed cluster fails during the execution of this QoS control strategy, QoS rate-limiting parameters may be lost, leading to rate-limiting failure. Summary of the Invention

[0004] The purpose of this invention is to solve the problems that existing distributed storage QoS control strategies cannot individually limit the rate of file storage directories and are prone to system rate limiting failure due to the failure of a single storage node.

[0005] To achieve the above objectives, the present invention provides a distributed storage QoS control method and system.

[0006] According to a first aspect of the present invention, a distributed storage QoS control method is provided, the QoS control method being applied to a distributed storage system, comprising the following steps:

[0007] Set rate limiting parameters for the target file storage directory through the storage management interface of the distributed storage system, and write the rate limiting parameters into the ECTD cluster;

[0008] The NAS storage node reads the rate limiting parameter and transmits the rate limiting parameter to the corresponding type of shared service module;

[0009] The shared service module performs corresponding rate limiting on the target file storage directory.

[0010] Optionally, setting rate limiting parameters for the target file storage directory through the storage management interface of the distributed storage system includes:

[0011] The storage management interface of the distributed storage system pools all disks on the storage server and creates a file storage directory list containing the target file storage directory based on the formed storage pool.

[0012] Optionally, the rate limiting parameters include the name of the target file storage directory, the bandwidth rate limiting value, and the sharing service module type. The bandwidth rate limiting value is a global bandwidth rate limiting value or a specified client IP bandwidth rate limiting value. The sharing service module type includes at least one of NFS sharing service module, FTP sharing service module, and SMB sharing service module.

[0013] Optionally, the NAS storage node may pass the rate limiting parameters to the corresponding shared service module, including:

[0014] The type of the shared service module is obtained based on the speed limit parameters;

[0015] The speed limit parameters are passed to the configuration file of the corresponding type of shared service module.

[0016] Optionally, the shared service module may perform rate limiting on the target file storage directory, including:

[0017] Read the speed limit parameters from its own configuration file;

[0018] Modify its own code to implement the predetermined dmClock-based distributed rate limiting strategy;

[0019] When a client requests access to the target file storage directory, the system applies global bandwidth limiting or specified client IP bandwidth limiting to the target file storage directory based on the dmClock distributed rate limiting strategy, and feeds back the rate limiting result to the client.

[0020] According to a second aspect of the present invention, a distributed storage QoS control system is provided, the QoS control system comprising a processor and a memory, wherein the processor executes a computer program stored in the memory to implement any of the above-described distributed storage QoS control methods.

[0021] The beneficial effects of this invention are as follows:

[0022] The distributed storage QoS control method of the present invention first sets rate limiting parameters for the target file storage directory through the storage management interface of the distributed storage system and writes the rate limiting parameters into the ECTD cluster; secondly, the NAS storage node reads the rate limiting parameters and transmits the rate limiting parameters to the corresponding type of shared service module; finally, the shared service module performs corresponding rate limiting processing on the target file storage directory.

[0023] The distributed storage QoS control method of this invention not only provides a graphical user interface for easily setting rate limiting parameters for file storage directories, but also implements QoS control for commonly used file sharing services through a shared service module. Furthermore, it addresses the single point of failure issue in the event of a distributed storage node failure based on an ETCD cluster, ensuring that the transmission of overall rate limiting parameters is not affected when a storage node in the distributed storage system fails. Therefore, the distributed storage QoS control method of this invention effectively solves the problems of existing distributed storage QoS control strategies being unable to individually rate limit file storage directories and being prone to system rate limiting failure due to the failure of a single storage node.

[0024] The distributed storage QoS control system of the present invention and the above-mentioned distributed storage QoS control method belong to the same general inventive concept, and have at least the same beneficial effects as the above-mentioned distributed storage QoS control method, the beneficial effects of which will not be elaborated here.

[0025] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0026] The present invention can be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which the same or similar reference numerals are used throughout the drawings to denote the same or similar parts.

[0027] Figure 1 A flowchart illustrating the implementation of a distributed storage QoS control method according to an embodiment of the present invention is shown.

[0028] Figure 2 A flowchart illustrating the implementation of setting speed limit parameters and writing them to the ETCD cluster via an interface according to an embodiment of the present invention is shown.

[0029] Figure 3 A flowchart illustrating the implementation of a NAS storage node reading rate limiting parameters and transmitting them to a shared service module according to an embodiment of the present invention is shown.

[0030] Figure 4 A flowchart illustrating the implementation of a shared service module that handles rate limiting and provides feedback to the client according to an embodiment of the present invention is shown. Detailed Implementation

[0031] To enable those skilled in the art to more fully understand the technical solutions of the present invention, exemplary embodiments of the present invention will be described more comprehensively and in detail below with reference to the accompanying drawings. Obviously, the one or more embodiments of the present invention described below are merely one or more specific ways to implement the technical solutions of the present invention, and are not exhaustive. It should be understood that other ways belonging to a general inventive concept can be used to implement the technical solutions of the present invention, and should not be limited to the embodiments described exemplary. Based on one or more embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] Example: Traditional file storage rate limiting methods can only limit the bandwidth accessed by the client and cannot individually limit the rate of multiple file directories. Although NAS storage provides a rate limiting method for block storage, it has never supported a QoS solution for multiple shared modules in file storage. Traditional QoS combined with distributed storage may cause the loss of QoS rate limiting parameters and result in rate limiting failure if one of the storage nodes in the backend distributed cluster fails.

[0033] To address the aforementioned issues, this invention proposes a distributed storage QoS control method. This method utilizes clustered active-active technology and borrows the dmClock algorithm on the storage service server to achieve rate limiting for file storage directories. Clients read and write storage resources via NFS, iSCSI, CIFS, and FTP protocols. Distributed file storage systems are typically used after being mounted by clients. A single distributed file storage system serves multiple clients, and user services are distributed across these clients, making QoS control on the client side impossible. Using an ETCD cluster on the distributed storage server to control the reading of QoS parameters and handle rate limiting of file storage directories based on the correspondence between NAS file directories and shared service modules is a crucial approach to solving distributed storage QoS. For file storage, QoS aims to rationally allocate limited I / O resources, achieving on-demand resource allocation so that each NAS storage subdirectory can utilize resources appropriately. Therefore, rate limiting of file directories in distributed storage primarily involves modifying the shared service module on the distributed server, leveraging the dmClock token bucket concept to achieve rate limiting for specified file directories.

[0034] Figure 1 A flowchart illustrating the implementation of the distributed storage QoS control method according to an embodiment of the present invention is shown. (Refer to...) Figure 1 The distributed storage QoS control method of this invention, applied to a distributed storage system, specifically includes the following steps:

[0035] Step S100: Set rate limiting parameters for the target file storage directory through the storage management interface of the distributed storage system, and write the rate limiting parameters into the ECTD cluster;

[0036] Step S200: The NAS storage node reads the rate limiting parameter and transmits the rate limiting parameter to the corresponding type of shared service module;

[0037] Step S300: The shared service module performs corresponding rate limiting on the target file storage directory.

[0038] In step S100, the rate limiting parameters include the name of the target file storage directory, the bandwidth rate limiting value, and the sharing service module type. The bandwidth rate limiting value is a global bandwidth rate limiting value or a specified client IP bandwidth rate limiting value. The sharing service module type includes at least one of NFS sharing service module, FTP sharing service module, and SMB sharing service module.

[0039] In this embodiment of the invention, the speed limit parameter setting and speed limit processing of the file storage directory of the NAS distributed storage system can be completed through steps S100 to S300. The purpose of speed limiting the file storage directory is to prevent a single file storage directory from occupying too many resources, which would affect the performance of other file storage directories. By limiting the speed, the resource contention problem can be solved, resources can be allocated reasonably, the needs of different levels of business can be met, and better service quality can be provided for applications using the storage system in specific storage fields.

[0040] This invention sets different rate-limiting parameters for different NAS file storage directories through a storage management interface, including global rate limiting and rate limiting based on client IP. These rate-limiting parameters are then stored in the ETCD cluster. The NAS storage nodes read these parameters and pass them to the corresponding shared service module of the NAS distributed storage system. The shared service module accepts client access requests using the C library provided by librados, modifies its source code, reads the mapping relationship between file storage directories and read / write bandwidth from the newly added configuration file, and saves it to the token pool of the mount point. It limits the number of data packets processed per second by the NFS Ganesha, miniFTP, and SMB shared service modules. When the total number of data packets is less than the number of tokens in the token pool, the data packets are allowed to pass. When the total number of data packets is greater than the number of tokens in the token pool, the data packets are rate-limited, and data transmission is prohibited for a period of time to replenish the tokens, thereby limiting the read / write bandwidth of NAS storage subdirectories.

[0041] Furthermore, Figure 2 This document illustrates a flowchart of the implementation of setting speed limit parameters and writing them to the ETCD cluster via an interface, according to an embodiment of the present invention. (Refer to...) Figure 2In this embodiment of the invention, step S100 includes:

[0042] Step S101: Pool all disks on the storage server through the storage management interface of the distributed storage system, and create a file storage directory list containing the target file storage directory based on the formed storage pool.

[0043] Step S102: Set global bandwidth or specify client IP rate limit for the target file storage directory;

[0044] Step S103: Write the rate limiting parameters into the ETCD cluster.

[0045] This invention supports two rate-limiting modes: global rate limiting for specific file storage directories and client-side rate limiting based on client IP addresses. First, all disks on the storage server are pooled through the storage management interface of the distributed storage system. Then, a list of file storage directories is created based on the storage pool. Finally, a specific file storage directory is selected for QoS rate limiting settings. The interface allows users to enter bandwidth limits or specify client-side rate limiting, including the client's IP address. Multiple IP addresses can be entered, and rate limiting is prioritized based on the specified IP address.

[0046] Furthermore, Figure 3 This diagram illustrates the implementation flowchart of a NAS storage node reading rate-limiting parameters and transmitting them to the shared service module, according to an embodiment of the present invention. (Refer to...) Figure 3 In this embodiment of the invention, step S200 includes:

[0047] Step S201: The NAS storage node reads the ETCD cluster rate limiting parameters;

[0048] Step S202: According to the type of shared service module, the configuration file of the NFS shared service module, FTP shared service module and / or SMB shared service module will be passed to it.

[0049] In this embodiment of the invention, high availability needs to be considered in practical applications of storage clusters. Shutting down one storage server within the cluster does not affect the continuity of the entire service. However, if the rate limiting parameter for the file storage directory is set on a single storage node, the QoS of the file storage directory may fail if that storage node goes down. To solve this problem, this embodiment of the invention uses an ETCD cluster approach. An ETCD database cluster is configured, and even if only one storage node in the cluster is working, the rate limiting setting for the file storage directory is not affected. The NAS storage node reads the rate limiting parameter from the ETCD database using jetcd-core.jar and then passes the retrieved parameter value to the configuration file corresponding to the shared service module.

[0050] In storage systems, the underlying storage Bluestore interface can be used to rate-limit OSD file storage. This mainly involves two steps: first, designing a rate-limiting algorithm such as dmclock and successfully adding the code interface to the Bluestore transaction processing interface functions Do_TRANSACTIONS and DEAL_WITH_TRANSACTIONS; second, setting parameters in the token bucket algorithm (bucket depth, token generation speed, etc.) to limit the transmission speed. However, this embodiment of the invention uses a method of first saving the ETCD cluster rate limit and directory mapping relationship to the configuration files of each shared service module, and then modifying the shared service module code to limit the transmission speed. The purpose of this is to allow rate limiting for different shared services. The input interface function public String qosConfig(int switch, String dirName, String[]IPS, Stringbandwith, String type) indicates whether to limit NFS, FTP, and SMB modules, the directory name for rate limiting, the client IP address, bandwidth, and the type of shared service module. The qosConfig rate-limiting interface function writes the file directory and rate-limiting parameter mapping relationship into the configuration files of their respective modules.

[0051] Furthermore, Figure 4 This document illustrates a flowchart illustrating the implementation of the shared service module's handling of rate limiting and feedback to the client, according to an embodiment of the present invention. (Refer to...) Figure 4 In this embodiment of the invention, step S300 includes:

[0052] Step S301: The shared service module reads the rate limiting parameters from the configuration file;

[0053] Step S302: The NFS, FTP, and SMB sharing service modules process the rate limiting and send the rate limiting results back to the client.

[0054] In this embodiment of the invention, the shared service module reads the rate limiting parameters in its own configuration file; modifies its own code to implement a predetermined dmClock-based distributed rate limiting strategy; when it receives a client's access request for the target file storage directory, it performs global bandwidth rate limiting or specified client IP bandwidth rate limiting on the target file storage directory based on the dmClock distributed rate limiting strategy, and feeds back the rate limiting result to the client.

[0055] In this embodiment of the invention, the shared service module program stores user data in a reliable distributed storage cluster via libcephfs. The file system library (libcephfs) runs on top of the RADOS library (librados). Rate limiting is achieved by modifying the source code of the third-party open-source shared service program. A token bucket approach is used to modify each shared service module, adding tokens to the bucket at a specific rate. Data packets are first classified according to preset matching rules. Data packets that do not meet the matching rules do not need to be processed by the token bucket and are sent directly. Data packets that meet the matching rules need to be processed by the token bucket. When there are enough tokens in the bucket, the message can continue to be sent, and the number of tokens in the bucket is reduced accordingly based on the length of the data packet. When there are insufficient tokens in the bucket, the data packet cannot be sent until new tokens are generated in the bucket. This limits the message traffic to a rate less than or equal to the rate at which tokens are generated, thus achieving bandwidth limiting.

[0056] The FTP sharing service module's rate limiting interface includes the get_limit_by_ip(session_t*sess) interface for reading rate limiting parameters. This mainly involves obtaining the rate limiting parameters of the file directory from the config file corresponding to the exported module, and then processing the rate limiting value using get_limit_by_ip(session_t*sess).

[0057] Each shared service module modifies its source code to implement rate limiting, then sends the data to a request for processing. This achieves rate limiting for the file storage directory, addressing issues like network latency and congestion. It provides better service capabilities for rate limiting of specified file storage directories. By modifying the shared service module's source code, it reads the mapping relationship between file directories and read / write bandwidth from the newly added configuration file and stores it in the mount point's token pool. The total size of data packets processed by the shared service module per second is limited to be less than the number of tokens in the token pool, thus implementing read / write bandwidth limiting at the subdirectory level for file storage. This prevents a single subdirectory from consuming excessive resources and impacting the performance of other subdirectories.

[0058] The client communicates with the storage system server through the Socket network communication mechanism. When the server limits the total rate of sending or receiving all messages on a storage interface, it classifies and redistributes the messages according to a policy, setting different traffic levels for data packets. This allows directories or clients without rate limits to use storage resources first, avoiding packet loss or transmission failures caused by network congestion. In the event of network congestion, high-priority data packets can pass through the file storage gateway first, achieving the goal of limiting the client's use of storage resources.

[0059] The distributed storage QoS control method of this invention sets the bandwidth of the file storage directory or fills in the client IP through a web interface, saves the rate limiting parameters to the ETCD cluster, and then reads the ETCD cluster rate limiting parameters through NFS, Samba, and FTP sharing service modules. By modifying the source code of each sharing module based on the dmClock token bucket concept, the method controls the total size of data packets processed per second to be less than the number of tokens in the token pool. This achieves read and write bandwidth limiting at the NAS storage file directory level, prevents a single subdirectory from consuming too many resources, avoids storage resource shortages caused by client access, ensures the quality of critical business services, improves the security and reliability of IO access bandwidth, and allocates limited resources more rationally.

[0060] The distributed storage QoS control method of this invention also controls the QoS of three commonly used file sharing services—NFS, SMB, and FTP—through a shared service module. Furthermore, it employs an ETCD cluster to prevent single-point-of-failure issues in the event of a distributed storage node failure. Borrowing the concept of the dmClock token bucket, when a client's access speed to storage resources exceeds a set bandwidth, its transmission is limited to that bandwidth. When setting the rate-limiting parameters, a global bandwidth can be set based on a specific file directory, or the bandwidth accessed by different clients can be limited. QoS parameters are stored in the ETCD cluster database, ensuring that the transmission of overall rate-limiting parameters is not affected when one node in the distributed storage system fails. This invention not only resolves competition for storage resource access from clients but also provides higher service quality for high-priority services, solving the problem of not being able to rate-limit storage clients. It helps administrators better allocate storage performance resources, enabling each application client to fully and stably utilize backend storage performance, ensuring that user services are not affected.

[0061] Accordingly, based on the distributed storage QoS control method proposed in this embodiment of the invention, this embodiment of the invention also proposes a distributed storage QoS control system. This QoS control system includes a processor and a memory. When the processor executes a computer program stored in the memory, it implements the distributed storage QoS control method proposed in this embodiment of the invention.

[0062] While one or more embodiments of the present invention have been described above, those skilled in the art will recognize that the present invention can be implemented in any other form without departing from its spirit and scope. Therefore, the embodiments described above are illustrative and not restrictive, and many modifications and substitutions will be apparent to those skilled in the art without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A distributed storage QoS control method applied to a distributed storage system, characterized in that, The distributed storage QoS control method comprises: Setting a rate limiting parameter for a target file storage directory through a storage management interface of the distributed storage system, and writing the rate limiting parameter into an ECTD cluster; The NAS storage node reads the rate limiting parameter and transmits the rate limiting parameter into a corresponding type of shared service module; The shared service module performs corresponding rate limiting processing on the target file storage directory; The setting of the rate limiting parameter for the target file storage directory through the storage management interface of the distributed storage system comprises: Pools all disks on the storage server through the storage management interface of the distributed storage system, and creates a file storage directory list containing the target file storage directory based on the formed storage pool; The rate limiting parameter comprises the name of the target file storage directory, a bandwidth rate limiting value, and a shared service module type, the bandwidth rate limiting value is a global bandwidth rate limiting value or a specified client IP bandwidth rate limiting value, and the shared service module type at least comprises one of an NFS shared service module, an FTP shared service module, and an SMB shared service module; The NAS storage node transmits the rate limiting parameter into the corresponding shared service module comprises: Obtaining the shared service module type according to the rate limiting parameter; Transmitting the rate limiting parameter into the configuration file of the shared service module of the corresponding type; The shared service module performs corresponding rate limiting processing on the target file storage directory comprises: Reading the rate limiting parameter in the configuration file of the shared service module; Modifying the code to realize a predetermined dmClock-based distributed rate limiting strategy; When receiving an access request of a client for the target file storage directory, performing global bandwidth rate limiting or specified client IP bandwidth rate limiting on the target file storage directory based on the dmClock-based distributed rate limiting strategy, and feeding back the rate limiting result to the client; The shared service module accepts the access request of the client according to the C library provided by librados, reads the mapping relationship between the file storage directory and the read-write bandwidth in the newly added configuration file by modifying the source code, and saves the mapping relationship into the token pool of the mounting point; the number of data packets processed per second of the NFS Ganesha, miniFTP, and SMB shared service modules is limited, when the total number of data packets is less than the number of tokens in the token pool, the data packets are allowed to pass, when the total number of data packets is greater than the number of tokens in the token pool, the data packets are rate limited, and a period of time is waited for to prohibit data transmission to supplement the number of tokens, so as to realize the read-write bandwidth limitation of the NAS storage subdirectory file. The shared service module program stores user data into reliable distributed storage cluster through file system library, the file system library runs on the RADOS library, and the speed limiting purpose is realized by modifying the third party open source shared service program source code; the token bucket thought is used to modify each shared service module, tokens are put into the token bucket at a specific rate; according to the preset matching rule, the data packets are classified first, the data packets not meeting the matching rule do not need to be processed by the token bucket and are directly sent; the data packets meeting the matching rule need to be processed by the token bucket; when there are enough tokens in the bucket, the message can be continuously sent, and the token amount in the token bucket is reduced according to the length of the data packet; when the token in the token bucket is insufficient, the data packet cannot be sent, and only when new tokens are generated in the bucket, the data packet can be sent.

2. A distributed storage QoS control system, characterized by, The distributed storage QoS control method comprises a processor and a memory, and the processor realizes the distributed storage QoS control method of claim 1 when executing the computer program stored in the memory.

Images