A Method and System for Quality Control of Distributed File System Services Based on KVSSD

Abstract

This invention discloses a service quality control method and system for a distributed file system based on KVSSD. The method includes: abstracting the overall I / O capacity of the backend KVSSD device storage cluster into a global logical resource pool; when a client initiates a read / write request and reaches the metadata service of the distributed file system, the service quality service module embedded in the metadata service performs logical resource quota checks and admission control on the read / write request based on the current resource usage status of the global logical resource pool; when the request passes the admission control, the metadata service returns a data replica service address to the client, enabling the client to initiate a data access request to the data replica service corresponding to the data replica service address. The service quality service module and the data replica service work together to complete the execution and monitoring of read / write requests, as well as the feedback and adjustment of logical resource quotas. This invention can achieve storage resource sharing and isolation, improving resource utilization efficiency.

Description

Technical Field

[0001] This invention relates to the field of data storage and management technology, specifically to the field of resource isolation and quality of service assurance technology based on KVSSD, and particularly to a method and system for quality of service control of a distributed file system based on KVSSD. Background Technology

[0002] In distributed system environments, high-performance distributed file systems are widely used to store and manage massive amounts of application data. A typical architecture usually consists of two core modules: a metadata service and a data replica service. When a client performs read or write operations, it first queries the metadata service for metadata information about the target file, such as data block location, replica distribution strategy, and access permissions. Then, it directly communicates with the corresponding data replica service node to complete the actual reading or writing of business data. This decoupled design effectively decouples the control plane from the data plane, improving the system's scalability and concurrent processing capabilities.

[0003] In recent years, Key-Value SSDs (KVSSDs) have gradually become a key hardware choice for building next-generation high-performance storage platforms due to their low latency, high throughput, and hardware interface characteristics that are naturally compatible with key-value semantics. Compared to traditional block devices, KVSSDs can bypass the address mapping overhead of the file system layer and directly access data in key-value pairs, significantly improving I / O efficiency and providing more efficient and flexible underlying storage support for distributed file systems.

[0004] However, in real-world deployments where multiple tenants or applications share the same KVSSD resource pool, the lack of fine-grained resource management mechanisms can easily lead to problems such as I / O resource contention, performance interference, and even loss of Quality of Service (QoS). For example, high-priority applications may experience latency spikes due to sudden traffic surges from low-priority tasks, or critical services may fail to meet Service Level Agreement (SLA) requirements due to bandwidth preemption. These issues not only diminish the performance advantages of KVSSD but also limit its scalable application in production-grade distributed file systems.

[0005] Therefore, there is an urgent need for a resource isolation and QoS guarantee mechanism that can deeply coordinate with the hardware characteristics of KVSSD and be embedded into the distributed file system architecture to achieve end-to-end, fine-grained control over key performance indicators (such as IOPS, bandwidth, latency, etc.). Based on this, this invention proposes a KVSSD-based distributed file system service quality control method and system. This solution fully leverages the high-performance potential of KVSSD while ensuring predictable, isolated, and guaranteed service quality between different applications or tenants through a hardware-software co-control strategy, providing key technical support for building a highly reliable and efficient cloud-native storage infrastructure. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a method and system for quality control of distributed file system services based on KVSSD.

[0007] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a KVSSD-based distributed file system service quality control method, comprising the following steps: (1) Construct a global logical resource pool in the metadata service of the distributed file system. The global logical resource pool is used to abstractly represent the overall I / O capability of the physical storage cluster composed of multiple KVSSD devices in the backend. (2) When a client initiates a read / write request and reaches the metadata service of the distributed file system, the service quality service module embedded in the metadata service performs logical resource quota checks and admission control on the read / write request based on the current resource usage status of the global logical resource pool. (3) When a read / write request passes the admission control, the metadata service returns the data replica service address to the client so that the client can initiate a data access request to the data replica service corresponding to the data replica service address. The service quality service module and the data replica service in the metadata service jointly complete the execution and monitoring of the read / write request, as well as the feedback and adjustment of the logical resource quota.

[0008] Furthermore, the construction of a global logical resource pool in the metadata service of the distributed file system specifically includes: Obtain the I / O capability parameters of each KVSSD device in the backend physical storage cluster, and aggregate them to generate the total capacity of the global logical resource pool; the I / O capability parameters include the IOPS limit and the bandwidth limit.

[0009] Furthermore, step (1) also includes: A global logical resource view matching the global logical resource pool is created to record the total capacity, the current resource usage recorded at the file volume or directory level, and the current remaining available resources. The current resource usage is obtained through a maintained real-time consumption counter, which is atomically incremented according to the admission threshold when read / write requests pass admission control, and periodically calibrated according to the real-time I / O performance indicators reported by the data replica service, so that the current resource usage reflects the execution status of read / write requests.

[0010] Furthermore, step (1) also includes: When creating a file volume in a distributed file system, the metadata structure is extended by introducing resource quota attributes to configure the logical resource quota allocated from the global logical resource pool for that file volume. The resource quota attributes include the upper limit of read IOPS, the upper limit of write IOPS, the upper limit of read and write bandwidth, and the average latency target. The metadata includes the volume name, volume owner, volume size, address information, namespace identifier, and access permissions. Associating logical resource quotas with a specified directory path under a file volume enables all read and write requests under the specified directory path to share the logical resource quota bound to that directory, thereby achieving directory-level resource isolation and control. Establish a mapping relationship between the logical resource quotas allocated to file volumes or directories and the backend KVSSD devices to form a global resource mapping table with physical traceability and topology awareness.

[0011] Furthermore, the Quality of Service (QoS) module performs logical resource quota checks and admission control on read and write requests based on the current resource usage status of the global logical resource pool, specifically including: To maintain an active request pool and a pending request pool for read and write requests, the active request pool is used to store currently being processed read and write requests, and the pending request pool is used to cache read and write requests that have passed verification but need to wait for resource release. Based on the file volume or directory to which the read / write request belongs, the current cumulative resource usage is queried in real time using a real-time consumption counter, and compared with the preset logical resource quota: If the current resource usage does not exceed the preset logical resource quota, the read / write request is allowed to continue. The metadata service returns the metadata of the corresponding data replica service to the client to guide the client to complete the data access. If the current resource usage exceeds the preset logical resource quota and the active request pool has not reached its capacity limit, the read / write request will be placed in the active request pool, and the read / write requests in the active request pool will be allowed to continue to be executed in order. The metadata service returns the metadata of the corresponding data replica service to the client to guide the client to complete the data access. If the current resource usage exceeds the preset logical resource quota, and the active request pool has reached its capacity limit, but the pending request pool has not reached its capacity limit, then the software rate limiting mechanism will be used to rate limit the read / write request, put it into the pending request pool, delay the response, and prompt the client to actively wait or retry. If the current resource usage exceeds the preset logical resource quota, and both the active request pool and the pending request pool have reached their capacity limits, the read / write request will be rejected directly, and a standardized error code will be returned.

[0012] Furthermore, the execution and monitoring of read / write requests, as well as the feedback and adjustment of logical resource quotas, jointly completed by the Quality of Service module and the Data Replication Service in the metadata service, specifically include: After receiving a data access request from a client, the data replica service converts it into key-value operation instructions that conform to the KVSSD interface specification, and then executes the key-value operation instructions on the KVSSD devices managed by the data replica service. The data replica service monitors and continuously collects real-time I / O performance metrics of the KVSSD devices it manages, including IOPS throughput, bandwidth utilization, and request latency distribution. The data replica service periodically or on demand reports real-time I / O performance metrics and local resource usage status to the service quality service module in the metadata service to update the resource usage status of the global logical resource pool and the global logical resource view, and dynamically adjusts the logical resource quota accordingly.

[0013] A second aspect of this invention provides a system for implementing the above-described KVSSD-based distributed file system service quality control method, comprising: The client is responsible for providing access interfaces to users and initiating read and write requests to the distributed file system. The metadata service node is used to construct a global logical resource pool, which is used to abstractly represent the overall I / O capability of a physical storage cluster composed of multiple KVSSD devices in the backend. The metadata service node includes a service quality service module, which responds to read and write requests from clients, performs logical resource quota checks and admission control on read and write requests based on the current resource usage status of the global logical resource pool, and returns the data replica service address to the client after passing the admission control. The data replica service node is used to receive data access requests initiated by clients based on the data replica service address, and works with the service quality service module to complete the execution and monitoring of read and write requests, as well as the feedback and adjustment of logical resource quotas; The backend KVSSD device, as the underlying storage medium, is used to provide data persistence capabilities for the upper-layer metadata service nodes and data replica service nodes.

[0014] Furthermore, the metadata service node also includes a quota management service module, which specifically includes: The quota information management service module is used to utilize the file volume management function to introduce resource quota attributes to extend the metadata structure when creating a file volume, so as to configure the logical resource quota allocated from the global logical resource pool for that file volume; The namespace management service module is used to associate logical resource quotas with specified directory paths under file volumes, so that all read and write requests under the specified directory path share the logical resource quotas bound to that directory, thereby achieving directory-level resource isolation and control. The global resource management service module is used to establish a mapping relationship between the logical resource quotas allocated to file volumes or directories and the backend KVSSD devices, so as to form a global resource mapping table with physical traceability and topology awareness; the global resource mapping table is used to record the mapping relationship between the logical resource quotas allocated to each file volume or directory and its mapped backend KVSSD devices.

[0015] Furthermore, the data replication service node includes a replication execution and monitoring service module, which includes: The replica execution service module includes an instruction conversion module and an instruction execution module. The instruction conversion module is used to convert the client's data access request into key-value operation instructions that conform to the KVSSD interface specification. The instruction execution module is used to send the key-value operation instructions to its KVSSD device and perform data read and write operations according to the key-value operation instructions. The monitoring service module includes a local resource monitoring module and a monitoring data reporting module. The local resource monitoring module is used to monitor and continuously collect real-time I / O performance indicators of the KVSSD devices it manages. The monitoring data reporting module is used to periodically or as needed to report real-time I / O performance indicators and local resource usage status to the service quality service module in order to update the resource usage status of the global logical resource pool and the global logical resource view, and dynamically adjust the logical resource quota of each file volume or directory based on the global logical resource view.

[0016] Compared with the prior art, the beneficial effects of the present invention are:

[0017] (1) Achieve end-to-end, guaranteed performance isolation and quality of service: This invention deeply integrates the QoS service module into the metadata service of the distributed file system and deploys the replica execution and monitoring service in the data replica service to build a closed-loop control mechanism of "global decision-making + local execution". This mechanism enables I / O requests to be subject to admission control at the initiation stage, avoiding invalid or excessive requests from impacting the network and back-end storage from the source. It can provide end-to-end IOPS, bandwidth and latency indicators from the client to the physical storage medium for different applications or tenants, effectively solving the performance interference problem in a multi-tenant environment.

[0018] (2) Significantly improves resource utilization efficiency and system stability: This invention constructs a multi-dimensional, configurable logical resource quota view, abstracting the total I / O capacity of the physical KVSSD cluster into a unified logical resource pool. Through the collaboration of the global logical resource view and real-time monitoring feedback, this invention can accurately perceive the resource consumption of each file volume or directory and perform fine-grained scheduling according to preset quotas. This mechanism avoids the idleness or contention caused by the "one-size-fits-all" resource allocation in traditional solutions. While ensuring the performance of critical business operations, it maximizes the resource utilization efficiency of the entire storage cluster and effectively prevents the overall system performance degradation caused by local overload.

[0019] (3) Achieving fine-grained and flexible resource configuration and management: The quota management service of this invention not only supports configuring resource limits such as IOPS and bandwidth at the file volume level, but also innovatively supports associating quota policies with specific directory paths within the volume, realizing fine-grained resource control at the directory level and even the subtree level. At the same time, by establishing a physical topology mapping between quota policies and backend KVSSD devices, resource allocation becomes traceable and perceptive, providing administrators with extremely high flexibility and enabling them to customize differentiated resource policies according to different business priorities and application characteristics.

[0020] (4) Fully leverage the performance advantages of KVSSD hardware: The data replication service in this invention directly converts client requests into key-value operation instructions natively supported by KVSSD for execution; in conjunction with fine-grained QoS services for scheduling, this mechanism can safely and orderly leverage the low latency and high throughput hardware characteristics of KVSSD devices in multi-tenant high-concurrency scenarios, avoiding performance decline caused by resource contention, and fully releasing the performance potential of new storage hardware.

[0021] (5) Possesses adaptive adjustment capabilities and good scalability: The monitoring service in the data replica service of this invention continuously collects real-time performance indicators of local KVSSD devices and periodically reports them to the metadata service, enabling the global logical resource view to be dynamically updated. This closed-loop feedback mechanism gives the system the ability to adaptively adjust the quota strategy according to the real-time load, laying the foundation for intelligent operation and maintenance. At the same time, this invention adopts a distributed architecture, separating QoS decision-making from data replica execution. Each component can be horizontally scaled to cope with larger-scale clusters and more tenant requests, possessing good horizontal scalability. Attached Figure Description

[0022] Figure 1 This is a flowchart of the KVSSD-based distributed file system service quality control method of the present invention; Figure 2 This is the overall deployment architecture diagram of the KVSSD-based distributed file system service quality control system of the present invention; Figure 3 This is an architecture diagram of the quota management service module of the present invention; Figure 4 This is an architecture diagram of the QoS service module of the present invention; Figure 5 This is an architecture diagram of the copy execution and monitoring service module of the present invention; Figure 6 This is a flowchart of the write request processing of the present invention. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0025] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0026] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0027] This invention provides a method and system for Quality of Service (QoS) control of a distributed file system based on KVSSD (Key-Value SSD). The invention aims to address the lack of effective I / O resource management and performance isolation mechanisms in existing distributed file systems when dealing with new storage hardware like KVSSD. Building upon traditional metadata and data replication services, this invention integrates a control system for I / O resources of KVSSD storage clusters. This QoS control system is deeply integrated into the distributed file system, providing end-to-end, guaranteed resource isolation for different applications or tenants accessing storage through the distributed file system. This enables end-to-end, fine-grained management of key performance indicators such as IOPS (IOPS), bandwidth, and latency.

[0028] See Figure 1 The KVSSD-based distributed file system service quality control method of the present invention is built upon the capabilities of existing distributed file systems and specifically includes the following steps:

[0029] (1) First, initialize the storage space. When initializing the storage space, a unified global logical resource pool is built in the metadata service of the distributed file system, which can effectively guarantee resource isolation and service quality in a multi-tenant environment. Among them, the global logical resource pool is used to abstractly represent the overall I / O capability of the physical storage cluster composed of multiple KVSSD devices in the backend.

[0030] It should be understood that the global logical resource pool is essentially a storage resource virtualization architecture that pools and isolates the I / O resources provided by a physical storage cluster composed of multiple KVSSD devices. Multiple KVSSD devices are aggregated into a unified physical storage cluster, serving as the underlying hardware support. Overall I / O capability (such as total IOPS, total bandwidth, etc.) is the core performance indicator of the pooled resources. The aggregated physical I / O performance serves as the upper limit of the total capacity of the global logical resource pool.

[0031] Furthermore, a global logical resource pool is constructed within the metadata service of the distributed file system. This specifically includes obtaining the I / O capability parameters of each KVSSD device in the backend physical storage cluster and aggregating them to generate the total capacity of the global logical resource pool. The I / O capability parameters include IOPS limits and bandwidth limits, among others.

[0032] Furthermore, a global logical resource view matching the global logical resource pool is created to record global resource status information such as total capacity, current resource usage recorded at the file volume or directory level, and current remaining available resources. The current resource usage is obtained through a maintained real-time consumption counter, which records the amount of resources currently allocated and used. This real-time consumption counter is atomically incremented according to the admission criteria when read / write requests pass admission control, and periodically calibrated based on real-time I / O performance metrics reported by the data replica service to ensure that the current resource usage reflects the actual execution status of read / write requests. When an upper-layer service initiates an I / O request (i.e., a read / write request), the QoS service module intercepts the I / O resource demand corresponding to the I / O request, queries the global logical resource view (rather than directly manipulating the global logical resource pool), and then, based on this global logical resource view, determines whether the I / O resource demand exceeds the logical resource quota limit, adopting different control strategies to achieve on-demand allocation and fine-grained management of storage resources. Simultaneously, the real-time consumption counter is updated, and the current resource usage in the global logical resource view is refreshed.

[0033] It should be noted that the global logical resource view and the global resource mapping table work together: the global logical resource view maintains the dynamic resource status, and the global resource mapping table maintains the static topology mapping relationship between logical resource quotas and backend KVSSD devices. The QoS service module uses both of them when performing subsequent logical resource quota checks.

[0034] Furthermore, step (1) also includes the following sub-steps:

[0035] (1.1) When creating a file volume in a distributed file system, the metadata structure is extended by introducing resource quota attributes to configure the logical resource quota allocated from the global logical resource pool for that file volume. The metadata structure refers to the metadata content accompanying the creation of the file volume. The metadata mainly includes basic attributes such as volume name, volume owner, allocated storage space size, address information, namespace identifier, target replica location, replica distribution strategy, and access permissions. The resource quota attribute, as a unified data structure, specifically includes QoS metrics such as read IOPS limit, write IOPS limit, read / write bandwidth limit, and average latency target, which can be configured individually.

[0036] (1.2) Associate logical resource quotas with a specified directory path under the file volume, so that all I / O requests under the specified directory path share the logical resource quota bound to the directory, thereby achieving fine-grained resource isolation and control at the directory level or even the subtree level.

[0037] (1.3) Establish a mapping relationship between the logical resource quota allocated to the file volume or directory and the backend KVSSD device to form a global resource mapping table with physical traceability and topology awareness. In this way, during subsequent scheduling, the resource consumption can be traced back to the specific physical KVSSD device according to the mapping relationship in the global resource mapping table, ensuring that the resource allocation has physical traceability and topology awareness, and providing support for subsequent precise scheduling control.

[0038] Specifically, the above mapping relationship supports the following two modes: ① Explicit binding: When configuring resource quota attributes, a specific KVSSD device serial number, the node ID where the KVSSD device is located, or a specific physical storage pool is directly specified; the metadata service strongly binds these identifiers with the universally unique identifier (UUID) of the resource quota attribute. ② Implicit mapping: Based on the attributes of the file volume / directory (such as business type, performance requirements, etc.), it automatically maps it to a set of KVSSD devices that meet the upper limit conditions of the corresponding resource quota attribute; when performing implicit mapping, common data placement algorithms can be used, such as consistent hashing algorithm, controlled replication under scalable hashing (CRUSH algorithm), etc., for placement mapping.

[0039] (2) When a client initiates a read / write request and reaches the metadata service of the distributed file system, the QoS service module embedded in the metadata service performs logical resource quota checks and admission control on the read / write request based on the current resource usage status of the global logical resource pool.

[0040] Furthermore, the QoS service module performs logical resource quota checks and admission control on read and write requests based on the current resource usage status of the global logical resource pool. Specifically, this includes maintaining two logical request queues: an active request pool and a pending request pool. The active request pool stores currently processed I / O requests, while the pending request pool caches I / O requests that have passed initial verification but are awaiting resource release. While the distributed file system's metadata service handles necessary routine metadata operations such as file path resolution, access permission acquisition, permission verification, and target replica location acquisition, the QoS service module synchronously intervenes. Based on the file volume or directory to which the read / write request (i.e., I / O request) belongs, it queries the current cumulative resource usage (including IOPS, read / write bandwidth, etc.) in real-time using a real-time consumption counter and compares it with the preset logical resource quota. If the current resource usage does not exceed (i.e., is less than or equal to) the preset logical resource quota, the read / write request is allowed to continue. The metadata service returns the corresponding data replica service's metadata, such as address information and namespace identifiers, to the client to guide the client to complete subsequent data access. If the current resource usage exceeds the preset logical resource quota, and the active request pool has not reached its capacity limit, the read / write request is placed in the active request pool, and read / write requests in the active request pool are allowed to continue execution in order. The metadata service returns the metadata of the corresponding data replica service to the client to guide the client to complete data access. If the current resource usage exceeds the preset logical resource quota, and the active request pool has reached its capacity limit, but the pending request pool has not reached its capacity limit, a software rate limiting mechanism is used to rate limit the read / write request, placing it in the pending request pool, delaying the response, and prompting the client to actively wait or retry. Read / write requests in the active request pool are executed synchronously, and resources are released after execution. After resource release, read / write requests in the pending request pool are moved into the active request pool in order, which can effectively avoid resource overload and performance degradation. If the current resource usage exceeds the preset logical resource quota, and both the active request pool and the pending request pool have reached their capacity limits, the read / write request will be rejected directly, and a standardized error code (such as QUOTA_EXCEEDED) will be returned to avoid resource overload and performance degradation.

[0041] It's important to note that in a distributed file system, the metadata service and the data replication service work collaboratively to process and execute read / write requests and control local resources. The metadata service provides the address of the data replication service; the data replication service is responsible for providing the actual data read / write operations.

[0042] (3) When a read / write request passes the admission control, the metadata service returns the data replica service address to the client so that the client can initiate a data access request to the data replica service corresponding to the data replica service address. The QoS service module and the data replica service in the metadata service jointly complete the execution and monitoring of the read / write request, as well as the feedback and adjustment of the logical resource quota.

[0043] Furthermore, the QoS service module and data replica service in the metadata service jointly complete the execution and monitoring of read and write requests, as well as the feedback and adjustment of logical resource quotas, specifically including the following sub-steps:

[0044] (3.1) After receiving the data access request from the client, the data replica service converts it into a key-value operation instruction that conforms to the KVSSD interface specification, and then sends the actual I / O processing instruction to the KVSSD device managed by the data replica service through the key-value operation instruction to complete the execution of the read and write request.

[0045] (3.2) The data replica service runs its internal resource monitoring module to monitor local resources, monitors and continuously collects the real-time I / O performance indicators of the KVSSD devices it manages, including IOPS throughput, bandwidth utilization, request latency distribution, etc.

[0046] (3.3) The data replica service reports real-time I / O performance indicators and local resource usage status to the QoS service module in the metadata service periodically or as needed through monitoring data reporting. Based on the reported real-time I / O performance indicators, the real-time consumption counter is periodically calibrated, thereby updating the current resource usage, and then dynamically updating the resource usage status of the global logical resource pool and the global logical resource view, and dynamically adjusting the logical resource quota accordingly.

[0047] It should be noted that there are two ways to report monitoring data: periodic or on-demand. Periodic refers to reporting according to a set period (e.g., 10s-30s). On-demand refers to triggering reporting through command-line operations on a specified data replica service.

[0048] In summary, the method described in this invention covers logical resource quotas, QoS resource monitoring, and control mechanisms, effectively solving the problem of traditional distributed file systems lacking fine-grained resource management capabilities in new KVSSD hardware environments. It provides a reliable technical foundation for application scenarios sensitive to I / O performance, such as cloud-native, AI training, and big data analysis.

[0049] It is worth mentioning that this invention also provides a KVSSD-based distributed file system service quality control system to implement the KVSSD-based distributed file system service quality control method described in the above embodiments. This system is built upon the capabilities of existing distributed file systems, and its overall deployment architecture is as follows: Figure 2 As shown, it specifically includes the following components: client, metadata service node, data replica service node, and backend KVSSD device.

[0050] The architecture consists of several layers: the client provides access interfaces to users and initiates read and write requests to the distributed file system; the metadata service node runs metadata services to handle various metadata operations within the distributed file system, such as directory structure, permission information, and file attributes; the data replica service node provides distributed read / write and replica management capabilities; and the backend KVSSD device, a solid-state storage device supporting key-value interfaces, serves as the underlying storage medium, providing efficient and low-latency data persistence capabilities for the upper-layer metadata service node and data replica service node. Building upon the traditional distributed file system architecture, a quota management service module and a QoS service module are further introduced and deeply integrated into the metadata service node. Simultaneously, a replica execution and monitoring service module is embedded within the data replica service node. Through the collaborative operation of the QoS service module and the replica execution and monitoring service module, a closed-loop control mechanism of global decision-making and local execution is constructed, enabling fine-grained, end-to-end management of KVSSD device resources (including key performance indicators such as IOPS, bandwidth, and latency), effectively ensuring resource isolation and service quality in a multi-tenant environment.

[0051] In this embodiment, the metadata service node is used to construct a global logical resource pool. The global logical resource pool is used to abstractly represent the overall I / O capability of the physical storage cluster composed of multiple KVSSD devices in the backend. Multiple metadata service nodes collaboratively maintain the global logical resource pool and the global logical resource view through a consistency mechanism. Specifically, in a deployment mode where multiple metadata service nodes coexist, the global logical resource pool is maintained across nodes, and the global logical resource view is synchronized among multiple metadata service nodes through a consistency mechanism. The metadata service node includes a QoS service module, which is used to respond to read and write requests from clients, perform logical resource quota checks and admission control on read and write requests based on the current resource usage status of the global logical resource pool, and return the data replica service address to the client after passing the admission control.

[0052] It should be noted that in a distributed file system deployment with multiple metadata service nodes, the global logical resource pool, as a unified logical abstraction across nodes, is collaboratively maintained by multiple metadata service nodes. Each metadata service node can maintain the consistency of the global logical resource view through a master-slave consistency mechanism or a distributed state synchronization protocol (such as Raft, Paxos, etc.). One metadata service node can act as the master node of the global logical resource pool, maintaining the authoritative view. Other metadata service nodes obtain view copies through synchronization or caching mechanisms, thereby performing low-latency quota checks and admission control based on their local view copies. The data replica service reports real-time I / O performance metrics to the master node or all metadata service nodes to ensure the real-time performance and consistency of the global logical resource view.

[0053] Furthermore, the metadata service node also includes a quota management service module, which is built into the metadata service node and is responsible for quota management and control at the tenant, user, or file volume level. The architecture of the quota management service module is as follows: Figure 3 As shown, it specifically includes a quota information management service module, a namespace management service module, and a global resource management service module.

[0054] The quota information management service module is tightly integrated with the metadata service of the distributed file system, enabling quota management and control at the tenant, user, or file volume level. Based on the existing namespace and metadata management capabilities of the metadata service (such as file path resolution and replica location mapping), the quota information management service module utilizes file volume management functions to extend the metadata structure by introducing resource quota attributes when creating a file volume in the distributed file system. This allows for the configuration of logical resource quotas allocated from the global logical resource pool for that file volume, and the logical resource quotas of the KVSSD device are persistently stored as part of the file volume metadata. The configured resource quota attribute parameters are shown in Table 1.

[0055] Table 1: Resource Quota Attribute Parameters Quota parameters Description ReadIOPSLimit Read IOPS limit WriteIOPSLimit Write IOPS limit ReadBandwidthLimit Read bandwidth limit WriteBandwidthLimit Write bandwidth limit LatencyTarget Average delay target

[0056] To enhance the flexibility and granularity of resource control, the namespace management service module supports path- or directory-level quota settings. This allows logical resource quotas to be associated with specified directory paths within a file volume, enabling all I / O requests under a given directory path to share the logical resource quota bound to that directory. This achieves finer-grained directory-level resource isolation and control. For example, certain directories can be assigned higher priority and larger bandwidth quotas; such as configuring a high-priority, high-bandwidth quota for the ` / data / high_priority_app / ` directory, while setting a lower IOPS limit for the ` / data / archive / ` directory. This functionality is achieved by extending existing directory metadata attributes. The metadata service automatically parses and applies the corresponding logical resource quotas when processing path-related requests.

[0057] The Global Resource Management Service module establishes a mapping between the logical resource quotas allocated to file volumes or directories and the backend KVSSD devices, forming a global resource mapping table with physical traceability and topology awareness. This ensures that resource allocation possesses physical traceability and topology awareness. The global resource mapping table records the mapping relationship between the logical resource quotas allocated to each file volume or directory and its mapped backend KVSSD device. This global resource mapping table supports dynamic updates to adapt to specific resource load balancing requirements.

[0058] Furthermore, the metadata service node includes a QoS service module, integrated within the metadata service. This module is responsible for performing logical resource quota checks and admission control at the request entry point, effectively preventing invalid or excessive requests from causing congestion in network transmission and backend storage nodes. The architecture of the QoS service module is as follows: Figure 4 As shown, the specific process steps are as follows:

[0059] Step S1, I / O Request Interception and Processing: When a client initiates a file I / O request, such as a write request `write(fileX, offset, data)`, it first sends a metadata operation request to the metadata service, such as retrieving the target copy location. During this process, the QoS service module transparently intercepts the I / O request before initiating the logical resource quota check in Step S2. Specifically, the QoS service module mounts an interceptor in the metadata service's I / O request processing chain. When the client's metadata request arrives, the interceptor directly captures the I / O request and extracts key context parameters for quota checking. This includes extracting context information such as the file volume / directory to which the I / O request belongs, and combining this with a real-time consumption counter to query the current resource usage. The extracted current resource usage does not need to be retransmitted over the network; instead, it is directly stored in the metadata service's internal memory and passed as context to the QoS service module for processing.

[0060] Step S2: Logical Resource Quota Check: The metadata service queries the preset logical resource quota configuration based on the file volume or directory to which the I / O request belongs, and queries the current resource usage (such as IOPS, bandwidth) in real time through a real-time consumption counter. The current resource usage is compared with the preset logical resource quota to determine whether the current resource usage exceeds the preset logical resource quota limit. The real-time consumption counter is maintained by the metadata service and dynamically updated based on performance feedback data periodically reported by the data replica service.

[0061] It should be noted that real-time consumption counters are of several types, such as IOPS counters for recording IOPS and bandwidth counters for recording bandwidth. These counters are implemented using atomic variables at the underlying level. The real-time consumption counters are maintained using a two-level mechanism of "admission counting + actual measurement calibration": ① Admission-level counting: The real-time consumption counters are maintained by the metadata service, and their statistical scope is the number of requests that have passed admission control. Whenever an I / O request passes admission control, the metadata service increments the IOPS counter by 1 or accumulates the bandwidth counter by the number of data bytes of that request. Each counter is bound to a sliding time window, sampling the current counter value at a fixed time granularity (e.g., every 10 milliseconds) and pushing the sampling points into a circular buffer. When the QoS service module needs to check logical resource quotas, it sums or weighted averages all sampling points within the current sliding window to obtain the real-time IOPS per second or real-time bandwidth (MB / s). Since the counting action occurs locally in the metadata service, quota checks do not require additional network interaction, ensuring low latency in admission decisions. ② Actual Measurement Calibration: Since read and write requests are sent directly from the client to the data replica service for execution after passing admission control, actual execution conditions (such as client request cancellation, network transmission failure, client retries, and multi-replica writes) can cause discrepancies between the admission-level count and the actual resource consumption. Therefore, the monitoring service module in the data replica service continuously collects its actual I / O performance metrics and reports them to the metadata service periodically or as needed. The metadata service calibrates the real-time consumption counter based on the measured values ​​and refreshes the current resource usage recorded in the global logical resource view with the calibrated results, thereby ensuring that the current resource usage accurately reflects the actual resource consumption on the backend KVSSD device.

[0062] Step S3, Admission Control: The system maintains two logical request queues: an active request pool and a pending request pool. The active request pool stores currently processed I / O requests, while the pending request pool caches I / O requests that have passed initial verification but are awaiting resource release. The system sets a maximum timeout threshold for I / O requests in the pending request pool. If an I / O request waits in the pending request pool for longer than this maximum timeout threshold, the system will actively wake it up and reject it (returning a timeout error) to prevent client I / O requests from deadlocking. When an I / O request in the active request pool completes execution and releases resources, the system triggers a resource release event, retrieves one or more I / O requests from the pending request pool, moves them back to the active request pool, and wakes up the corresponding client thread. Based on the quota check results of Step S2, the system decides whether to allow or deny, or rate-limit, read / write requests to ensure the service quality for each tenant or application.

[0063] Specifically, based on the quota check results of step S2, the following strategy is executed, such as... Figure 6 As shown:

[0064] Step S31: If the current resource usage does not exceed the preset logical resource quota, then the I / O request is allowed to pass, that is, the I / O request is immediately accepted and allowed to continue to execute, continue to complete metadata parsing, determine one or more data replica service nodes responsible for storing the target data, and return the address list of these nodes to the client, that is, return the target data replica service address list to the client, allow it to initiate subsequent data read and write, and guide the client to complete data access.

[0065] Step S32: If the current resource usage exceeds the preset logical resource quota and the active request pool has not reached its capacity limit, then I / O requests are allowed to pass through. That is, I / O requests are immediately accepted, new I / O requests are placed into the active request pool, and I / O requests are allowed to continue to be executed.

[0066] Step S33: If the current resource usage exceeds the preset logical resource quota and the I / O request exceeds the capacity limit of the active request pool, but the pending request pool is not yet full, then the I / O request is rate-limited. That is, the I / O request is placed in the pending request pool and enters the rate-limiting state. The software rate-limiting mechanism can be used to implement rate-limiting measures such as delayed response and reduced scheduling priority for the I / O request. The I / O requests in the active request pool are processed synchronously. After the I / O request is completed and the resources are released, the I / O request in the pending request pool is put back into the active request pool for processing.

[0067] Step S34: If the current resource usage exceeds the preset logical resource quota, and both the active request pool and the pending request pool are full, newly arrived over-limit I / O requests will be directly rejected, and a standardized error code QUOTA_EXCEEDED will be returned to the client, indicating insufficient resource quota.

[0068] In this embodiment, the data replica service node is used to receive data access requests initiated by the client based on the data replica service address, and works with the QoS service module to complete the execution and monitoring of read and write requests, as well as the feedback and adjustment of logical resource quotas.

[0069] Furthermore, the data replica service node includes a replica execution and monitoring service module, responsible for KVSSD instruction conversion, execution, and local resource monitoring. The architecture of the replica execution and monitoring service module is as follows: Figure 5 As shown, it specifically includes a replica execution service module and a monitoring service module.

[0070] The replica execution service module comprises an instruction conversion module and an instruction execution module. The former converts client data access requests into key-value operation instructions conforming to the KVSSD interface specification, such as the write instruction PUT(key,value), the read instruction GET(key), and the delete instruction DELETE(key). PUT(key,value) writes data (value) to the specified key (key), GET(key) reads the corresponding data (value) based on the specified key (key), and DELETE(key) deletes the specified key (key) and its corresponding data. The latter sends the converted key-value operation instructions to its corresponding KVSSD device and executes the actual data read / write operations based on the instructions. This instruction conversion and execution process bypasses traditional block device or POSIX file interfaces, fully leveraging the low latency and high concurrency hardware advantages of KVSSD devices.

[0071] The monitoring service module includes a local resource monitoring module and a monitoring data reporting module. The local resource monitoring module is a monitoring agent deployed on the data replica service node, used to monitor and continuously collect real-time I / O performance metrics of the KVSSD devices it manages, such as IOPS, bandwidth, and latency, as shown in Table 2. The monitoring data reporting module periodically or on demand reports real-time I / O performance metrics and local resource usage status to the QoS service module of the metadata service node to update the resource usage status of the global logical resource pool and the global logical resource view. Based on the global logical resource view, it dynamically adjusts the logical resource quotas of each file volume or directory to achieve adaptive QoS assurance and resource optimization. For example, when monitoring data shows that the resource utilization of a KVSSD device (e.g., IOPS utilization is below 30%) and the device is in good health, it is determined that the KVSSD device has resource redundancy. Then, the logical resource quota is dynamically adjusted, dynamically increasing the logical resource quota limit of high-priority file volumes on the KVSSD device, or allowing new bursts of traffic, thereby improving the overall resource utilization and throughput of the physical storage cluster.

[0072] Table 2: I / O Performance Indicators Performance indicators Description ReadIOPS Read IOPS WriteIOPS Write IOPS ReadBandwidth Read bandwidth WriteBandwidth Write bandwidth Latency Request Delay List

[0073] In Table 2, the request latency list refers to the latency percentile statistics, including P50_Latency (median latency, 50% of requests take less than this value), P99_Latency (long-tail latency, 99% of requests take less than this value, reflecting performance under extreme conditions), and P999_Latency (99.9% of requests take less than this value, reflecting the most extreme jitter of the system).

[0074] In summary, the system described in this invention, through its integrated resource management and QoS guarantee mechanisms, not only improves the resource utilization and stability of the distributed file system in high-concurrency environments but also provides reliable quality of service guarantees for multi-tenant scenarios, meeting the needs of modern large-scale data storage and processing. Addressing the requirements of resource control, security, and performance isolation in multi-tenant environments, this invention integrates a QoS control system into the distributed file system, achieving comprehensive awareness and fine-grained control of the I / O resources of the backend KVSSD device storage cluster. This provides end-to-end, guaranteed resource isolation and QoS services for different applications or tenants accessing the storage via the distributed file system.

[0075] For example, taking a typical write request as an example, the workflow of the method and system described in this invention will be explained. The write process is as follows: Figure 6 As shown, the specific steps include the following:

[0076] Step 1, Write Request: The client calls the file write interface to initiate a data write request to the distributed file system, such as write(fileX, offset, data), which means writing the data content data to the target file fileX, specifically starting from the offset of the file.

[0077] Step 2: The QoS service module of the metadata service node intercepts the I / O request: The I / O request first arrives at the metadata service. Before performing regular metadata operations (such as path resolution and replica location), the embedded QoS service module transparently intercepts the I / O request and triggers the logical resource quota check process.

[0078] Step 3: Identify the path and load the logical resource quota configuration: The metadata service parses the file volume and directory to which the file path belongs based on the file path in the I / O request, and queries the associated preset logical resource quota. For example, the system determines that the preset logical resource quota configuration for this directory is: a maximum read IOPS of 5000, a maximum write IOPS of 5000, and a maximum total bandwidth of 100 MB / s.

[0079] Step 4, Admission Control: The QoS service module combines the real-time resource usage of the current directory (e.g., current read IOPS is 3000, write IOPS is 4000) with the preset logical resource quota limit to determine whether to allow the write request to pass, and makes the following decisions based on the system's internal queue status: If the current resource usage does not exceed the preset logical resource quota, the request is allowed to pass directly and proceeds to Step 5; If the current resource usage exceeds the preset logical resource quota, and the active request pool is not full, the request is allowed to pass and placed in the active request pool, proceeding to Step 5; If the current resource usage exceeds the preset logical resource quota, and the active request pool is full, but the pending request pool is not full, the write request is placed in the pending request pool for rate limiting and waiting. The system will continue to execute Step 5 after subsequent resource release and write requests are scheduled to the active request pool; If the current resource usage exceeds the preset logical resource quota, and both the active request pool and the pending request pool are full, the write request is directly rejected, a standardized error code QUOTA_EXCEEDED is returned to the client, and the process ends.

[0080] Step 5: Return the target data replica service address: After completing the access control check, the metadata service returns a list of network addresses of one or more target data replica services to the client based on the writable replica information of the file.

[0081] Step 6: The client sends write data to the data replica service: Based on the returned address, the client sends the actual data write request to the selected data replica service node.

[0082] Step 7, Command Conversion and Command Execution: After receiving a write request, the command conversion module converts the request into a key-value operation command conforming to the KVSSD interface specification, such as PUT(key,value). The command execution module then writes the command to the locally managed KVSSD device to perform the actual data read and write operations, fully leveraging its low latency and high throughput hardware characteristics.

[0083] While read and write operations are in progress, the monitoring service module in the data replica service synchronously updates the local resource usage status (such as write IOPS, bandwidth consumption, etc.) and periodically reports I / O performance metrics (including IOPS, bandwidth, latency, etc.) to the metadata service.

[0084] The metadata service aggregates feedback data from various data replica services, performs on-site testing and calibration of the admission count value of the real-time consumption counter, and dynamically refreshes and updates the global logical resource view, providing real-time basis for subsequent quota checks, scheduling decisions, and adaptive QoS adjustments.

[0085] In summary, the above process fully embodies the technical characteristics of this invention: "global awareness, forward interception, collaborative execution, and closed-loop feedback." This ensures that the resources of KVSSD devices are subject to refined and reliable quality of service control in a high-concurrency, multi-tenant environment. This invention achieves refined, end-to-end management and control of KVSSD cluster I / O resources, effectively guaranteeing performance isolation and quality of service in a multi-tenant environment.

[0086] Through the aforementioned automated and intelligent resource isolation and QoS control system, this invention not only achieves efficient and secure storage resource sharing and isolation, significantly improving the overall resource utilization efficiency of the system, but also provides solid technical support for the distributed file system to maintain stable and efficient business operation in high-concurrency and multi-tenant scenarios through a multi-dimensional and configurable resource abstraction and collaborative monitoring mechanism.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A quality control method for distributed file system services based on KVSSD, characterized in that, Includes the following steps: (1) Construct a global logical resource pool in the metadata service of the distributed file system. The global logical resource pool is used to abstractly represent the overall I / O capability of the physical storage cluster composed of multiple KVSSD devices in the backend. (2) When a client initiates a read / write request and reaches the metadata service of the distributed file system, the service quality service module embedded in the metadata service performs logical resource quota checks and admission control on the read / write request based on the current resource usage status of the global logical resource pool. The Quality of Service (QoS) module performs logical resource quota checks and admission control on read and write requests based on the current resource usage status of the global logical resource pool, specifically including: To maintain an active request pool and a pending request pool for read and write requests, the active request pool is used to store currently being processed read and write requests, and the pending request pool is used to cache read and write requests that have passed verification but need to wait for resource release. Based on the file volume or directory to which the read / write request belongs, the current cumulative resource usage is queried in real time using a real-time consumption counter, and compared with the preset logical resource quota: If the current resource usage does not exceed the preset logical resource quota, the read / write request is allowed to continue. The metadata service returns the metadata of the corresponding data replica service to the client to guide the client to complete the data access. If the current resource usage exceeds the preset logical resource quota and the active request pool has not reached its capacity limit, the read / write request will be placed in the active request pool, and the read / write requests in the active request pool will be allowed to continue to be executed in order. The metadata service returns the metadata of the corresponding data replica service to the client to guide the client to complete the data access. If the current resource usage exceeds the preset logical resource quota, and the active request pool has reached its capacity limit, but the pending request pool has not reached its capacity limit, then the software rate limiting mechanism will be used to rate limit the read / write request, put it into the pending request pool, delay the response, and prompt the client to actively wait or retry. If the current resource usage exceeds the preset logical resource quota, and both the active request pool and the pending request pool have reached their capacity limits, the read / write request will be rejected directly, and a standardized error code will be returned. (3) When a read / write request passes the admission control, the metadata service returns the data replica service address to the client so that the client can initiate a data access request to the data replica service corresponding to the data replica service address. The service quality service module and the data replica service in the metadata service jointly complete the execution and monitoring of the read / write request, as well as the feedback and adjustment of the logical resource quota.

2. The KVSSD-based distributed file system service quality control method according to claim 1, characterized in that, The construction of a global logical resource pool in the metadata service of the distributed file system specifically includes: Obtain the I / O capability parameters of each KVSSD device in the backend physical storage cluster, and aggregate them to generate the total capacity of the global logical resource pool; the I / O capability parameters include the IOPS limit and the bandwidth limit.

3. The KVSSD-based distributed file system service quality control method according to claim 1, characterized in that, Step (1) further includes: A global logical resource view matching the global logical resource pool is created to record the total capacity, the current resource usage recorded at the file volume or directory level, and the current remaining available resources. The current resource usage is obtained through a maintained real-time consumption counter, which is atomically incremented according to the admission threshold when read / write requests pass admission control, and periodically calibrated according to the real-time I / O performance indicators reported by the data replica service, so that the current resource usage reflects the execution status of read / write requests.

4. The KVSSD-based distributed file system service quality control method according to claim 1, characterized in that, Step (1) further includes: When creating a file volume in a distributed file system, the metadata structure is extended by introducing resource quota attributes to configure the logical resource quota allocated from the global logical resource pool for that file volume. The resource quota attributes include the upper limit of read IOPS, the upper limit of write IOPS, the upper limit of read / write bandwidth, and the average latency target. The metadata includes the volume name, volume owner, volume size, address information, namespace identifier, and access permissions. Associating logical resource quotas with a specified directory path under a file volume enables all read and write requests under the specified directory path to share the logical resource quota bound to that directory, thereby achieving directory-level resource isolation and control. Establish a mapping relationship between the logical resource quotas allocated to file volumes or directories and the backend KVSSD devices to form a global resource mapping table with physical traceability and topology awareness.

5. The KVSSD-based distributed file system service quality control method according to claim 1, characterized in that, The execution and monitoring of read / write requests, as well as the feedback and adjustment of logical resource quotas, are jointly completed by the Quality of Service (QoS) module and the Data Replication Service within the metadata service. Specifically, this includes: After receiving a data access request from a client, the data replica service converts it into key-value operation instructions that conform to the KVSSD interface specification, and then executes the key-value operation instructions on the KVSSD devices managed by the data replica service. The data replica service monitors and continuously collects real-time I / O performance metrics of the KVSSD devices it manages, including IOPS throughput, bandwidth utilization, and request latency distribution. The data replica service periodically or on demand reports real-time I / O performance metrics and local resource usage status to the service quality service module in the metadata service to update the resource usage status of the global logical resource pool and the global logical resource view, and dynamically adjusts the logical resource quota accordingly.

6. A system for implementing the KVSSD-based distributed file system service quality control method according to any one of claims 1-5, characterized in that, include: The client is responsible for providing access interfaces to users and initiating read and write requests to the distributed file system. The metadata service node is used to construct a global logical resource pool, which is used to abstractly represent the overall I / O capability of a physical storage cluster composed of multiple KVSSD devices in the backend. The metadata service node includes a service quality service module, which responds to read and write requests from clients, performs logical resource quota checks and admission control on read and write requests based on the current resource usage status of the global logical resource pool, and returns the data replica service address to the client after passing the admission control. The data replica service node is used to receive data access requests initiated by clients based on the data replica service address, and works with the service quality service module to complete the execution and monitoring of read and write requests, as well as the feedback and adjustment of logical resource quotas; The backend KVSSD device, as the underlying storage medium, is used to provide data persistence capabilities for the upper-layer metadata service nodes and data replica service nodes.

7. The system according to claim 6, characterized in that, The metadata service node also includes a quota management service module, which specifically includes: The quota information management service module is used to utilize the file volume management function to introduce resource quota attributes to extend the metadata structure when creating a file volume, so as to configure the logical resource quota allocated from the global logical resource pool for that file volume; The namespace management service module is used to associate logical resource quotas with specified directory paths under file volumes, so that all read and write requests under the specified directory path share the logical resource quotas bound to that directory, thereby achieving directory-level resource isolation and control. The global resource management service module is used to establish a mapping relationship between the logical resource quotas allocated to file volumes or directories and the backend KVSSD devices, so as to form a global resource mapping table with physical traceability and topology awareness; the global resource mapping table is used to record the mapping relationship between the logical resource quotas allocated to each file volume or directory and its mapped backend KVSSD devices.

8. The system according to claim 6, characterized in that, The data replication service node includes a replication execution and monitoring service module, which includes: The replica execution service module includes an instruction conversion module and an instruction execution module. The instruction conversion module is used to convert the client's data access request into key-value operation instructions that conform to the KVSSD interface specification. The instruction execution module is used to send the key-value operation instructions to its KVSSD device and perform data read and write operations according to the key-value operation instructions. The monitoring service module includes a local resource monitoring module and a monitoring data reporting module. The local resource monitoring module is used to monitor and continuously collect real-time I / O performance indicators of the KVSSD devices it manages. The monitoring data reporting module is used to periodically or as needed to report real-time I / O performance indicators and local resource usage status to the service quality service module in order to update the resource usage status of the global logical resource pool and the global logical resource view, and dynamically adjust the logical resource quota of each file volume or directory based on the global logical resource view.

Images