A storage resource management method and system

Abstract

The application belongs to the technical field of storage, and discloses a storage resource management method and system, which comprises the following steps: defining a cluster storage resource controller, storing bandwidth information of each resource, and performing full-dimension event monitoring on cluster resources; deploying an adjusting unit in each node through the cluster storage resource controller, and allocating a local resource space by using the adjusting unit; according to the bandwidth information in the cluster storage resource controller, calculating a global reference weight of each adjusting unit according to a weight priority in each resource, and performing normalization processing; adding metadata information of a storage resource of each adjusting unit to a ring queue, constructing a storage service quality scheduling virtual ring queue, and dynamically allocating the storage resource of the adjusting unit by using pointer scheduling and time period segmentation. The application improves the flexibility of storage resource allocation among different applications.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a storage resource management method and system. Background Technology

[0002] Edge computing is a product of network evolution and cloud computing technology development. It provides computing, storage, and network infrastructure close to users, and delivers edge cloud services to users by deploying and running applications on this infrastructure. The edge computing market is growing rapidly, and with the surge in application scale and data transmission volume, network bandwidth and throughput have become performance bottlenecks for applications.

[0003] The construction of an edge computing platform involves multiple technical fields, including computing, storage, networking, and security. Kubernetes (K8S) is an open-source container orchestration system that automates the deployment, scaling, and management of containerized applications. In edge computing architectures, K8S can be used to manage containerized applications on edge nodes, enabling better management and scheduling of edge computing resources. K8S applications in edge computing architectures primarily include edge node management, edge node autoscaling, edge node high availability, edge node security, and edge computing application control. The application of K8S in edge computing architectures can improve the efficiency, stability, and reliability of the edge computing platform.

[0004] K3S is a highly available, CNCF-certified Kubernetes distribution designed for unattended, resource-constrained, remote, or IoT-enabled production workloads. It features simple installation, small size, few dependencies, high scalability, and support for heterogeneous resources. Based on these characteristics, K3S is ideally suited for building edge computing infrastructure.

[0005] Regarding storage resources, users expect throughput to simultaneously meet the needs of applications such as video playback, cloud desktop office work, and large-scale games, while maintaining stable throughput without intermittent interruptions. Therefore, providing efficient and stable storage service quality has become the core expectation of users for storage service systems.

[0006] Quality of Service (QoS) is a technology that guarantees a certain level of storage performance. For example, a database server's disk IOPS needs to reach a specific value to ensure high performance. QoS ensures that workload storage resources maintain an appropriate performance level. Although a hard drive controller can handle multiple requests simultaneously, this does not mean they are truly parallel. The physical structure of a hard drive (such as read / write heads and disks) is typically single-threaded, meaning it can only handle one physical operation (such as read or write) at a time. Concurrency is achieved through rapid switching, not true parallelism.

[0007] SSDs offer incredibly fast storage speeds, making technologies like Storage Service Quality (SSQ) previously unnecessary. Early SSDs also had very small capacities, meaning they couldn't be shared by multiple workloads. However, the situation is completely different today. Over the past few years, SSD capacity has steadily increased, and shared workloads have become the new normal, especially in environments like cloud computing, edge computing platforms, and microservices. Storage resources are the most fundamental public infrastructure for applications. As applications grow in size and complexity, multiple workloads compete for IOPS, making SSG technologies essential to ensure the availability of storage resources for mission-critical applications.

[0008] Another benefit of Storage Service Quality (SSQ) is that it makes storage performance more predictable. Without SQ, workload storage resource usage would be limited, potentially resulting in one application generating storage I / O while degrading the I / O performance of other applications. SQ can prioritize available storage bandwidth, ensuring that mission-critical applications receive the necessary bandwidth even during periods of high storage resource contention.

[0009] Quality of Service (QoS) is a technology used to guarantee specific storage performance for applications, ensuring that a particular application or workload always receives specific access performance. QoS is an effective means of resolving storage resource contention, primarily used for storage resource contention between different services and within storage (such as resource contention between internal data recovery and normal business access in the event of a failure), as well as the reasonable allocation of storage resources between different directories of the same application.

[0010] Existing edge computing platforms built on k3s have serious deficiencies in controlling and supporting the quality of storage services. In particular, when the demand for storage resources exceeds the capacity of the storage system, the platform is unable to provide stable and differentiated storage service quality for specific applications.

[0011] Therefore, how to provide a storage resource management method and system is an urgent problem to be solved. Summary of the Invention

[0012] This invention provides a storage resource management method and system to address the serious deficiencies in the control and support of storage service quality in existing edge computing platforms, particularly the inability of the platform to provide stable and differentiated storage service quality for specific applications when storage resource demands exceed the storage system's capacity.

[0013] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0014] According to a first aspect of the present invention, a storage resource management method is provided.

[0015] In one embodiment, the storage resource management method includes:

[0016] Define a cluster storage resource controller to store bandwidth information for each resource and perform full-dimensional event monitoring of cluster resources;

[0017] By using a cluster storage resource controller, scheduling units are deployed on each node, and these scheduling units are used to allocate local resource space.

[0018] Based on the bandwidth information in the cluster storage resource controller, and according to the weight priority of each resource, the global reference weight of each scheduling unit is calculated and normalized.

[0019] The metadata information of the storage resources of each scheduling unit is added to a circular queue to build a virtual circular queue for storage service quality scheduling. Then, pointer scheduling and time period segmentation are used to dynamically allocate the storage resources of the scheduling units.

[0020] In one embodiment, a cluster storage resource controller is defined to store bandwidth information for each resource and perform full-dimensional event monitoring of cluster resources, including:

[0021] The pre-acquired bandwidth information is stored in the cluster storage resource controller;

[0022] The cluster storage resource controller monitors the creation and destruction of scheduling units and the mounting / unmounting of storage volumes.

[0023] Based on the cluster storage resource controller, changes to namespaces and directories in the scheduling unit are monitored;

[0024] The bandwidth information includes: application storage volume information, storage volume bandwidth information, application owner weight, application namespace weight, scheduling unit storage volume list and weight, and weight of read / write priority directories in the storage volume.

[0025] In one embodiment, deploying scheduling units on each node through a cluster storage resource controller and using these scheduling units to allocate local resource space includes:

[0026] Storage resources are proxied through a cluster storage resource controller, utilizing scheduling units deployed on each node.

[0027] The scheduling unit in each node manages the allocation of resource space for local hard disks and quasi-local hard disk spaces within the node.

[0028] In one embodiment, the global reference weight of each scheduling unit is calculated based on the bandwidth information in the cluster storage resource controller and the weight priority of each resource, and then normalized, including:

[0029] Based on the priority of each type of weight in the bandwidth information, calculate the global reference weight for each scheduling unit;

[0030] The weights of database and storage service scheduling units are set higher than those of applications without disk read / write operations, which serves as the basis for calculation.

[0031] In one embodiment, the expression for calculating the global reference weight of the scheduling unit is:

[0032] ;

[0033] ;

[0034] In the formula, Indicates the global reference weight of the scheduling unit. This represents the base weight of the user who created the application. This represents the weight of each role in the list of roles created for the user. This indicates the weight of the namespace to which the scheduling unit belongs. This indicates the storage resource weight of the scheduling unit. Indicates the weight of the storage volume. This indicates the weight of the directory on the storage volume mounted by the scheduling unit. This indicates the storage volume mounted by the scheduling unit. This indicates the directory of the storage volume mounted by the scheduling unit. This indicates the impact of setting directories in the storage disk volume of the scheduling unit on the quality of storage services, and R represents the list of roles for creating users.

[0035] In one embodiment, the metadata information of each scheduling unit's storage resources is added to a circular queue to construct a virtual circular queue for storage service quality scheduling. Dynamic allocation of the scheduling unit's storage resources is then performed using pointer scheduling and time period segmentation.

[0036] By using the scheduling unit of the nodes in the virtual circular queue to schedule storage resources in accordance with storage service quality, and in combination with pointer scheduling, storage resource read and write requests are processed.

[0037] The corresponding pointers are set according to the number of CPU cores, and a virtual circular queue is scheduled in conjunction with the storage service quality scheduling, while multiple read and write requests for storage resources are processed.

[0038] The metadata information includes: the storage resource weights and weight ratios of the scheduling unit, the list and weights of storage volumes, and the directories and weights of the storage volumes mounted by the scheduling unit.

[0039] In one embodiment, processing storage resource read / write requests by scheduling the nodes in the virtual circular queue through storage service quality scheduling, combined with pointer scheduling, includes:

[0040] Based on the cluster storage resource controller, the pointer is pointed to the starting node in the virtual circular queue for storage service quality scheduling, and the scheduling unit of that node is used to process the read and write requests for storage resources.

[0041] After a pre-set time has elapsed, and based on the current level of storage activity, the pointer is moved to the next node in the storage service quality scheduling virtual circular queue.

[0042] In one embodiment, the expression for the busyness level of storage operations is:

[0043] ;

[0044] In the formula, This indicates the disk of the current database server. This indicates the maximum number of disks supported by the database server. Indicates the current bandwidth utilization rate. Indicates the maximum bandwidth. This indicates the disk's adjustment factor. The bandwidth adjustment factor. This indicates the current level of activity in storage operations.

[0045] In one embodiment, setting corresponding pointers based on the number of CPU cores, combining storage service quality scheduling with a virtual circular queue, and simultaneously processing multiple read / write requests for storage resources includes:

[0046] Set multiple pointers based on the number of CPU cores, and the number of pointers shall not exceed the number of CPU cores;

[0047] By creating a new scheduling unit, calculating the global reference weight for the new scheduling unit, adding it to the circular queue, updating the virtual circular queue for storage service quality scheduling, and controlling the storage resources of the node where the scheduling unit is located.

[0048] According to a second aspect of the present invention, a storage resource management system is provided.

[0049] In one embodiment, the storage resource management system includes:

[0050] The resource storage module is used to define the cluster storage resource controller, store the bandwidth information of each resource, and perform full-dimensional event monitoring of cluster resources.

[0051] The deployment and allocation module is used to deploy scheduling units on each node through the cluster storage resource controller, and to allocate local resource space using the scheduling units.

[0052] The weight normalization module is used to calculate the global reference weight of each scheduling unit based on the bandwidth information in the cluster storage resource controller and the weight priority of each resource, and then perform normalization processing.

[0053] The resource management module is used to add the metadata information of the storage resources of each scheduling unit to a circular queue, build a virtual circular queue for storage service quality scheduling, and dynamically allocate the storage resources of the scheduling unit using pointer scheduling and time period segmentation.

[0054] According to a third aspect of the present invention, a computer device is provided.

[0055] In some embodiments, the computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0056] According to a fourth aspect of the present invention, a computer-readable storage medium is provided.

[0057] In one embodiment, a computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the above method.

[0058] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0059] 1. This invention prioritizes storage resources based on edge computing platform users, namespaces, Pod categories, storage volumes, and directories, thereby improving the flexibility of storage resource allocation among different applications.

[0060] 2. This invention constructs a storage resource scheduling ring and combines it with a pointer scheduling mechanism to allocate different storage request processing times to Pods. At the same time, the multi-pointer scheduling mechanism further improves the concurrency and speed of resource scheduling. When Pods are reading and writing, a preemptible time slice allocation mode is adopted to prioritize the service quality of storage resource-sensitive applications.

[0061] 3. When a new Pod is added or scheduled to a cluster node, the algorithm can quickly recalculate the weights and update the virtual ring, ensuring that the new Pod can obtain the required storage resources in a timely manner, thus enhancing the scalability and adaptability of the system.

[0062] 4. In response to the quality requirements of application storage services, this invention implements bandwidth QoS (read bandwidth, write bandwidth, total bandwidth) and IOPS (read IOPS, write IOPS, total IOPS) limits on storage resources. Suitable control methods can be flexibly combined to ensure that different applications have different read and write priorities for storage resources, and the same application has different read and write priorities for different directories.

[0063] 5. This invention can configure higher storage QoS for high-priority services, lower QoS for low-priority services, higher QoS for high-priority directories, and lower QoS for low-priority directories, thereby rationally allocating storage resources to meet the needs of different levels of services.

[0064] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0066] Figure 1 This is a flowchart illustrating a storage resource management method according to an exemplary embodiment;

[0067] Figure 2 This is a schematic diagram illustrating a storage resource management system according to an exemplary embodiment;

[0068] Figure 3 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0069] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some embodiments may include or substitute parts and features of other embodiments. The scope of the embodiments herein encompasses the entire scope of the claims and all available equivalents thereof. Throughout this document, the terms “first,” “second,” etc., are used only to distinguish one element from another without requiring or implying any actual relationship or order between the elements. Indeed, a first element can also be referred to as a second element, and vice versa. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a structure, apparatus, or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a structure, apparatus, or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the structure, apparatus, or device that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0070] The terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" used in this document to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings. They are used solely for the convenience of describing the document and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In the description herein, unless otherwise specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two elements; they can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0071] In this document, unless otherwise stated, the term "multiple" means two or more.

[0072] In this article, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0073] In this article, the term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0074] It should be understood that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0075] The modules in the apparatus or system of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0076] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0077] Figure 1 An embodiment of a storage resource management method according to the present invention is shown.

[0078] In this optional embodiment, the storage resource management method includes:

[0079] Step S101: Define a cluster storage resource controller to store the bandwidth information of each resource and perform full-dimensional event monitoring of cluster resources.

[0080] Step S102: Through the cluster storage resource controller, a scheduling unit is deployed in each node, and the scheduling unit is used to allocate local resource space.

[0081] Step S103: Based on the bandwidth information in the cluster storage resource controller, calculate the global reference weight of each scheduling unit according to the weight priority of each resource, and perform normalization processing.

[0082] Step S104: Add the metadata information of the storage resources of each scheduling unit to the circular queue to build a virtual circular queue for storage service quality scheduling, and use pointer scheduling and time period segmentation to dynamically allocate the storage resources of the scheduling units.

[0083] In this optional embodiment, a cluster storage resource controller is defined to store bandwidth information for each resource and to perform full-dimensional event monitoring of cluster resources. This includes: storing pre-acquired bandwidth information in the cluster storage resource controller; listening to the creation, destruction, and mounting / unmounting events of scheduling units and storage volumes through the cluster storage resource controller; and monitoring changes to namespaces and directories in scheduling units based on the cluster storage resource controller. The bandwidth information includes: application storage volume information, storage volume bandwidth information, application owner weight, application namespace weight, scheduling unit storage volume list and weight, and weight of read / write priority directories in storage volumes.

[0084] It should be noted that the scheduling unit is the Pod, which is the smallest unit of scheduling in Kubernetes and contains one or more containers.

[0085] In this optional embodiment, deploying scheduling units in each node through the cluster storage resource controller and allocating local resource space using the scheduling units includes: performing storage resource proxying through the cluster storage resource controller using the scheduling units deployed in each node; and managing the allocation of resource space for local hard disks and quasi-local hard disk spaces in each node through the scheduling units in each node.

[0086] In this optional embodiment, the global reference weight of each scheduling unit is calculated based on the bandwidth information in the cluster storage resource controller and the weight priority of each resource, and the normalization process includes: calculating the global reference weight of each scheduling unit based on the priority of each type of weight in the bandwidth information; setting the weight of database and storage service scheduling units higher than the weight of applications without disk read / write as the basis for calculation.

[0087] In this optional embodiment, the calculation expression for the global reference weight of the scheduling unit is:

[0088] ;

[0089] ;

[0090] In the formula, Indicates the global reference weight of the scheduling unit. This represents the base weight of the user who created the application. This represents the weight of each role in the list of roles created for the user. This indicates the weight of the namespace to which the scheduling unit belongs. This indicates the storage resource weight of the scheduling unit. Indicates the weight of the storage volume. This indicates the weight of the directory on the storage volume mounted by the scheduling unit. This indicates the storage volume mounted by the scheduling unit. This indicates the directory of the storage volume mounted by the scheduling unit. This indicates the impact of setting directories in the storage disk volume of the scheduling unit on the quality of storage services, and R represents the list of roles for creating users.

[0091] In this optional embodiment, the metadata information of the storage resources of each scheduling unit is added to a circular queue to construct a virtual circular queue for storage service quality scheduling. Dynamic allocation of storage resources for the scheduling units is performed using pointer scheduling and time period segmentation. This includes: processing storage resource read / write requests through the scheduling units of nodes in the virtual circular queue for storage service quality scheduling, combined with pointer scheduling; setting corresponding pointers based on the number of CPU cores, and simultaneously processing multiple read / write requests for storage resources in conjunction with the virtual circular queue for storage service quality scheduling; wherein the metadata information includes: the storage resource weight of the scheduling unit, the weight ratio, the list and weight of storage volumes, and the directory and weight of the storage volumes mounted by the scheduling unit.

[0092] In this optional embodiment, processing storage resource read / write requests through the scheduling unit of the nodes in the storage service quality scheduling virtual circular queue, combined with pointer scheduling, includes: based on the cluster storage resource controller, pointing the pointer to the starting node in the storage service quality scheduling virtual circular queue, and using the scheduling unit of that node to process the storage resource read / write requests; after a preset time has elapsed, and based on the current storage operation's busy level, moving the pointer to the next node in the storage service quality scheduling virtual circular queue.

[0093] In this optional embodiment, the expression for the busyness level of storage operations is:

[0094] ;

[0095] In the formula, This indicates the disk of the current database server. This indicates the maximum number of disks supported by the database server. Indicates the current bandwidth utilization rate. Indicates the maximum bandwidth. This indicates the disk's adjustment factor. The bandwidth adjustment factor. This indicates the current level of activity in storage operations.

[0096] In this optional embodiment, setting corresponding pointers based on the number of CPU cores, and combining the storage service quality scheduling virtual circular queue, while performing multiple processing on read and write requests for storage resources, includes: setting multiple pointers based on the number of CPU cores, with the number of pointers not exceeding the number of CPU cores; creating a new scheduling unit, calculating the global reference weight for the new scheduling unit, adding it to the circular queue, updating the storage service quality scheduling virtual circular queue, and controlling the storage resources of the node where the scheduling unit is located.

[0097] Figure 2 An embodiment of a storage resource management system according to the present invention is shown.

[0098] In this optional embodiment, the storage resource management system includes:

[0099] Resource storage module 201 is used to define the cluster storage resource controller, store the bandwidth information of each resource, and perform full-dimensional event monitoring of cluster resources.

[0100] The deployment allocation module 202 is used to deploy scheduling units in each node through the cluster storage resource controller, and to allocate local resource space using the scheduling units;

[0101] The weight normalization module 203 is used to calculate the global reference weight of each scheduling unit based on the bandwidth information in the cluster storage resource controller and the weight priority of each resource, and then perform normalization processing.

[0102] The resource management module 204 is used to add the metadata information of the storage resources of each scheduling unit to the circular queue, construct a virtual circular queue for storage service quality scheduling, and dynamically allocate the storage resources of the scheduling unit using pointer scheduling and time period segmentation.

[0103] To facilitate understanding of the above technical solutions of the present invention, the following further describes the above technical solutions of the present invention from the perspectives of architecture and principle, as follows:

[0104] The keywords used for retrieval, their English expressions, abbreviations, and key terms are defined as follows:

[0105] K8S is a container management engine (Kubernetes).

[0106] K3S, a simplified version of K8S, retains only the core functions.

[0107] A Pod is the smallest unit of scheduling in Kubernetes, containing one or more containers.

[0108] PV, in Kubernetes, is a data volume resource that provides persistent storage for Pods.

[0109] DaemonSet is a Pod that runs on each node in Kubernetes.

[0110] A Deployment is a resource object in Kubernetes that schedules a group of Pods.

[0111] The specific steps for implementing this plan are as follows:

[0112] 1) Define a cluster storage resource controller, storage-qos-controller. This controller is a deployment resource containing three Pod replicas. It mainly stores the QoS information of various resources on the storage platform, such as the weight of the application owner, the weight of the application namespace, the list and weight of the Pod's storage volumes, and the weight of the directories that need to be read and written according to priority in each storage volume. At the same time, it listens for Pod creation and destruction, storage volume mount / unmount events, namespace changes, directory changes, etc.

[0113] 2) Each node on the platform deploys a daemonset resource storage agent, storage-qos-agent. Its main function is to directly manage the resource space allocation of the node's local hard disk / quasi-local hard disk (supporting underlying file interfaces such as ext4, xfs, and btrfsd). Each data volume has three replicas, distributed across different nodes. When an application needs to read or write to a storage volume, storage-qos-agent requests the application's storage resource QoS configuration from storage-qos-controller. Then, it selects the node with the lowest operating system disk I / O load among the three data volume replicas as the master node for this read / write request. The scheduling is then performed as follows: after the master node completes the read / write operation, the Raft protocol is used to synchronize the data to the other two replica data volumes.

[0114] 3) Let the application namespace of the edge computing platform be... The namespace storage resource weight is namespace The storage resource weights of the Pods in the Pods are set as follows: The storage resource weight of the data volume mounted by the Pod is set to The weights of directories within the data volume that require priority control are set to... .

[0115] Each category has a weight value of 1-10. The larger the value, the higher the priority. Database and storage service Pods are set to high weight, while HTTP front-end and back-end applications and other applications without disk read / write are set to low weight.

[0116] 4) Calculate the global reference weight for each Pod. Its expression is as follows:

[0117] ;

[0118] ;

[0119] In the formula, Indicates the global reference weight of the scheduling unit. This represents the base weight of the user who created the application. This represents the weight of each role in the list of roles created for the user. This indicates the weight of the namespace to which the scheduling unit belongs. This indicates the storage resource weight of the scheduling unit. Indicates the weight of the storage volume. This indicates the weight of the directory on the storage volume mounted by the scheduling unit. This indicates the storage volume mounted by the scheduling unit. This indicates the directory of the storage volume mounted by the scheduling unit. This indicates the impact of setting directories in the storage disk volume of the scheduling unit on the quality of storage services, and R represents the list of roles for creating users.

[0120] in, This determines the impact of users, namespaces, Pods, and Pod storage volumes on the QoS of the final storage resources; This determines the impact of the directory settings in the storage disk volume of the Pod on the QoS of storage resources.

[0121] 5) To Normalization is performed. ,in, This represents the weight ratio of the i-th Pod, where n represents the number of Pods on the node. This represents the global reference weight of the Pod mentioned in step 4) of the i-th step. This represents the sum of the global reference weights of each Pod in this node.

[0122] 6) Configure a virtual circular queue for storage resource QoS scheduling, and add the metadata information of each Pod's storage resources to the circular queue; the metadata information mainly includes: Pod storage resource weight. Weighting ratio Storage volume list and weights, the directory and weights of storage volumes mounted by the scheduling unit, etc.

[0123] 7) Set pointer p to point to the start node of the virtual circular queue for storage resource QoS scheduling. Let the time period window be T, and the processing time allocated to each Pod be... When the current Pod's processing time is reached At that time, the pointer moves to the next node in the storage resource QoS scheduling virtual circular queue.

[0124] 8) When processing storage resource read / write requests for each Pod, the time period will be... Divide the data into time slices ΔT; calculate the relative weight ratio of each read / write directory. , where V p This refers to the storage volume that the Pod is mounted on, D v This represents the directory of the storage volume mounted by the Pod; each directory receives a base time slice of Ni = Di * ΔT. During each scheduling, time slices are preferentially allocated to directories under high-weight storage volumes, while allowing preemption of time slices that low-weight storage volume directories would otherwise have received; the expression for quantifying the busyness of the current storage operation using load L is as follows:

[0125] ;

[0126] In the formula, This indicates the disk of the current database server. This indicates the maximum number of disks supported by the database server. Indicates the current bandwidth utilization rate. Indicates the maximum bandwidth. This indicates the disk's adjustment factor. The bandwidth adjustment factor. This indicates the current level of activity in storage operations.

[0127] Where λ+μ=1, with default λ=0.6 and μ=0.4; when L<0.8, scheduling and rotation are performed normally according to the weight ratio; when L≥0.8, it indicates that the current load is high and time slice preemption may occur, requiring an extension of the scheduling time slice interval for low-priority storage volume directories, with the interval number n=L / L0, where L0 is the baseline load, defaulting to 0.5; the continuous unservice tolerance time for any storage volume must not exceed 3* (i.e., 3 complete Pod storage resource scheduling cycles) When the waiting time of a low-priority storage volume and directory reaches the tolerance time, a compensation time slice will be forcibly inserted in the next Pod storage resource scheduling cycle to schedule read and write requests for that directory.

[0128] 9) Multiple pointers p can be set according to the number of CPU cores. i ~p n The number of pointers is less than or equal to the number of CPU cores in the node, and the time interval between the start of each pointer rotation cycle is T. The processing is carried out according to the process described in steps 7) and 8).

[0129] 10) When a new Pod is created or scheduled to the current cluster node, repeat steps 4) to 8) to reconstruct the virtual ring and implement priority control of application storage resources.

[0130] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores static and dynamic information data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0131] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0132] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0133] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0134] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0135] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.

Claims

1. A storage resource management method, characterized in that, include: Define a cluster storage resource controller to store bandwidth information for each resource and perform full-dimensional event monitoring of cluster resources; By using a cluster storage resource controller, scheduling units are deployed on each node, and these scheduling units are used to allocate local resource space. Based on the bandwidth information in the cluster storage resource controller, and according to the weight priority in each resource, the global reference weight of each scheduling unit is calculated. Metadata information of storage resources for each scheduling unit is added to a circular queue to construct a virtual circular queue for storage service quality scheduling. Pointer scheduling and time-period segmentation are then used to dynamically allocate storage resources for each scheduling unit. Specifically, a global reference weight for each scheduling unit is calculated. The expression is as follows: ； ； In the formula, Indicates the global reference weight of the scheduling unit. This represents the base weight of the user who created the application. This represents the weight of each role in the list of roles created for the user. This indicates the weight of the namespace to which the scheduling unit belongs. This indicates the storage resource weight of the scheduling unit. Indicates the weight of the storage volume. This indicates the weight of the directory on the storage volume mounted by the scheduling unit. This indicates the storage volume mounted by the scheduling unit. This indicates the directory of the storage volume mounted by the scheduling unit. This indicates the impact of setting directories in the storage disk volume of the scheduling unit on the quality of storage services; R represents the list of roles for creating users. right Normalization is performed. , This represents the weight ratio of the i-th scheduling unit, and n represents the number of scheduling units on the node. This represents the global reference weight of the i-th scheduling unit. This represents the sum of global reference weights for each scheduling unit in the node; the metadata information includes: the storage resource weights, weight ratios, storage volume lists and weights of the scheduling units, and the directories and weights of the storage volumes mounted by the scheduling units. The dynamic allocation of storage resources for scheduling units using pointer scheduling and time period segmentation includes: Let the time period window be T, and the processing time allocated to each scheduling unit be... , This represents the weight ratio of the i-th scheduling unit, and the processing time when it reaches the current scheduling unit. At that time, the pointer moves to the next node in the virtual circular queue for storage service quality scheduling.

2. The storage resource management method according to claim 1, characterized in that, The defined cluster storage resource controller stores bandwidth information for each resource and performs full-dimensional event monitoring of cluster resources, including: The pre-acquired bandwidth information is stored in the cluster storage resource controller; The cluster storage resource controller monitors the creation and destruction of scheduling units and the mounting / unmounting of storage volumes. Based on the cluster storage resource controller, changes to namespaces and directories in the scheduling unit are monitored; The bandwidth information includes: application storage volume information, storage volume bandwidth information, application owner weight, application namespace weight, scheduling unit storage volume list and weight, and weight of read / write priority directories in the storage volume.

3. The storage resource management method according to claim 1, characterized in that, The step of deploying scheduling units on each node through a cluster storage resource controller and allocating local resource space using these scheduling units includes: Storage resources are proxied through a cluster storage resource controller, utilizing scheduling units deployed on each node. The scheduling unit in each node manages the allocation of resource space for local hard disks and quasi-local hard disk spaces within the node.

4. The storage resource management method according to claim 1, characterized in that, The step of adding the metadata information of each scheduling unit's storage resources to a circular queue to construct a virtual circular queue for storage service quality scheduling, and dynamically allocating the storage resources of the scheduling units using pointer scheduling and time period segmentation, includes: Storage resource read and write requests are processed by scheduling units of nodes in the virtual circular queue, combined with pointer scheduling, through storage service quality scheduling.

5. The storage resource management method according to claim 4, characterized in that, The step of adding the metadata information of each scheduling unit's storage resources to a circular queue to construct a virtual circular queue for storage service quality scheduling, and dynamically allocating the storage resources of the scheduling units using pointer scheduling and time period segmentation, further includes: Set multiple pointers based on the number of CPU cores, and the number of pointers shall not exceed the number of CPU cores; By creating a new scheduling unit, calculating the global reference weight for the new scheduling unit, adding it to the circular queue, updating the virtual circular queue for storage service quality scheduling, and controlling the storage resources of the node where the scheduling unit is located.

6. A storage resource management system, used to implement the steps of the storage resource management method according to any one of claims 1-5, characterized in that, include: The resource storage module is used to define the cluster storage resource controller, store the bandwidth information of each resource, and perform full-dimensional event monitoring of cluster resources. The deployment and allocation module is used to deploy scheduling units on each node through the cluster storage resource controller, and to allocate local resource space using the scheduling units. The weight normalization module is used to calculate the global reference weight of each scheduling unit based on the bandwidth information in the cluster storage resource controller and the weight priority of each resource, and then perform normalization processing. The resource management module is used to add the metadata information of the storage resources of each scheduling unit to a circular queue, build a virtual circular queue for storage service quality scheduling, and dynamically allocate the storage resources of the scheduling unit using pointer scheduling and time period segmentation.

Images