Cluster resource quota management method, system, device and storage medium

Abstract

The present invention discloses a cluster resource quota management method, system, device and storage medium. Based on the designed custom indicators, the resource usage of the task can be recorded, such as storage, memory, CPU and GPU resources, etc., which solves the problem of setting resource quotas at multiple dimensions (group, individual), as well as the problem of setting resource quotas for different types of GPUs, thereby achieving efficient resource quota management. In addition, the time series database of the Prometheus monitoring tool can record the time series change information of the quotas used by users and groups. Therefore, the present invention can obtain the changes in the amount of resources used by users and groups in the recent period, thereby more effectively monitoring the usage of cluster resources. In addition, the present invention is easy to expand and can support quota control of more types of GPUs.

Description

Technical Field

[0001] The present invention relates to the technical field of cluster resource quota management, and in particular to a cluster resource quota management method, system, device and storage medium. Background Art

[0002] Deep learning has achieved remarkable results in recent years, and this progress is inseparable from the support of powerful computing resources. With the popularity of containerization and efficient container orchestration tools such as Kubernetes (K8s), major internet companies and research institutions have begun to build clusters based on K8s to centrally manage computing resources. A comprehensive K8s cluster often needs to manage massive amounts of storage, memory, CPU, and various types of GPU resources, while also meeting the needs of users from different business groups.

[0003] To ensure efficient resource utilization, a quota management system is crucial in Kubernetes clusters. To effectively manage resource usage by groups and users, a Kubernetes quota management system must limit resource usage at multiple levels, such as group and individual. Furthermore, Kubernetes clusters often include multiple GPU types, such as the NVIDIA A100 and NVIDIA GTX 3090. These GPUs have varying levels of video memory, computing power, and application scenarios. Therefore, a quota management system must be able to distinguish between different types of GPUs and manage their quotas accordingly. However, Kubernetes's built-in quota mechanism controls resource quotas at only a single level (group or user) and cannot distinguish between different types of GPU resources, resulting in limited flexibility and scalability.

[0004] Specifically, the existing technology is the default resource quota mechanism of K8s, which divides cluster resources into four types: storage, memory, CPU and GPU. This mechanism needs to be used in conjunction with the namespace mechanism to limit the total amount of resources requested by all Pods under a specified namespace. When the total amount of resources occupied by all Pods under the namespace reaches the limit value, new Pods cannot be created under the namespace. For simple scenarios, such as only requiring resource restrictions at the user level, a unique namespace can be set for each user, which can be met by combining the quota mechanism that comes with K8s. However, for more complex application scenarios, the existing technology will expose obvious shortcomings. Mainly including:

[0005] (1) K8s clusters are often used by users in different business groups. To effectively control usage by groups and individuals, the platform must be able to implement resource restrictions at both the group and individual levels. Existing technologies rely on namespaces to function, but namespaces in K8s are independent and cannot be nested or hierarchical. Consequently, existing technologies can only implement resource restrictions at a single level (group or individual), which is insufficient to meet demand.

[0006] (2) Existing technologies categorize cluster resources into four types: storage, memory, CPU, and GPU, but fail to differentiate between different types of GPU resources. However, in most cases, K8s clusters often contain multiple types of GPUs, such as NVIDIA A100 and NVIDIA GTX 3090. These GPUs have different computing capabilities and are used in different scenarios. The cluster needs to be able to distinguish between different types of GPUs and manage quotas separately. Existing technologies can control total GPU usage, but cannot control the usage of two different types of GPUs, NVIDIA A100 and NVIDIA GTX 3090, separately, which cannot meet the needs. Summary of the Invention

[0007] The purpose of the present invention is to provide a cluster resource quota management method, system, device and storage medium, which can solve the quota management problem of K8s cluster in multiple levels (groups, users) and multi-type GPU scenarios, and realize efficient resource quota management.

[0008] The purpose of the present invention is achieved through the following technical solutions:

[0009] A cluster resource quota management method, comprising:

[0010] Stores the group resource quota of each group, the personal resource quota of each user's group, and information about each task;

[0011] For the tasks currently to be scheduled, a quota check is performed: the corresponding user number and group number are determined based on the information of the tasks currently to be scheduled, and the remaining resource quotas of the individual and group are determined in combination with the corresponding personal resource quotas and group resource quotas as well as the information obtained through the Prometheus monitoring tool. If the remaining quotas of the individual and group meet the resource requirements of the tasks currently to be scheduled, the corresponding custom indicators are generated using the task information through quota checking; all custom indicators of the task are pushed to the Pushgateway, and the application programming interface is called to create the corresponding Pod. After the Pod is scheduled by K8s, resources are allocated from the resource pool and run; each task generates multiple custom indicators, each custom indicator corresponds to a resource type, and the multiple custom indicators generated have several identical labels for identifying the basic information of the task, and are configured with different resource type labels. The resource requirements of the task are identified through all resource type labels;

[0012] Monitor the status changes of all Pods at all times. When the Pod corresponding to the task is completed normally, fails, or is deleted, the resources occupied by the Pod will be released, and the callback function will be triggered to delete all custom indicators of the Pod's corresponding task in Pushgateway. The Prometheus monitoring tool will no longer be able to query these indicators, and the resource quotas occupied by the individual and group of the task will be restored accordingly.

[0013] A cluster resource quota management system, comprising:

[0014] The database module is used to store the group resource quota of each group, the personal resource quota of each user's group, and the information of each task;

[0015] The task queue module is used to perform quota checks on the current tasks to be scheduled: determine the corresponding user number and group number based on the information of the current tasks to be scheduled, and determine the remaining resource quotas of the individual and group based on the corresponding personal resource quotas and group resource quotas as well as the information obtained through the Prometheus monitoring tool. If the remaining quotas of the individual and group meet the resource requirements of the current tasks to be scheduled, the quota check is passed; push all custom indicators of the task to the Pushgateway, and call the application programming interface to create the corresponding Pod. After the Pod is scheduled by K8s, it will allocate resources from the resource pool and run; each task generates multiple custom indicators, each custom indicator corresponds to a resource type, and the generated multiple custom indicators have several identical labels to identify the basic information of the task, and are configured with different resource type labels. The resource requirements of the task are identified by all resource type labels;

[0016] The Pod Run module is used to instantiate tasks that pass the quota check into Pods, allocate corresponding resources to the Pods from the cluster resource pool, and is responsible for the creation, scheduling, operation, and termination of the Pods;

[0017] Pushgateway, used to store custom metrics for tasks;

[0018] Prometheus monitoring tool, used to periodically collect all custom metrics in Pushgateway, query and aggregate metric information during quota checking, and calculate changes in resource usage for each user and group;

[0019] The Informer module is used to monitor the status changes of all Pods at all times. When the Pod corresponding to the task is completed normally, fails, or is deleted, the resources occupied by the Pod will be released, and the callback function will be triggered to delete all custom indicators of the Pod's corresponding task in Pushgateway. The Prometheus monitoring tool will no longer be able to query these indicators, and the resource quotas occupied by the individual and group of the task will be restored accordingly.

[0020] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0022] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0023] It can be seen from the technical solution provided by the present invention described above that the introduction of custom indicators to mark resource quantities and thus implement quota management solves the problem of setting resource quotas at multiple levels (groups, individuals), as well as the problem of setting resource quotas for different types of GPUs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of a cluster resource quota management method provided by an embodiment of the present invention;

[0026] Figure 2An exemplary schematic diagram of a cluster resource quota management method provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of a cluster resource quota management system provided by an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0030] First, the following terms may be used in this article:

[0031] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0032] The term "consisting of" excludes any technical features not explicitly listed. If used in a claim, this term renders the claim closed, excluding any technical features other than those explicitly listed, except for conventional impurities associated with them. If this term appears only in a clause of a claim, it limits only the elements explicitly listed in that clause; elements listed in other clauses are not excluded from the claim as a whole.

[0033] The cluster resource quota management method, system, device, and storage medium provided by the present invention are described in detail below. Any content not described in detail in the embodiments of the present invention belongs to the prior art known to professionals in the field. Where specific conditions are not specified in the embodiments of the present invention, the process is carried out in accordance with conventional conditions in the art or the conditions recommended by the manufacturer. Where the manufacturer of the reagents or instruments used in the embodiments of the present invention is not specified, they are all conventional products that can be purchased commercially.

[0034] Example 1

[0035] An embodiment of the present invention provides a cluster resource quota management method, which mainly includes:

[0036] 1. Store the group resource quota of each group, the personal resource quota of each user's group, and the information of each task.

[0037] In the embodiment of the present invention, the group resource quota of each group is the resource limit of each group. The total resource usage of all users in the group cannot exceed the group resource quota of the group. When a new group is created, a corresponding record is added, including the group number and group resource quota.

[0038] In this embodiment of the present invention, the personal resource quota is the resource limit for a user within their group. The total resource usage of a user within their group cannot exceed the personal resource quota. When each user joins a group, a corresponding record is created, including the user ID, group ID, and personal resource quota. Each user can join one or more groups, and the user's personal resource quota within each group is independent of each other.

[0039] In the embodiment of the present invention, both the group resource quota and the personal resource quota impose resource restrictions on storage, memory, CPU, and GPU resources, and for GPU resources, resource restrictions are also imposed on different types of GPU resources.

[0040] In an embodiment of the present invention, when a user submits a task, a corresponding record is added, including the task number, user number, group number, storage size requirement, memory size requirement, CPU quantity requirement, GPU type and corresponding GPU quantity requirement.

[0041] 2. Perform a quota check on the currently scheduled task: Determine the corresponding user ID and group ID based on the information of the currently scheduled task, and determine the remaining resource quotas for the individual and group based on the corresponding individual resource quotas and group resource quotas as well as the information obtained through the Prometheus monitoring tool. If the remaining individual and group quotas meet the resource requirements of the currently scheduled task, then perform a quota check and use the task information to generate the corresponding custom indicators. Then, push all custom indicators corresponding to the task to the Pushgateway, and call the application programming interface to create the corresponding Pod. After the Pod is scheduled by K8s, it will allocate resources from the resource pool and run. Each task generates multiple custom indicators, and each custom indicator corresponds to a resource type. These custom indicators have some common labels to identify the basic information of the task, and are configured with different resource type labels. The resource requirements of the task are identified through all resource type labels.

[0042] In an embodiment of the present invention, the detailed process of quota checking is as follows: determining the corresponding user number and group number based on the information of the current task to be scheduled, obtaining the corresponding personal resource quota and group resource quota from the storage; and, based on the Prometheus monitoring tool, querying the resource usage of the corresponding user in the corresponding group and the resource usage of the group, calculating the personal and group remaining quotas in combination with the personal resource quota and the group resource quota; if both the personal and group remaining quotas meet the resource requirements of the current task to be scheduled, the quota check is passed; otherwise, the current task to be scheduled is skipped, and the next task to be scheduled is taken out for quota checking.

[0043] In an embodiment of the present invention, Pushgateway is used to store custom indicators of tasks. The Prometheus monitoring tool regularly collects all custom indicators in Pushgateway for querying and aggregating indicator information during the quota checking phase, and for calculating changes in resource usage of each user and each group.

[0044] Those skilled in the art will understand that Pushgateway is a component in the Prometheus ecosystem that allows for the push of custom metrics to Prometheus monitoring tools. A pod is the smallest deployable computing unit that can be created and managed in Kubernetes. It contains one or more containers and is used to run user tasks. A resource pool is a collection of all node resources in Kubernetes.

[0045] In an embodiment of the present invention, after a task passes the quota check, four custom indicators will be generated; the same tags in the four custom indicators include the task number, user number and group number; the resource type tags of these indicators are different, corresponding to storage, memory, CPU and GPU resources respectively, and the values ​​of the custom indicators are used to represent the demand for the corresponding resource type; the custom indicator of GPU resources is also marked separately with a card-type tag.

[0046] 3. K8s's Informer constantly monitors the status changes of all Pods. When the Pod corresponding to the task completes normally, fails, or is deleted (in these cases, the resources occupied by the Pod will be released), the callback function is triggered to delete all custom indicators of the task corresponding to the Pod. The Prometheus monitoring tool will no longer be able to query these deleted indicators at the next moment, and the resource quotas occupied by the individual and group for the task will be restored accordingly.

[0047] The above solution provided by the embodiment of the present invention introduces custom indicators to mark resource quantities and thus implement quota management, thereby performing resource restrictions at multiple levels (groups, individuals), and performing resource restrictions on different types of GPUs, thereby achieving efficient resource quota management.

[0048] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail below with reference to specific embodiments.

[0049] 1. Overall introduction of the plan.

[0050] Prometheus is a very important monitoring tool in the K8s cluster. It is easy to deploy and can automatically collect a large amount of default indicator data from the K8s cluster, such as the total resource amount of the node, the resource utilization of the Pod, etc. Pushgateway is a component in the Prometheus ecosystem, which allows custom indicator data to be pushed to the Prometheus monitoring tool. Informer, as the core toolkit of K8s, can monitor events and trigger callback functions to implement complex business logic. The present invention can solve the quota management problem of K8s cluster in multiple levels (groups, users) and multi-type GPU scenarios by uploading custom quota indicators through the Prometheus monitoring tool.

[0051] In an embodiment of the present invention, a new quota management method is designed based on the custom indicators of the Prometheus monitoring tool. The present invention uses custom indicators to record the resource usage of tasks, such as storage, memory, CPU, and GPU resources. To address the issue of multi-dimensional resource restrictions (group, individual), group and user tags can be added to the custom indicators. To address the issue of implementing resource restrictions for different types of GPUs, GPU type tags can be added to the custom indicators. After a user's task passes the quota check, the present invention uploads the corresponding custom indicators to the Pushgateway. Leveraging the powerful indicator aggregation and query capabilities of the Prometheus monitoring tool, the present invention can easily aggregate the usage of various resources (including different types of GPU resources) by users and groups from a large number of custom indicators. Combined with the total quotas of users and groups stored in the database, the remaining quotas of users and groups can be calculated. In this way, when processing the next task, it is possible to quickly determine whether the remaining quota meets the resource requirements of the task and decide whether to run the task. After the task passes the quota check, the corresponding Pod will be created, scheduled, allocated resources, and run. The built-in Informer tool in K8s can constantly monitor Pod status changes. When it detects that a Pod has completed normally, failed, or been deleted (in these cases, the resources occupied by the Pod will be released), it can accordingly delete the custom indicators corresponding to the task in Pushgateway, thereby restoring the individual and group quotas occupied by the task. Based on the above ideas, the present invention has designed a final quota management method.

[0052] 2. Detailed introduction of the plan.

[0053] 1. Resource allocation quota design.

[0054] The embodiments of the present invention mainly involve two types of resource quotas: group resource quotas and personal resource quotas.

[0055] Group resource quota: Each business group in a Kubernetes cluster has a group resource quota, which indicates that the total resource usage of all users within the group cannot exceed the group resource quota. Group resource quota information is stored in the group database, and each group has a corresponding group resource quota.

[0056] Personal resource quota: When each user joins a group, a record is added to the group member database. This record includes the user ID, group ID, and a resource quota, which represents the user's personal resource quota within the group. The user's total resource usage within the group cannot exceed the personal resource quota, thus limiting the user. The same user can join multiple groups, and the user's personal resource quota in each group is independent of each other.

[0057] Resource quota example: Resource quotas include storage, memory, CPU, and GPU. Different types of GPUs need to be subdivided. Figure 1 As shown, the resource quota example designed by the present invention is as follows: {"gpu":{"A100":50,"3090":50},"cpu":1000,"memory":1T,"storage":100T}. ​​This quota indicates that the GPU resources include two types: one is NVIDIA A100, whose resource limit is 50, and the other is NVIDIA GTX 3090, whose resource limit is 50. In addition, the CPU resource limit is 1000, the memory limit is 1T, and the storage limit is 100T. Of course, GPU resources of other card types can be included according to actual conditions.

[0058] Task information example: Task information includes task number, user number, group number, storage size requirement, memory size requirement, CPU quantity requirement, GPU type and corresponding GPU quantity requirement. Figure 1 As shown in the figure, a task example is provided. In this example, the task number is 01, the user number is 01, the group number to which the user belongs is 01, and the resource requirements include 500G storage, 320G memory, 40 CPUs, and 4 GPUs, where the GPU type is NVIDIA A100.

[0059] like Figure 1As shown, the above resource quotas and task information are all stored in the database, which includes a task database, a member database, and a group database. Among them, the task database stores task information, the member database stores the relationship between groups and users and the resource quotas of users in the group, and the group database stores group information and group resource quotas.

[0060] 2. Custom indicator design.

[0061] Custom indicator design: as attached Figure 1 After the task passes the quota check, four custom metrics can be generated. The names of these four metrics are the same, "my_metric". These four metrics have some common tags, which are used to store basic information about the task, such as task number, user number, and group number. The resource type tags of these four metrics are different, namely storage, memory, CPU and GPU. The value of the custom metric is the demand for the corresponding resource type. In order to distinguish different types of GPUs, the GPU tag needs to be used together with the card type tag to represent a specific type of GPU resource. The custom metrics generated by the task will be uploaded to Pushgateway.

[0062] Custom Metric Query: The Prometheus monitoring tool regularly collects all custom metric data in Pushgateway. Prometheus provides a dedicated PromQL framework for aggregating and querying metric data. The sum statement in PromQL adds metric values, and the group by statement groups metrics by label. The following examples illustrate how to use PromQL with the custom metric my_metric. For example, if you want to query the CPU quota used by user 01, you can use the following query statement: sum(my_metric{resource="cpu",user="01"}), and the possible output result is 24, indicating that user 01 currently uses 24 CPUs. If you want to query the NVIDIA A100 model GPU quota used by group 01, you can use: sum(my_metric{resource="gpu",group="01",gpu_type="A100"}), and the possible output result is 8, indicating that group 01 currently uses 8 NVIDIA A100 type GPUs. If you want to query the quota of various types of GPUs currently used by user 01, you can use: sum by(gpu_type)(my_metric{resource="gpu",user="01"}), and the possible output results are {gpu_type="A100"}:4, {gpu_type="3090"}:2, indicating that user 01 currently uses 4 NVIDIA A100 type GPUs and 2 NVIDIA GTX 3090 type GPU; if you want to query the GPU quota used by user 01 in group 01, you can use: sum(my_metric{resource="gpu",user="01",group="01"}), and the possible output result is 12, indicating that user 01 currently uses 12 CPUs in group 01. In addition, the Prometheus monitoring tool uses a time series database to store data, so it can record the historical changes in indicator values, such as Figure 2 The following figure shows the results of a PromQL query on a real cluster. The query statement is sum(my_metric{resource="gpu",user="14",gpu_type="3090"}). The query results are displayed in a time series chart, showing the changes in user 14's NVIDIA GTX 3090 GPU usage over the past week. Depending on the actual situation, the Prometheus monitoring tool can be adjusted to retain metric data for the past few months. With the aggregation and query functions of the Prometheus monitoring tool, it is very convenient to obtain the usage and historical changes of various resources for any user and group.

[0063] 3. Overall process.

[0064] Based on the introductions in parts 1 and 2 above, the following describes the overall process using the process of scheduling a single task as an example.

[0065] First, the database stores resource quotas and custom indicators for tasks, and all queued tasks form a task queue.

[0066] Afterwards, the tasks to be scheduled are taken out of the task queue in turn for quota checking. During the quota checking phase, the individual resource quota and group resource quota are retrieved from the member database and group database respectively based on the user ID and group ID in the task information. Then, the Prometheus monitoring tool is used to query the user's individual resource usage in the group and the group's resource usage, including storage, memory, CPU, and various types of GPU resources. Then, the remaining quotas for the individual and group are calculated separately based on the total quota information for the individual and group. The remaining quota is compared with the resource requirements in the task information. If the remaining resources of both the individual and group meet the requirements, the task passes the quota check. Otherwise, the current task to be scheduled is skipped and the next task is taken for checking.

[0067] When the task passes the quota check, the task information is used to generate the corresponding four custom indicators and push them to Pushgateway. Then, the K8s API is called to create a corresponding Pod for the task. After the Pod is scheduled by K8s, resources are allocated from the resource pool and run. The Informer component of K8s can monitor the status changes of the Pod. When the Pod corresponding to the task is completed normally, fails, or is deleted (in these cases, the resources occupied by the Pod will be released), the Informer callback function will be triggered. In the callback function, the four custom indicators of the task corresponding to the Pod will be deleted from Pushgateway, the Prometheus monitoring tool will no longer be able to query these indicators, and the personal and group quotas occupied by the task will be restored accordingly.

[0068] Since the Prometheus monitoring tool can be set to collect data in the Pushgateway every second, the present invention needs to wait for at least one second before processing the next task to ensure that the custom indicators of the previous task that passed the quota check are collected by the Prometheus monitoring tool.

[0069] The above-mentioned solution provided by the present invention in real time: (1) It makes full use of the Prometheus monitoring tool and the Informer monitoring mechanism, and designs a new quota management system based on the custom indicator upload of the Prometheus monitoring tool, which solves the problem of setting resource quotas at multiple dimensions (groups, individuals), as well as the problem of setting resource quotas for different types of GPUs, which is far superior to the existing technology. (2) The time series database of the Prometheus monitoring tool can record the time series change information of the quotas used by users and groups. Therefore, the present invention can obtain the changes in the amount of resources used by users and groups in the recent period, thereby more effectively monitoring the use of cluster resources. (3) The present invention is easy to expand and can support quota management of more types of GPUs.

[0070] In order to verify the effect of the present invention, a test was carried out on a real K8s cluster. The cluster contains 70 nodes, a total of 300 GPUs, 4,000-core CPUs and 80T of memory, of which 5 types of GPUs are included. The cluster is used by 15 groups at the same time, with a total of 90 users, and each group has about 5 to 10 people. After applying the present invention, the average daily number of tasks running in the cluster reached 100, and the average daily number of custom quota indicators generated reached 400. The resource quota of each group and user in the cluster has been effectively managed, and different types of GPU resources can be effectively distinguished. In addition, the cluster also supports viewing the changes in resource usage of each group and user in the past two weeks. Through the above tests, the actual effect of the present invention has been fully verified.

[0071] Through the description of the above embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented through software or by using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) and includes a number of instructions for causing a computer device (such as a personal computer, a server, or a network device) to execute the methods described in the various embodiments of the present invention.

[0072] Example 2

[0073] The present invention also provides a cluster resource quota management system, which is mainly used to implement the method provided in the above embodiment, such as Figure 3 As shown, the system mainly includes:

[0074] The database module is used to store the group resource quota of each group, the personal resource quota of each user's group, and the information of each task;

[0075] The task queue module is used to perform quota checks on the currently scheduled tasks: the corresponding user ID and group ID are determined based on the information of the currently scheduled tasks, and the corresponding personal resource quotas and group resource quotas are combined with the information obtained through the Prometheus monitoring tool to determine the remaining resource quotas of the individual and group. If the remaining quotas of the individual and group meet the resource requirements of the currently scheduled tasks, the quota check is passed; all custom indicators of the task are pushed to the Pushgateway, and the application programming interface is called to create the corresponding Pod. After the Pod is scheduled by K8s, resources are allocated from the resource pool and run; each task generates multiple custom indicators, each custom indicator corresponds to a resource type, and these custom indicators have some common labels to identify the basic information of the task, and are configured with different resource type labels. The resource requirements of the task are identified by all resource type labels;

[0076] The Pod Run module instantiates tasks that pass quota checks into Pods, allocates resources to Pods from the cluster resource pool, and is responsible for creating, scheduling, running, and terminating Pods. This is a built-in program module in the K8s cluster.

[0077] Pushgateway, used to store custom metrics for tasks;

[0078] Prometheus monitoring tool, used to periodically collect all custom metrics in Pushgateway, query and aggregate metric information during quota checking, and calculate changes in resource usage for each user and group;

[0079] The Informer module is used to monitor the status changes of all Pods at all times. When the Pod corresponding to the task is completed normally, fails, or is deleted (in these cases, the resources occupied by the Pod will be released), the callback function is triggered to delete all custom indicators of the Pod corresponding to the task in Pushgateway. The Prometheus monitoring tool will no longer be able to query these deleted indicators, and the resource quotas occupied by the individual and group of the task will be restored accordingly.

[0080] Considering that the main technical details involved in the above system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0081] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0082] Example 3

[0083] The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0084] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0085] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example:

[0086] The input device can be a touch screen, image acquisition device, physical button or mouse;

[0087] The output device may be a display terminal;

[0088] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0089] Example 4

[0090] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0091] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0092] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A cluster resource quota management method, characterized in that: include: Stores the group resource quota of each group, the personal resource quota of each user's group, and information about each task; For the tasks currently to be scheduled, a quota check is performed: the corresponding user number and group number are determined based on the information of the tasks currently to be scheduled, and the remaining resource quotas of the individual and group are determined in combination with the corresponding personal resource quotas and group resource quotas as well as the information obtained through the Prometheus monitoring tool. If the remaining quotas of the individual and group meet the resource requirements of the tasks currently to be scheduled, the corresponding custom indicators are generated using the task information through quota checking; all custom indicators of the task are pushed to the Pushgateway, and the application programming interface is called to create the corresponding Pod. After the Pod is scheduled by K8s, resources are allocated from the resource pool and run; each task generates multiple custom indicators, each custom indicator corresponds to a resource type, and the multiple custom indicators generated have several identical labels for identifying the basic information of the task, and are configured with different resource type labels. The resource requirements of the task are identified through all resource type labels; Monitor the status changes of all Pods at all times. When the Pod corresponding to the task is completed normally, fails, or is deleted, the resources occupied by the Pod will be released, and the callback function will be triggered to delete all custom indicators of the Pod's corresponding task in Pushgateway. The Prometheus monitoring tool will no longer be able to query these indicators, and the resource quotas occupied by the individual and group of the task will be restored accordingly.

2. A cluster resource quota management method according to claim 1, characterized in that: The group resource quota of each group is the resource limit of each group, and the total resource usage of all users in the group cannot exceed the group resource quota of the group to which they belong.

3. A cluster resource quota management method according to claim 1, characterized in that: After each user joins a group, a corresponding record is added, including: user number, group number and personal resource quota. The personal resource quota represents the resource limit of the user in the group to which he belongs. Each user can join one or more groups, and the personal resource quota of the user in each group is independent of each other.

4. A cluster resource quota management method according to claim 1, 2 or 3, characterized in that: Both group resource quotas and personal resource quotas set resource limits for storage, memory, CPU, and GPU resources. Furthermore, for GPU resources, resource limits are set for different GPU types.

5. A cluster resource quota management method according to claim 1, characterized in that: Each task generates four custom metrics, one for each of storage, memory, CPU, and GPU resources; The common labels in the four custom indicators include task number, user number, and group number. The resource type labels of the custom indicators are different, namely storage, memory, CPU, and GPU resources. The value of the custom indicator represents the demand for the corresponding resource type. The custom indicator of GPU resources is also marked with a separate card-type label.

6. A cluster resource quota management method according to claim 1, characterized in that: During quota checking, the corresponding user ID and group ID are determined based on the information of the currently scheduled task, and the corresponding personal resource quota and group resource quota are obtained from the storage; Furthermore, based on the Prometheus monitoring tool, the user's resource usage in the corresponding group and the group's resource usage are queried, and the remaining quotas for the individual and group are calculated by combining the individual resource quota and the group resource quota. If both the individual and group remaining quotas meet the resource requirements of the current task to be scheduled, the quota check passes; otherwise, the current task to be scheduled is skipped, and the next task to be scheduled is taken out for quota check.

7. A cluster resource quota management method according to claim 1 or 6, characterized in that: The Prometheus monitoring tool regularly collects all custom indicators in Pushgateway for querying and aggregating indicator information during the quota checking phase, and for calculating changes in resource usage for each user and group.

8. A cluster resource quota management system, characterized in that: include: The database module is used to store the group resource quota of each group, the personal resource quota of each user's group, and the information of each task; The task queue module is used to perform quota checks on the current tasks to be scheduled: determine the corresponding user number and group number based on the information of the current tasks to be scheduled, and determine the remaining resource quotas of the individual and group based on the corresponding personal resource quotas and group resource quotas as well as the information obtained through the Prometheus monitoring tool. If the remaining quotas of the individual and group meet the resource requirements of the current tasks to be scheduled, the quota check is passed; push all custom indicators of the task to the Pushgateway, and call the application programming interface to create the corresponding Pod. After the Pod is scheduled by K8s, it will allocate resources from the resource pool and run; each task generates multiple custom indicators, each custom indicator corresponds to a resource type, and the generated multiple custom indicators have several identical labels to identify the basic information of the task, and are configured with different resource type labels. The resource requirements of the task are identified by all resource type labels; The Pod Run module is used to instantiate tasks that pass the quota check into Pods, allocate corresponding resources to the Pods from the cluster resource pool, and is responsible for the creation, scheduling, operation, and termination of the Pods; Pushgateway, used to store custom metrics for tasks; Prometheus monitoring tool, used to periodically collect all custom metrics in Pushgateway, query and aggregate metric information during quota checking, and calculate changes in resource usage for each user and group; The Informer module is used to monitor the status changes of all Pods at all times. When the Pod corresponding to the task is completed normally, fails, or is deleted, the resources occupied by the Pod will be released, and the callback function will be triggered to delete all custom indicators of the Pod's corresponding task in Pushgateway. The Prometheus monitoring tool will no longer be able to query these indicators, and the resource quotas occupied by the individual and group of the task will be restored accordingly.

9. A processing device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images