Method for allocating computing resources and computing equipment
By coordinating the node switching between management and resource pools, the problem of low utilization caused by independent management of AI and HPC computing resources is solved, efficient sharing and flexible switching of AI and HPC resources is achieved, and the overall resource utilization is improved.
Patent Information
- Application Number
- CN202411137947.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, the independent management of AI and HPC computing resources leads to low utilization of computing resources, and it is impossible to solve the problem of insufficient HPC resources when the AI computing node resources are sufficient, and vice versa.
By obtaining the computing nodes in the resource pool to be allocated, allocating them to the AI resource pool or HPC resource pool according to the requirements of the computing service, and determining and closing or recycling running tasks during node switching, ensuring that the node status changes from inactive to active, realizing that the implementation of Coordinated management and resource sharing of AI and HPC resource pools.
The overall utilization rate of AI and HPC resources is improved, and the node switching between AI and HPC resource pools is achieved, and the system's resource management efficiency is improved.
Smart Images

Figure CN119025276A_ABST
Abstract
Description
Technical Field
[0001] The present specification relates to the field of communication technology, and in particular to a method and computing device for allocating computing resources. Background Art
[0002] AI: Artificial Intelligence, stands for artificial intelligence.
[0003] HPC: High Performance Computing stands for high performance computing.
[0004] K8s: Kubernetes, an open source container orchestration platform for automating the deployment, scaling, and management of containerized applications.
[0005] Slurm: Simple Linux Utility for Resource Management is an open source cluster resource management and job scheduling system.
[0006] In recent years, with the rise of AI model training and the deepening of the concept of AI for science, AI models are playing an increasingly important role in HPC application fields such as materials science, life science, atmosphere and ocean, so the demand for the integration of AI and HPC is becoming stronger. On the one hand, the integration of AI and HPC can not only apply the capabilities of AI to HPC applications to assist scientific computing, but also improve the utilization rate of computing resources.
[0007] Due to the large differences in the use and management of computing resources between AI and HPC clusters, major manufacturers currently manage AI and HPC computing resource pools independently, but this management method is bound to reduce the overall computing resource utilization. For example, when AI computing nodes have sufficient resources, the problem of insufficient HPC resources cannot be solved in time, and vice versa. Summary of the invention
[0008] To overcome the problems existing in the related art, this specification provides a method and a computing device for allocating computing resources.
[0009] According to a first aspect of an embodiment of this specification, a method for allocating computing resources is provided, the method comprising:
[0010] Obtain a resource pool to be allocated, where the resource pool to be allocated includes a number of computing nodes;
[0011] According to computing service requirements, the target computing nodes are obtained from the resource pool to be allocated and allocated to the AI resource pool or HPC resource pool;
[0012] Among them, the target computing node is configured with a K8s service component or a Slum service component.
[0013] The computing nodes in the resource pool to be allocated have the same OS version and kernel.
[0014] Among them, the service of each computing node in the resource pool to be allocated is in an inactive state, and the service of each computing node includes: K8s service and Slum service.
[0015] The step of acquiring a target computing node from a resource pool to be allocated and allocating it to an AI resource pool or an HPC resource pool according to computing service requirements includes:
[0016] Determine the number of computing nodes and service types required for the computing service, where the service types include AI resource types or HPC resource types;
[0017] Obtain a computing node that meets the service type and the number of computing nodes as the target computing node, and change the state of the target computing node from the inactive state to the active state, thereby obtaining the target computing node from the resource pool to be allocated and allocating it to the AI resource pool or the HPC resource pool.
[0018] Wherein, the method further comprises:
[0019] When it is identified that there are computing nodes in the AI resource pool that need to be recycled, it is determined whether there are any running tasks in the recycled computing nodes;
[0020] If yes, the user is notified to close the task, and the recycled computing node is recycled after the user closes the task;
[0021] If not, the recycled computing node is recycled.
[0022] The method further comprises:
[0023] When it is identified that there are computing nodes in the HPC resource pool that need to be recycled, it is determined whether there are any running tasks in the recycled computing nodes;
[0024] If yes, the user is notified to close the task, and the recycled computing node is recycled after the user closes the task;
[0025] If not, the recycled computing node is recycled.
[0026] The step of recycling the recycled computing node includes:
[0027] The state of the recycled computing node is set to inactive.
[0028] It can be seen from the above embodiments that by allocating computing nodes to the AI resource pool or the HPC resource pool through the resource pool to be allocated, the coordinated allocation of computing resources can be achieved in the AI and HPC fusion scenario, and the overall resource utilization of the system can be improved by switching nodes in the AI resource pool and the HPC resource pool.
[0029] According to a second aspect of an embodiment of this specification, a computing device is provided, the computing device comprising:
[0030] An acquisition module, used to acquire a resource pool to be allocated, wherein the resource pool to be allocated includes a plurality of computing nodes;
[0031] A processing module is used to obtain a target computing node from the resource pool to be allocated and allocate it to the AI resource pool or the HPC resource pool according to computing service requirements;
[0032] Among them, the target computing node is configured with a K8s service component or a Slum service component.
[0033] Wherein, each of the plurality of computing nodes in the resource pool to be allocated has the same OS version and kernel.
[0034] The service of each computing node in the resource pool to be allocated is in an inactive state, wherein the service of each computing node includes: K8s service and Slum service;
[0035] The processing module is used to determine the number of computing nodes and the service type required for the computing service, wherein the service type includes an AI resource type or an HPC resource type;
[0036] Obtain a computing node that meets the service type and the number of computing nodes as the target computing node, and change the state of the target computing node from the inactive state to the active state, thereby obtaining the target computing node from the resource pool to be allocated and allocating it to the AI resource pool or the HPC resource pool.
[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.
[0039] Figure 1 It is a flowchart of a method for allocating computing resources according to an exemplary embodiment of the present specification.
[0040] Figure 2It is a schematic diagram of an AI and HPC fusion resource management architecture shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION
[0041] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.
[0042] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0043] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0044] At present, all major manufacturers manage AI and HPC computing resource pools independently. AI resource pools are mainly based on K8s clusters to manage computing resources, while HPC resource pools are mainly based on Slurm to manage computing resources. K8s clusters natively support node-level scaling capabilities. By calling K8s interfaces, nodes can be added or deleted, which means scaling the resource pool. Slurm clusters support scaling cluster nodes by modifying slurm configurations.
[0045] The current Fusion Platform only deploys the management end of K8s and Slurm in one cluster, but the computing node resources in the K8s and Slurm clusters are still completely isolated. AI and HPC computing resources cannot be shared, resulting in low overall computing resource utilization in the system.
[0046] To solve the above technical problems, the present disclosure provides a method for allocating computing resources, such as Figure 1 As shown, the method includes:
[0047] S101 obtains a resource pool to be allocated, where the resource pool to be allocated includes a number of computing nodes;
[0048] S102 obtains a target computing node from the resource pool to be allocated and allocates it to the AI resource pool or the HPC resource pool according to the computing service demand;
[0049] Among them, the target computing node is configured with a K8s service component or a Slum service component.
[0050] In this embodiment, the system administrator can construct a resource pool to be allocated, which includes a number of nodes (computing nodes).
[0051] like Figure 2 As shown, the resource pool to be allocated includes nodes 1 to N. When the platform is initialized, the managed computing node resources can be added to the resource pool to be allocated. In order to avoid incompatibility problems caused by different OS (operating system) versions and kernels of the computing nodes, in this embodiment, the OS and kernel of each computing node in the resource pool to be allocated are consistent.
[0052] In this embodiment, each computing node in the resource pool to be allocated may be pre-configured with k8s and Slurm as well as service components such as a file system and user synchronization.
[0053] Assuming that there are computing nodes 1 to 100 in the resource pool to be allocated, computing nodes 1 to 50 can be configured with k8s service components, where the k8s service components may include: kubelet, kube-proxy, docker, etc., and computing nodes 51 to 100 can be configured with Slurm service components, where the Slurm service components may include: slurmd, slurmdbd, munge, etc.
[0054] In another example, the administrator can store k8s service components and Slurm service components in a storage medium, that is, the service components of the computing nodes are not configured in advance. When computing nodes need to be allocated to the AI resource pool or HPC resource pool according to computing requirements, the target computing nodes to be allocated are first determined, and then the k8s service components or Slurm service components are downloaded from the storage medium, and then the k8s service components or Slurm service components are configured for the target computing nodes.
[0055] In this embodiment, before the computing nodes in the resource pool to be allocated are allocated, the service status of each computing node is inactive. When the target computing node is determined according to the computing service requirements, the target computing node is started by changing the inactive status of the target computing node to active status.
[0056] It can be seen from the above embodiments that by configuring the resource pool to be allocated, it is possible to achieve coordinated management of the computing nodes of the AI resource pool and the HPC resource pool, and also to achieve integrated management of the computing nodes of the AI resource pool and the HPC resource pool.
[0057] In this embodiment, when it is identified that there are computing nodes that need to be recycled in the AI resource pool, it is determined whether there are any running tasks in the recycled computing nodes;
[0058] If yes, the user is notified to close the task, and the recycled computing node is recycled after the user closes the task;
[0059] If not, the recycled computing node is recycled.
[0060] Alternatively, when it is identified that there are computing nodes in the HPC resource pool that need to be recycled, it is determined whether there are any running tasks in the recycled computing nodes;
[0061] If yes, the user is notified to close the task, and the recycled computing node is recycled after the user closes the task;
[0062] If not, the recycled computing node is recycled.
[0063] As shown in the following example:
[0064] Scenario 1: The nodes in the AI resource pool are running normally and need to be switched to the HPC resource pool.
[0065] a: Check whether there are any running tasks in the AI computing node (the computing node to be recycled). If there are no tasks running, switch directly; if there are tasks running, the system administrator needs to confirm whether to force the switch, and prompt that forced switching will terminate the user task.
[0066] b: Set the node as unschedulable kubectl cordon NODE_NAME to ensure that new pods will not continue to be scheduled to this node. At the same time, remove the pods running on the current node. The scheduled task detects whether all pods have ended. The forced removal time can be set to 5 minutes by default. All pods that have not ended within 5 minutes are forcibly deleted. Execute kubectl delete pod POD_NAME --grace-period=0-force.
[0067] c: Remove the node from the k8s cluster. Shut down each service in AI-service in turn, systemctl stopAI-service, and wait until the service status is inactive. If the shutdown fails, try again three times and an error message will be displayed to facilitate the administrator to intervene and troubleshoot.
[0068] d: Start each service in HPC-service, systemctl start HPC-service, and wait until the service status is active. If the startup fails, retry 3 times. If it fails, the reason for the failure needs to be given to facilitate the administrator to troubleshoot.
[0069] e: Add the node to the Slurm cluster, modify the slurm.conf and node.conf files, and execute scontrolreconfigure and systemctl restart slurmctld to complete the node management.
[0070] f: Check whether the node is in a schedulable state. If not, set the node to be schedulable: scontrolupdate NodeName=node1 state=RESUME.
[0071] g: Complete the switch from AI node to HPC node.
[0072] In the above example, if the switch fails, a rollback operation needs to be performed according to the process. The node where the rollback fails is marked as abnormal, prompting the administrator to intervene and check the abnormal situation.
[0073] Scenario 2: The nodes in the HPC resource pool are running normally and need to be switched to the AI resource pool.
[0074] a: Check whether there are any running tasks in the HPC computing node. If no tasks are running, switch directly; if there are tasks running, the system administrator needs to confirm whether to force the switch, and prompt that forced switching will terminate the user task.
[0075] b2: Set the node to be unschedulable scontrol update NodeName = node1 state = DRAIN to ensure that new tasks will not continue to be scheduled to this node. At the same time, remove the tasks currently running on the current node. The scheduled task checks whether all tasks have been completed or canceled. The default cancellation start time is 5 minutes. Execute scancel<job_id> .
[0076] c: Remove the node from the Slurm cluster. Shut down each service in HPC-service in turn, systemctl stopHPC-service, and wait until all services are inactive. If the shutdown fails, retry 3 times and a prompt will be given.
[0077] d: Start each service in AI-service. Run systemctl start AI-service. Wait until the service status changes to active. If the startup fails, retry three times. If it fails, the reason for the failure needs to be given to facilitate the administrator to troubleshoot.
[0078] e: Add the node to the k8s cluster, call the k8s cluster API interface or execute kubeadm join<master_node_ip> :<master_node_port> --token <token>--discovery-token-ca-cert-hash<discovery_token> Complete node management.
[0079] f: Check whether the node is schedulable. If not, set the node to schedulable kubectl cordon<node_name> .
[0080] g: Complete the switch from HPC node to AI node.
[0081] Similarly, if the switch fails, a rollback operation needs to be performed according to the process. The node where the rollback failed is marked as abnormal, prompting the administrator to intervene and check the abnormal situation.
[0082] It can be seen from the above embodiments that the two computing resource pools in the AI and HPC fusion scenario can be switched on demand, and related services such as k8s and slurm can be flexibly started and stopped. By regularly detecting the status of each service and the schedulable status of the node, AI and HPC computing resources can be quickly switched, which improves the overall resource utilization of the system.
[0083] Based on the above method embodiments, the present disclosure further provides a computing device, the computing device comprising:
[0084] An acquisition module is used to acquire a resource pool to be allocated, where the resource pool to be allocated includes a number of computing nodes;
[0085] A processing module is used to obtain a target computing node from the resource pool to be allocated and allocate it to the AI resource pool or the HPC resource pool according to computing service requirements;
[0086] Among them, the target computing node is configured with a K8s service component or a Slum service component.
[0087] Wherein, each of the plurality of computing nodes in the resource pool to be allocated has the same OS version and kernel.
[0088] The service of each computing node in the resource pool to be allocated is in an inactive state, wherein the service of each computing node includes: K8s service and Slum service;
[0089] The processing module is used to determine the number of computing nodes and the service type required for the computing service, wherein the service type includes an AI resource type or an HPC resource type;
[0090] Obtain a computing node that meets the service type and the number of computing nodes as the target computing node, and change the state of the target computing node from the inactive state to the active state, thereby obtaining the target computing node from the resource pool to be allocated and allocating it to the AI resource pool or the HPC resource pool.
[0091] In this embodiment, the computing device may include: a server, a host, or other device equipped with a computing processor and storage.
[0092] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.
[0093] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0094] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification and examples are to be considered exemplary only, and the true scope and spirit of the specification are indicated by the following claims.
[0095] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.
[0096] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.< / token>
Claims
1. A method for allocating computing resources, It is characterized in that The method comprises: Obtain a resource pool to be allocated, where the resource pool to be allocated includes a number of computing nodes; According to computing service requirements, the target computing nodes are obtained from the resource pool to be allocated and allocated to the AI resource pool or HPC resource pool; Among them, the target computing node is configured with a K8s service component or a Slum service component.
2. The method according to claim 1, It is characterized in that The computing nodes in the resource pool to be allocated have the same OS version and kernel.
3. The method according to claim 1, It is characterized in that The service of each computing node in the resource pool to be allocated is in an inactive state, wherein the service of each computing node includes: K8s service and Slum service.
4. The method according to claim 3, It is characterized in that The step of obtaining a target computing node from a resource pool to be allocated and allocating it to an AI resource pool or an HPC resource pool according to computing service requirements includes: Determine the number of computing nodes and service types required for the computing service, where the service types include AI resource types or HPC resource types; Obtain a computing node that meets the service type and the number of computing nodes as the target computing node, and change the state of the target computing node from the inactive state to the active state, thereby obtaining the target computing node from the resource pool to be allocated and allocating it to the AI resource pool or the HPC resource pool.
5. The method according to claim 1, It is characterized in that The method further comprises: When it is identified that there are computing nodes in the AI resource pool that need to be recycled, it is determined whether there are any running tasks in the recycled computing nodes; If yes, the user is notified to close the task, and the recycled computing node is recycled after the user closes the task; If not, the recycled computing node is recycled.
6. The method according to claim 1, It is characterized in that The method further comprises: When it is identified that there are computing nodes in the HPC resource pool that need to be recycled, it is determined whether there are any running tasks in the recycled computing nodes; If yes, the user is notified to close the task, and the recycled computing node is recycled after the user closes the task; If not, the recycled computing node is recycled.
7. The method according to 5 or 6, It is characterized in that Recycling the recycled computing node includes: The state of the recycled computing node is set to inactive.
8. A computing device, It is characterized in that The computing device comprises: An acquisition module, used to acquire a resource pool to be allocated, wherein the resource pool to be allocated includes a plurality of computing nodes; A processing module is used to obtain a target computing node from the resource pool to be allocated and allocate it to the AI resource pool or the HPC resource pool according to computing service requirements; Among them, the target computing node is configured with a K8s service component or a Slum service component.
9. The device according to claim 8, It is characterized in that The computing nodes in the to-be-allocated resource pool have the same OS version and kernel.
10. The device according to claim 8, It is characterized in that The service of each computing node in the resource pool to be allocated is in an inactive state, wherein the service of each computing node includes: K8s service and Slum service; The processing module is used to determine the number of computing nodes and the service type required for the computing service, wherein the service type includes an AI resource type or an HPC resource type; Obtain a computing node that meets the service type and the number of computing nodes as the target computing node, and change the state of the target computing node from the inactive state to the active state, thereby obtaining the target computing node from the resource pool to be allocated and allocating it to the AI resource pool or the HPC resource pool.
