Method and system for dynamically switching computing power node between super computing cluster and intelligent computing cluster

Abstract

The invention discloses a method for dynamically switching computing power nodes between a super computing cluster and an intelligent computing cluster, and belongs to the technical field of computers. The method comprises the steps that a mode of constructing a shared resource pool is adopted, an intelligent computing cluster and a super computing cluster are respectively introduced into a super-plug component, and the components are responsible for monitoring and predicting the resource pressure of the clusters in real time and registering or applying for nodes to the shared resource pool according to the resource pressure condition. According to the method, resource allocation is optimized, the calculation efficiency is improved, task load fluctuation can be effectively dealt with, and deep fusion of super calculation and intelligent calculation is promoted. Therefore, the resource utilization rate is improved, the development of the intelligent computing technology is accelerated, and the innovation of various scientific and engineering applications is promoted.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for dynamically switching computing nodes between a supercomputing cluster and an intelligent computing cluster. Background Art

[0002] Supercomputing clusters and intelligent computing clusters are important development directions in today's computing field. Supercomputing clusters are widely used in complex computing tasks such as climate simulation, genomics, and materials science with their powerful computing capabilities. Through parallel computing and high-performance hardware, these clusters can process massive amounts of data in a short period of time, promoting scientific research and technological innovation. Intelligent computing clusters focus on artificial intelligence and machine learning tasks, combining deep learning frameworks and big data processing technologies to improve the efficiency of data analysis and model training. With the surge in data volume and the development of AI technology, intelligent computing clusters have gradually become an important infrastructure for enterprises and research institutions. They support intelligent decision-making, accurate prediction, and automated processing, and promote the digital transformation of various industries.

[0003] At present, the global competition in the field of supercomputing and intelligent computing is becoming increasingly fierce. Countries have increased their investment and strived to take the lead in technological innovation and application scenarios. my country has also made a series of progress in this regard and established a number of supercomputing and intelligent computing centers with international influence to meet the growing scientific research and commercial needs. In the future, the deep integration of supercomputing and intelligent computing will bring more opportunities for scientific research and social development, and promote the arrival of the intelligent era.

[0004] Supercomputing clusters generally use scheduling frameworks such as SLURM for resource management and task scheduling, and their programming models mostly use traditional parallel computing standards such as MPI and OpenMPI, while intelligent computing clusters generally use scheduling frameworks such as Kubernetes for resource management and task scheduling, and their programming models mostly use deep learning frameworks such as TensorFlow and PyTorch. This technical incompatibility makes it difficult for the two to share resources and perform collaborative computing.

[0005] As application requirements diversify, the nature of computing tasks is becoming more and more complex. Different tasks have significantly different requirements for computing resources. Some tasks may be more suitable for running in supercomputing clusters, while others perform better in intelligent computing clusters. The traditional static resource configuration model is stretched when faced with dynamically changing loads, often resulting in resource waste or computing delays. Therefore, it is particularly important to study a method and system for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for dynamically switching computing nodes between a supercomputing cluster and an intelligent computing cluster, aiming to compress idle nodes as much as possible by running a timing scheduler in the two clusters respectively, and add the idle nodes to a shared resource pool. When the load of a cluster increases and more resources are needed to run tasks, the required number of nodes are divided from the shared resource pool, and the additional nodes are dynamically added to the corresponding cluster to meet the needs of task operation. In this way, flexible scheduling and dynamic switching of computing nodes are achieved.

[0007] To solve the above technical problems, the present invention obtains the current resource allocation and the remaining allocatable resource amount of each node in the cluster;

[0008] Get the resource application amount of the tasks to be scheduled in the cluster;

[0009] The resource pressure of the cluster is calculated based on the current resource allocation, the remaining allocatable resources and the resource application amount; the resource pressure conditions include high pressure, normal pressure and low pressure;

[0010] For a cluster with low resource pressure, the idle nodes in the cluster are registered to the shared resource pool according to the optimized scheduling strategy;

[0011] For clusters with high resource pressure, the amount of resources that the cluster needs to increase is calculated based on the current resource allocation and resource application amount, and resources are applied for from the shared resource pool;

[0012] Based on the dynamic programming algorithm, the shared resource pool calculates the optimal node set according to the amount of resources that the cluster needs to increase;

[0013] The nodes in the optimal node set in the shared resource pool are added to the cluster whose corresponding resource pressure situation is high.

[0014] Preferably, the resource pressure of the cluster is calculated according to the current resource allocation, the remaining allocatable resource and the resource application amount, which specifically includes the following steps:

[0015] If the resource allocatable watermark of the cluster is less than or equal to 20%, the cluster's resource pressure is considered high;

[0016] Resource allocatable waterline = the sum of remaining allocatable resources of all nodes / the sum of resources of all nodes * 100%;

[0017] If the resource allocation prediction waterline is equal to 90%, the cluster's resource pressure is considered high;

[0018] Resource allocation prediction waterline = (the sum of current resource allocations of all nodes + the sum of resource requests of all tasks to be scheduled) / the sum of resources of all nodes * 100%;

[0019] If the resource allocation waterline of the cluster is less than 30% and the resource allocation forecast waterline is less than 50%, the resource pressure of the cluster is judged to be low;

[0020] Resource allocated waterline = the sum of current resource allocations of all nodes / the total resources of all nodes * 100%;

[0021] In other cases, the resource pressure of the cluster is considered normal.

[0022] Preferably, the optimization scheduling strategy is:

[0023] Use genetic algorithms to find the optimal scheduling strategy in clusters with low resource pressure:

[0024] The input of the genetic algorithm is all the current nodes of the cluster, all the task instances and the resource application amount of the instances, and the node where each task instance is located;

[0025] The objective function of the genetic algorithm is:

[0026] After scheduling, the number of nodes used is the minimum; denoted as NodeNumMin;

[0027] After scheduling, the sum of all node resources is the minimum; NodeResourceMin;

[0028] After scheduling, the number of task instances that need to be migrated is minimal; TaskTransferNumMin;

[0029] Objective function:

[0030] WNodeNumMin*NodeNumMin+WNodeResourceMin*NodeResourceMin+WTaskTransferNumMin*TaskTransferNumMin

[0031] Where: WNodeNumMin represents the minimum target weight of the number of nodes used;

[0032] WNodeResourceMin represents the minimum target weight of the total of all node resources;

[0033] WTaskTransferNumMin represents the minimum target weight of the task instance that needs to be transferred;

[0034] The sum of the target weights is 1;

[0035] Constraints of the genetic algorithm: If there are nodes in the cluster that cannot be scheduled or tasks that cannot be migrated, they can be used as constraints of the genetic algorithm and will be skipped and not calculated during the final scheduling.

[0036] Preferably, the calculation formula for the amount of resources that the cluster needs to increase is:

[0037] The amount of resources that need to be increased = (the sum of the current resource allocations of all nodes + the sum of resource requests of all tasks to be scheduled) * 90% - the sum of all node resources.

[0038] Preferably, the dynamic programming algorithm is:

[0039] Define the state:

[0040] Let dp[j] represent the total amount of resources j that can be achieved, where j ranges from 0 to target.

[0041] Initialize dp[0]=0, which means the total amount of resources is 0 when no node is selected.

[0042] State transfer:

[0043] For each node's resource amount resources[i], update the dp array.

[0044] Traverse j from back to front, from target to resources[i], and update dp[j]:

[0045] dp[j]=max(dp[j],dp[j-resources[i]]+resources[i])

[0046] Make sure each node is selected only once;

[0047] Final result:

[0048] Traverse the dp array and find the maximum value dp[j] such that j is less than or equal to target.

[0049] Preferably, adding the nodes in the optimal node set in the shared resource pool to the corresponding cluster with high resource pressure specifically includes the following steps:

[0050] First, get the cluster-related configuration based on the cluster ID;

[0051] After obtaining the cluster-related configuration, SSH to the node to configure the relevant files and start the relevant services, and add the nodes in the optimal node set in the shared resource pool to the corresponding cluster with high resource pressure.

[0052] Preferably, obtaining the current resource allocation and remaining allocatable resource amount of each node in the cluster specifically includes the following steps:

[0053] The scheduler-plugin component calls the Kubernetes API server or slurm to obtain the current resource allocation and remaining allocatable resources of each node in the cluster.

[0054] Preferably, obtaining the resource application amount of the to-be-scheduled tasks of the cluster specifically includes the following steps:

[0055] The scheduler-plugin component calls the Kubernetes API server or slurm to obtain the resource application amount of the task to be scheduled.

[0056] Preferably, the method further comprises the following steps:

[0057] Initialize the idle nodes registered to the shared resource pool.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. A shared resource pool is introduced as a shared resource pool for intelligent computing clusters and supercomputing clusters. Idle clusters can be placed in the shared resource pool. When resources are tight, resources can be applied from the contribution resource pool to join the cluster to meet the needs of task operation. This dynamic resource switching method can achieve efficient reuse of intelligent computing cluster and supercomputing cluster resources.

[0060] 2. Introduce the scheduler-plugin component in the intelligent computing cluster and supercomputing cluster to dynamically sense the resource pressure of the cluster, so as to determine whether the cluster should add or reduce resources; in the scenario of reducing resources, the optimization scheduler of the scheduler-plugin component uses genetic algorithms to select the optimal scheduling strategy from the cluster to ensure the optimal state of the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0062] Figure 1 It is an architectural diagram of a method for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters;

[0063] Figure 2 It is a flow chart for judging and handling cluster resource pressure;

[0064] Figure 3 Flowchart of the processing logic of the shared resource pool. DETAILED DESCRIPTION

[0065] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.

[0066] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0067] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0068] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0069] Embodiment 1: A method for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters is implemented by building a shared resource pool. The intelligent computing cluster and supercomputing cluster respectively introduce scheduler-plugin components, which are responsible for real-time monitoring and prediction of cluster resource pressure, and register or apply for nodes to the shared resource pool according to the resource pressure situation. The present invention can effectively cope with task load fluctuations and promote the deep integration of supercomputing and intelligent computing by optimizing resource allocation and improving computing efficiency. This not only improves resource utilization, but also helps to accelerate the development of intelligent computing technology and promote innovation in various scientific and engineering applications.

[0070] like Figure 1 As shown in the figure, a method and system architecture diagram for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters is shown. The specific logic is as follows:

[0071] (1) Deploy the scheduler-plugin component in the intelligent computing cluster and supercomputing cluster respectively. The scheduler-plugin component interacts with the cluster's scheduling system in real time to obtain the cluster's current resource pressure and predicted pressure to determine whether to add nodes or whether the current cluster can idle nodes.

[0072] (2) When the scheduler-plugin component determines that the cluster resource pressure is high and additional nodes are needed, it initiates a request to apply for a node from the pool-scheduler of the idle resource pool. When the scheduler-plugin component determines that the cluster resource pressure is low and there are idle nodes that can be vacated, it initiates a request to register a node from the pool-scheduler of the idle resource pool.

[0073] (3) When the pool-scheduler of the idle resource pool receives the application node, it searches for a suitable node in the shared resource pool according to the specifications and quantity of the application node, and installs the slurmd component or kubelet component to add the node to the corresponding cluster. When the pool-scheduler of the idle resource pool receives the registration node, the pool-scheduler first initializes the node to be registered, and then records the node information in the database;

[0074] (4) Through continuous cycles of the above three steps, dynamic switching of nodes between the intelligent computing cluster and the supercomputing cluster is realized, and the same batch of physical resources can support the scheduling and use of two clusters at the same time, ensuring efficient use of resources.

[0075] 1. Determining and handling cluster resource pressure

[0076] Deploy the scheduler-plugin component in the intelligent computing cluster and supercomputing cluster respectively. The scheduler-plugin component is a scheduled task that periodically determines the pressure level of the cluster and performs corresponding processing according to the pressure level. Figure 2The cluster resource pressure judgment and processing flow shown in the figure. The scheduler-plugin component is a scheduled task that is triggered periodically. The execution cycle of the scheduled task can be configured through the configuration file. The default value is 3 hours. When the execution cycle is small, more timely resources will be achieved, and there will not be a shortage of cluster resources or idle cluster resources for too long; however, a small execution cycle will cause frequent scheduling of tasks in the cluster, which will have a certain impact on task execution. Conversely, when the execution cycle is large, it will not cause frequent task migration and dynamic adjustment of scheduling sources, and the impact on tasks will be small, but there will be a shortage of cluster resources or idle cluster resources for too long. In actual practice, the setting size of the execution cycle can be determined according to the impact of frequent migration scheduling on the business. The specific logic of each execution cycle of the scheduler-plugin component is as follows:

[0077] (1) First, the scheduler-plugin component calls the Kubernetes API server or slurm to obtain cluster node details, mainly obtaining the cluster node list and the current resource allocation and remaining allocatable resource amount of each node;

[0078] (2) The scheduler-plugin component calls the Kubernetes API server or slurm to obtain the details of the tasks to be scheduled, mainly obtaining the resource application amount of the tasks to be scheduled in the cluster;

[0079] (3) The resource pressure of the cluster is judged based on the current resource allocation, remaining resources, and pending tasks of the cluster. Resource pressure is divided into three states: high pressure, normal pressure, and low pressure. The specific judgment logic is as follows:

[0080] ① If the resource allocatable waterline of the cluster is less than or equal to 20%, the cluster is judged to be under high pressure;

[0081] Resource allocatable waterline = the sum of remaining allocatable resources of all nodes / the sum of resources of all nodes * 100%;

[0082] ② If the resource allocation prediction waterline is equal to 90%, the cluster status is judged to be under great pressure;

[0083] Resource allocation prediction waterline = (the sum of current resource allocations of all nodes + the sum of resource requests of all tasks to be scheduled) / the sum of resources of all nodes * 100%;

[0084] ③ If the resource allocation waterline of the cluster is less than 30% and the resource allocation prediction waterline is less than 50%, the cluster status is judged to be low pressure;

[0085] Resource allocated waterline = the sum of current resource allocations of all nodes / the total resources of all nodes * 100%;

[0086] ④ In other cases, the cluster status is judged to be normal pressure;

[0087] Note: Cluster resources have many dimensions, such as CPU, memory, GPU / NPU, etc. Each resource dimension is calculated separately, and the final watermark value is taken from the maximum value of each dimension;

[0088] (4) When the cluster is in a normal pressure state, no processing is performed;

[0089] (5) When the cluster is under high pressure, calculate the amount of resources that the cluster needs to increase; the calculation logic is as follows:

[0090] The amount of resources to be increased = (the sum of the current resource allocation of all nodes + the sum of resource requests of all tasks to be scheduled) * 90% - the sum of all node resources

[0091] Each resource dimension is calculated separately;

[0092] (6) Call the resource application interface of pool-scheduler. The interface parameters include cluster ID and the requested resource amount (CPU resource amount, memory resource amount, GPU / NPU resource amount, etc.)

[0093] (7) When the cluster is in a low-pressure state, the optimization scheduler of the scheduler-plugin component is triggered to calculate the cluster's need for optimization scheduling strategy; the calculation uses a genetic algorithm to find the optimal scheduling strategy in the cluster; the calculation logic is as follows:

[0094] The input of the genetic algorithm is all the current nodes of the cluster, all the task instances and the resource application amount of the instances, and the node where each task instance is located;

[0095] The objective function of the genetic algorithm is:

[0096] After scheduling, the number of nodes used is the minimum; denoted as NodeNumMin;

[0097] After scheduling, the sum of all node resources is the minimum; NodeResourceMin;

[0098] After scheduling, the number of task instances that need to be migrated is minimal; TaskTransferNumMin;

[0099] Objective function: WNodeNumMin*NodeNumMin+WNodeResourceMin*NodeResourceMin+WTaskTransferNumMin*TaskTransferNumMin

[0100] WNodeNumMin represents the minimum target weight of the number of nodes used, and its default value is 0.25; WNodeResourceMin represents the minimum target weight of the total number of node resources, and its default value is 0.5;

[0101] WTaskTransferNumMin represents the minimum target weight of the task instance that needs to be migrated, and the default value is 0.25;

[0102] The default value can be adjusted according to the different latitudes of interest, and the sum can be 1;

[0103] Constraints of the genetic algorithm: If there are nodes in the cluster that cannot be scheduled or tasks that cannot be migrated, they can be used as constraints of the genetic algorithm. In this case, they will be skipped and not calculated during the final scheduling.

[0104] The scheduling strategy set is calculated by the above calculation method, that is,

[0105] [The XX task instance on the XX node is migrated to the XX node, the XX task instance on the XX node is migrated to the XX node, ...]

[0106] (8) Call the Kubernetes API server or Slurm to execute the scheduling decisions of the scheduling strategy set in step (7) one by one;

[0107] (9) After the scheduling decision is executed, the pool-scheduler registration node interface is called on the idle nodes in the cluster to register the idle nodes to the shared resource pool; the interface parameters are the node IP and node resource details (CPU, memory, GPU / NPU and other resource quantities);

[0108] 2. Processing Logic of Shared Resource Pool

[0109] The pool-scheduler component is the core component of the shared resource pool. It receives and processes node applications or node registration interfaces from the scheduler-plugin component in the supercomputing cluster or intelligent computing cluster. Figure 3 The processing logic of the shared resource pool shown is as follows:

[0110] (1) The pool-scheduler component has two interfaces: one is the node application interface and the other is the node registration interface. It receives node application or node registration requests from the scheduler-plugin component in the supercomputing cluster or intelligent computing cluster through HTTP.

[0111] (2) When pool-scheduler receives the node registration request, it first logs in to the node remotely using SSH according to the node's IP address, and then deletes the original cluster-related services and related information files on the node. For example, if the node originally belongs to an intelligent computing cluster, it will stop services such as Kubelet and Docker on the node, and delete files such as kubeconfig and docker-deamon.json; if the node originally belongs to a supercomputing cluster, it will stop the slurmd service on the node and delete slurmd-related files;

[0112] (3) pool-scheduler writes the information of the initialized node into the MySQL database table of the shared resource pool, mainly writing the node IP and the node resource amount;

[0113] (4) When the pool-scheduler receives a node application request, it first obtains information about all nodes in the shared resource pool from the MySQL database of the shared resource pool, including the node IP and the node's resource quantity;

[0114] (5) Then select the optimal node set from all nodes in the shared resource pool so that the total amount of resources in the node set is as close as possible to the requested amount of resources. In order to improve the computing efficiency, a dynamic programming algorithm is introduced to speed up the calculation. The specific logic is as follows:

[0115] ①Define the status:

[0116] Let dp[j] represent the total amount of resources j that can be achieved, where j ranges from 0 to target.

[0117] Initialize dp[0]=0, which means the total amount of resources is 0 when no node is selected.

[0118] ②State transfer:

[0119] For each node's resource amount resources[i], we need to update the dp array.

[0120] Traverse j from back to front (from target to resources[i]), and update dp[j]:

[0121] dp[j]=max(dp[j],dp[j-resources[i]]+resources[i])

[0122] This ensures that each node is selected only once.

[0123] ③Final result:

[0124] Traverse the dp array and find the maximum value dp[j] such that j is less than or equal to target.

[0125] (6) After obtaining the optimal node set in step 5, each node in the node set is added to the cluster of the applicant node one by one. The logic of adding is as follows:

[0126] ①First, obtain cluster-related configurations based on the cluster ID. These configurations are configured in the configuration file of the pool-scheduler service. The relevant configurations include the cluster type corresponding to the cluster ID (intelligent computing cluster or supercomputing cluster) and the cluster configuration information; for example, if it is an intelligent computing cluster, it is the Master IP, Kubeconfig, and image warehouse address of the intelligent computing cluster Kubernetes; if it is a supercomputing cluster, it is the supercomputing cluster Slurm connection address, etc.;

[0127] ②After obtaining the cluster-related configuration, SSH to the node to configure related files and start related services. For example, if it is an intelligent computing cluster, configure the Kubeconfig file and docker-deamon.json file, and then start the Kubelet service. In this way, the node will be successfully added to the intelligent computing cluster; if it is a supercomputing cluster, configure the slurm.conf file and start the Slurmd service. In this way, the node will be successfully added to the supercomputing cluster;

[0128] (7) Delete the nodes that have successfully joined the cluster from the MySQL database of the contribution resource pool to prevent duplicate addition;

[0129] The method of dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters in this embodiment has the following advantages:

[0130] 1. A method and system for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters has been applied in a certain efficient computing center. Through this method, the resource utilization rate of computing resources can be effectively improved by more than 10% and the cost can be saved by more than 20%;

[0131] 2. Through the application of a method and system for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters, it is possible to support the execution of larger-scale tasks without expanding the resources of intelligent computing clusters or supercomputing clusters, thereby achieving true cost reduction and efficiency improvement.

[0132] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules, modules or units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units, modules or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0133] The units may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0134] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0135] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present invention are executed. It should be noted that the above-mentioned computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above.

[0136] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0137] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for dynamically switching computing nodes between a supercomputing cluster and an intelligent computing cluster, characterized in that: include: Get the current resource allocation and remaining allocatable resources of each node in the cluster; Get the resource application amount of the tasks to be scheduled in the cluster; The resource pressure of the cluster is calculated based on the current resource allocation, the remaining allocatable resources and the resource application amount; the resource pressure conditions include high pressure, normal pressure and low pressure; For a cluster with low resource pressure, the idle nodes in the cluster are registered to the shared resource pool according to the optimized scheduling strategy; For clusters with high resource pressure, the amount of resources that the cluster needs to increase is calculated based on the current resource allocation and resource application amount, and resources are applied for from the shared resource pool; Based on the dynamic programming algorithm, the shared resource pool calculates the optimal node set according to the amount of resources that the cluster needs to increase; The nodes in the optimal node set in the shared resource pool are added to the cluster whose corresponding resource pressure situation is high.

2. The method for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters according to claim 1, characterized in that: The resource pressure of the cluster is calculated based on the current resource allocation, remaining allocatable resources, and resource application amount. The specific steps include: If the resource allocatable watermark of the cluster is less than or equal to 20%, the cluster's resource pressure is considered high; Resource allocatable waterline = the sum of remaining allocatable resources of all nodes / the sum of resources of all nodes * 100%; If the resource allocation prediction waterline is equal to 90%, the cluster's resource pressure is considered high; Resource allocation prediction waterline = (the sum of current resource allocations of all nodes + the sum of resource requests of all tasks to be scheduled) / the sum of resources of all nodes * 100%; If the resource allocation waterline of the cluster is less than 30% and the resource allocation forecast waterline is less than 50%, the resource pressure of the cluster is judged to be low; Resource allocated waterline = the sum of current resource allocations of all nodes / the total resources of all nodes * 100%; In other cases, the resource pressure of the cluster is considered normal.

3. The method for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters according to claim 2, characterized in that: The optimization scheduling strategy is: Use genetic algorithms to find the optimal scheduling strategy in clusters with low resource pressure: The input of the genetic algorithm is all the current nodes of the cluster, all the task instances and the resource application amount of the instances, and the node where each task instance is located; The objective function of the genetic algorithm is: After scheduling, the number of nodes used is the minimum, which is denoted as NodeNumMin; After scheduling, the sum of all node resources is the minimum; NodeResourceMin; After scheduling, the number of task instances that need to be migrated is minimal; TaskTransferNumMin; Objective function: WNodeNumMin*NodeNumMin+WNodeResourceMin*NodeResourceMin+WTaskTransferNumMin*TaskTransferNumMin Where: WNodeNumMin represents the minimum target weight of the number of nodes used; WNodeResourceMin represents the minimum target weight of the total of all node resources; WTaskTransferNumMin represents the minimum target weight of the task instance that needs to be transferred; The sum of the target weights is 1; Constraints of the genetic algorithm: If there are nodes in the cluster that cannot be scheduled or tasks that cannot be migrated, they can be used as constraints of the genetic algorithm and will be skipped and not calculated during the final scheduling.

4. The method for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters according to claim 3, characterized in that: The calculation formula for the amount of resources that the cluster needs to increase is: The amount of resources that need to be increased = (the sum of the current resource allocations of all nodes + the sum of resource requests of all tasks to be scheduled) * 90% - the sum of all node resources.

5. The method for dynamically switching computing nodes between a supercomputing cluster and an intelligent computing cluster according to claim 4, characterized in that: The dynamic programming algorithm is: Define the state: Let dp[j] represent the total amount of resources j that can be realized, where j ranges from 0 to target; Initialize dp[0]=0, which means the total amount of resources when no node is selected is 0; State transfer: For each node’s resource amount resources[i], update the dp array; Traverse j from back to front, from target to resources[i], and update dp[j]: dp[j]=max(dp[j],dp[j-resources[i]]+resources[i]) Make sure each node is selected only once; Final result: Traverse the dp array and find the maximum value dp[j] such that j is less than or equal to target.

6. The method for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters according to claim 5, characterized in that: Adding nodes in the optimal node set in the shared resource pool to the corresponding cluster with high resource pressure includes the following steps: First, get the cluster-related configuration based on the cluster ID; After obtaining the cluster-related configuration, SSH to the node to configure the relevant files and start the relevant services, and add the nodes in the optimal node set in the shared resource pool to the corresponding cluster with high resource pressure.

7. The method for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters according to claim 6, characterized in that: Get the current resource allocation and remaining allocatable resource amount of each node in the cluster, including the following steps: The scheduler-plugin component calls the Kubernetes API server or slurm to obtain the current resource allocation and remaining allocatable resources of each node in the cluster.

8. The method for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters according to claim 7, characterized in that: Obtain the resource application amount of the tasks to be scheduled in the cluster, including the following steps: The scheduler-plugin component calls the Kubernetes API server or slurm to obtain the resource application amount of the task to be scheduled.

9. The method for dynamically switching computing nodes between supercomputing clusters and intelligent computing clusters according to claim 8, characterized in that: The following steps are also included: Initialize the idle nodes registered to the shared resource pool.

10. A system for dynamically switching computing nodes between a supercomputing cluster and an intelligent computing cluster, used to implement the method for dynamically switching computing nodes between a supercomputing cluster and an intelligent computing cluster as described in any one of claims 1 to 9, characterized in that: include: The first acquisition module is used to obtain the current resource allocation and remaining allocatable resource amount of each node in the cluster; The second acquisition module is used to obtain the resource application amount of the tasks to be scheduled in the cluster; A resource pressure judgment module is used to calculate the resource pressure of the cluster according to the current resource allocation, the remaining allocatable resources and the resource application amount; the resource pressure conditions include high pressure, normal pressure and low pressure; A registration module is used to register idle nodes in a cluster with low resource pressure to a shared resource pool according to an optimized scheduling strategy; The application module is used to calculate the amount of resources that need to be increased for clusters with high resource pressure based on the current resource allocation and resource application amount, and apply for resources from the shared resource pool; The calculation module is used to calculate the optimal node set based on the dynamic programming algorithm and the shared resource pool according to the amount of resources that need to be added to the cluster; An adding module is used to add nodes in the optimal node set in the shared resource pool to a cluster with a high resource pressure.

Images