A distributed automatic expansion method in high performance computing

Abstract

The present application provides a distributed automatic expansion method in high-performance computing, including: predicting future computing resource requirements according to the task increase rate, and judging whether current resources can meet the requirements; applying for new nodes in advance when it is predicted that resources will be insufficient; monitoring the utilization rate of nodes, and releasing redundant nodes to avoid resource waste; judging the node operation status through heartbeat detection, and realizing real-time detection of faulty nodes; after discovering a fault, determining the order of task recovery of the faulty node according to the importance of calculation; recovering tasks from the fault point according to the priority, and repeating the detection until the task is completed; if the recovery time exceeds the specified time, terminating the recovery and releasing the node resources; when the amount of new tasks changes unevenly, evaluating the load status of each node; when high-performance computing is overloaded, forming a scheduling plan for tasks according to the load, and remapping the tasks to the nodes according to the scheduling plan to improve resource utilization efficiency.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a distributed automatic expansion method in high performance computing. Background Art

[0002] High-performance computing refers to the use of aggregated computing power to handle data-intensive computing tasks that cannot be completed by standard workstations, including simulation, modeling, and rendering. High-performance computing can effectively overcome the limitations of computing power by using specialized or high-end hardware or integrating the computing power of multiple units. However, with the increasing demand for computing resources, traditional computing resource management systems often face some problems. First, due to the uncertainty of task growth rate, it is difficult for traditional systems to accurately predict future resource requirements, which may lead to insufficient resources and affect task execution. Second, traditional systems often cannot detect and handle node failures in a timely manner, resulting in tasks that cannot be restored or take too long to restore. In addition, traditional systems often cannot effectively utilize idle resources and cannot dynamically adjust according to the load status of nodes, resulting in low resource utilization efficiency. Therefore, there is an urgent need for a distributed high-performance computing solution that can effectively improve computing resource utilization and self-heal. Summary of the invention

[0003] The present invention provides a distributed automatic expansion method in high performance computing, which mainly includes:

[0004] Predict future computing resource demands based on the rate at which tasks increase, and determine whether current resources can meet the demand. When it is predicted that resources will be insufficient, apply for new nodes in advance. Monitor node utilization and release redundant nodes to avoid resource waste. Determine the node operation status through heartbeat detection to achieve real-time detection of faulty nodes. After a fault is discovered, determine the order of task recovery for the faulty node based on the calculation importance. Recover tasks from the fault point based on priority, and repeat detection until the task is completed. If the recovery time exceeds the specified time, terminate the recovery and release node resources. When the amount of new tasks changes unevenly, evaluate the load status of each node. When high-performance computing is overloaded, form a scheduling plan for tasks based on the load, and remap tasks to nodes based on the scheduling plan to improve resource utilization efficiency.

[0005] As a preferred solution, predicting future computing resource requirements according to the task increase rate and judging whether current resources can meet the requirements include:

[0006] According to the historical task data of all nodes in the high-performance computing system, the task quantity, task type, and task completion time information are obtained to construct a historical task data set; a task increase rate prediction model is established according to the linear regression method, and the task increase rate prediction model is trained using the historical task data set until the accuracy of the task increase rate prediction model reaches the preset threshold, and the training is terminated to obtain a trained task increase rate prediction model; the task increase rate prediction model is used to predict real-time data to obtain the task growth rate; the required resource occupancy is calculated based on the task growth rate, task type, and average task completion time; the current resource capacity information is obtained to determine the available amount of each type of resource; it is determined whether the resource occupancy exceeds the resource capacity. When the resource occupancy exceeds the resource capacity, an application is made to add a distributed node.

[0007] As a preferred solution, when it is predicted that resources will be insufficient, applying for adding new nodes in advance includes:

[0008] Based on the predicted resource requirements, the configuration requirements of the new node are determined, including the number of CPU and GPU cores, memory size, and storage capacity; apply for new nodes from the distributed node resource pool, obtain resource pool information, and determine the types and quantities of resources that can be applied for at present. Resource types include GPU, CPU, and storage; according to resource requirements, task priority, and node configuration, tasks and nodes are sorted separately to obtain the task order and node order, and tasks are arranged in the distributed nodes in the node order according to the task order.

[0009] As a preferred solution, the utilization rate of the monitoring nodes and releasing redundant nodes to avoid resource waste include:

[0010] Obtain the resource utilization of each node and determine whether there are inefficient nodes; when the average resource utilization of a node is lower than the threshold, it is marked as an inefficient node; select the target node based on task requirements and other node load conditions, and migrate the tasks on the inefficient node to the target node; gradually migrate tasks until there are no active tasks running on the inefficient node; if the inefficient node still has task data or status information, place it in external storage; retract the inefficient node to the distributed node resource pool, and update the node list and resource overview information in high-performance computing.

[0011] As a preferred solution, the node operation status is judged by heartbeat detection to achieve real-time detection of faulty nodes, including:

[0012] A fixed heartbeat detection mechanism is set up, and heartbeat packets are sent regularly between the node and the monitoring center; the heartbeat response time and response status are obtained to determine whether the node is currently reachable; when the heartbeat does not respond for more than the preset number of times, the node status is determined to be abnormal and the connection is lost; according to the node connection information in the heartbeat packet and the time of the node's last response, the failure time point is calculated; the fault type is determined, including hardware failure, network failure, software failure, and permission failure; according to the importance of the node and the scope of the fault impact, the fault level is determined, which is divided into fatal failure, serious failure, and general failure; if it is determined to be a fatal level failure, a fault alarm is issued immediately; other levels of failures are logged, sorted according to the fault level, and enter the fault queue for processing.

[0013] As a preferred solution, after the fault is found, calculating the importance and determining the task recovery order of the faulty node includes:

[0014] For all running tasks, set key task identifiers and task weights, and the weights are determined according to the business line to which the tasks belong. Once a faulty node is found, obtain information on all affected tasks. Evaluate the task running progress and determine the cost and impact of task recalculation. Analyze task dependencies and determine subsequent related tasks that are directly dependent on the affected tasks. Calculate the comprehensive importance of tasks based on business weights and the degree of task loss. Determine the priority of task recovery based on task importance, and recover the most important tasks first. At the same time, obtain the complexity of task recalculation, and recover the tasks with the lowest complexity in advance.

[0015] As a preferred solution, the task is recovered from the failure point according to the priority, and the detection is repeated until the task is completed, including:

[0016] Check the resource status of the standby node to ensure that it meets the recovery task requirements; match tasks according to priority and available resources, and schedule the recovery of failed tasks; restart and resume tasks that can be directly restarted; set migration strategies and new node information for tasks that need to be migrated, and execute migration; monitor the recovery status of failed tasks in real time to detect whether the recovery is successful; if there are still tasks that fail to recover after all tasks have been recovered, re-formulate a scheduling plan until all tasks are recovered; detect whether the original failed node is repaired, and automatically schedule new tasks to the repaired node after the repair.

[0017] As a preferred solution, if the recovery time exceeds a limited time, the recovery is terminated and the node resources are released, including:

[0018] Set the task recovery deadline based on task priority and business attributes; monitor the duration and number of task recovery attempts; terminate the recovery when the cumulative recovery duration exceeds the task recovery deadline or the number of attempts exceeds the preset threshold; roll back the state of the rollbackable tasks that have terminated recovery to the last checkpoint before the failure; obtain the intermediate state and data of the non-rollbackable tasks and save them to external storage; release the failed node and return it to the distributed node resource pool.

[0019] As a preferred solution, when the amount of newly added tasks changes unevenly, evaluating the load status of each node includes:

[0020] Record the relevant characteristic information of the newly added tasks, including task type, required resources, and business attributes; collect real-time operation and load data of each computing node; monitor the CPU, memory, and disk utilization of each node. When the node load limit is exceeded, the node is considered to be overloaded; monitor the load changes of each node. When all nodes exceeding the preset threshold are overloaded, the high-performance computing is determined to be overloaded and expansion is triggered; when all nodes exceeding the preset threshold are overloaded, the high-performance computing is determined to be in a high-load state, and traffic control and frequency limiting strategies are initiated.

[0021] As a preferred solution, when high-performance computing is overloaded, a scheduling scheme is formed for tasks according to the load, and tasks are remapped to nodes according to the scheduling scheme to improve resource utilization efficiency, including:

[0022] Collect historical resource usage and historical task load data of each node in high-performance computing, and build a task load data set; establish a task load model based on autoregressive integrated moving average, and use the task load data set to train the task load model to obtain a trained task load model; input the current resource usage and task load data of each node into the task load model to evaluate the load contribution of each node in executing tasks; obtain the migration difficulty and cost of different task types, obtain the upstream and downstream dependencies of tasks, and build a task execution logic chain; form a task queue for all tasks to be run and in operation; generate a task scheduling plan based on the task queue and the task execution logic chain, and the scheduling plan includes several nodes, and each node is assigned several tasks in the task queue; map all tasks according to the task scheduling plan, allocate them in turn according to the load contribution of each task in high-performance computing, and perform migration; monitor the execution effect of task migration, and adjust the tasks assigned to each node until the load of each node tends to be balanced.

[0023] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0024] The present invention discloses a distributed automatic expansion method in high-performance computing, which can predict future computing resource requirements based on the task increase rate and determine whether the current resources can meet the requirements. When it is predicted that resources will be insufficient, it is possible to apply for new nodes in advance to prevent insufficient resources from affecting the tasks. At the same time, the system can also detect low utilization of idle resources and gradually release redundant nodes to avoid waste of resources. Once a faulty node is found, the system can calculate the importance of the node and determine the recovery order of the tasks of the faulty node. The system will recover the task from the fault point according to the priority and continue to detect until the task is completed. If the recovery time exceeds the specified time, the system will terminate the recovery and release the node resources.

[0025] When the amount of new tasks changes unevenly, the system will evaluate the load status of each node to improve the efficiency of resource utilization. In this way, the system can make full use of existing resources, ensure the smooth execution of tasks, and improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The present invention is a flow chart of a distributed automatic expansion method in high performance computing.

[0027] Figure 2 A schematic diagram of a distributed automatic expansion method in high performance computing according to the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0029] In this embodiment, a distributed automatic expansion method in high performance computing may specifically include:

[0030] S101. Predict future computing resource requirements according to the task increase rate and determine whether current resources can meet the requirements.

[0031] According to the historical task data of all nodes in the high-performance computing system, the task quantity, task type, and task completion time information are obtained to construct a historical task data set; a task increase rate prediction model is established according to the linear regression method, and the task increase rate prediction model is trained using the historical task data set until the accuracy of the task increase rate prediction model reaches the preset threshold, and the training is terminated to obtain a trained task increase rate prediction model; the task increase rate prediction model is used to predict real-time data to obtain the task growth rate; the required resource occupancy is calculated based on the task growth rate, task type, and average task completion time; the current resource capacity information is obtained to determine the available amount of each type of resource; it is determined whether the resource occupancy exceeds the resource capacity. When the resource occupancy exceeds the resource capacity, an application is made to add a distributed node.

[0032] For example, there are 100 nodes in a high-performance computing system. Based on the historical task data of these nodes, a historical task data set can be constructed. Each sample contains three features: the number of tasks, the type of task, and the time to complete the task. The linear regression method is used to establish a task increase rate prediction model. The number of tasks is selected as the prediction target, and the type of task and the time to complete the task are selected as independent variables. The historical task data set is divided into a training set and a test set. The training set contains the first 80 samples, and the test set contains the last 20 samples. The task increase rate prediction model is trained using the data in the training set. During the training process, the difference between the predicted value and the true value is minimized by continuously adjusting the model parameters until the accuracy of the task increase rate prediction model reaches the preset threshold. The trained task increase rate prediction model is: task increase rate = 0.5*number of tasks + 2*task completion time + 1*task type. The trained task increase rate prediction model is used to predict the real-time data to obtain the task growth rate. The current real-time data is as follows: number of tasks = 180, task type = data analysis, and task completion time = 25 minutes. According to the task increase rate prediction model, the task growth rate can be calculated as 0.5*180+2*25+1*2=90+50+2=142. Combining the task growth rate, task type, and average task completion time, the required resource occupancy can be calculated. The average task completion time is 15 minutes, and the resource occupancy coefficient of the task type data analysis is 1. The required resource occupancy is calculated to be 142*2*15=4260. Obtain the current resource capacity information and determine the available quota of each resource. If the current resource capacity is 4000, the resource occupancy will exceed the resource capacity, and you need to apply for a new distributed node.

[0033] S102: When it is predicted that resources will be insufficient, apply for adding new nodes in advance.

[0034] Based on the predicted resource requirements, the configuration requirements of the new node are determined, including the number of CPU and GPU cores, memory size, and storage capacity; apply for new nodes from the distributed node resource pool, obtain resource pool information, and determine the types and quantities of resources that can be applied for at present. Resource types include GPU, CPU, and storage; according to resource requirements, task priority, and node configuration, tasks and nodes are sorted separately to obtain the task order and node order, and tasks are arranged in the distributed nodes in the node order according to the task order.

[0035] For example, a new distributed node is needed to process a batch of tasks. Based on the predicted resource requirements, the number of CPU cores, GPU cores, memory size, and storage capacity need to be considered. Based on the nature and priority of the task, the resources for the new node are configured as follows: 16 CPU cores, 4 GPU cores, 64GB memory size, and 1TB storage capacity. Next, apply to the distributed node resource pool for a new node and obtain resource pool information. The following resources are available in the resource pool: 10 GPUs, 50 CPUs, and 2TB storage. Based on the resource requirements and resource pool information, the types and quantities of resources that can be applied are as follows: 4 GPUs, 16 CPUs, and 1TB storage. Sort the tasks and nodes according to the priority of the task and the node configuration. There are the following tasks to be processed, and their priorities and resource requirements are as follows: Task 1 has a priority of 1, requiring 8 CPU cores, 2 GPU cores, 32 GB memory, and 500 GB storage capacity; Task 2 has a priority of 2, requiring 4 CPU cores, 1 GPU core, 16 GB memory, and 250 GB storage capacity; Task 3 has a priority of 3, requiring 2 CPU cores, 1 GPU core, 8 GB memory, and 100 GB storage capacity. According to the priorities and resource requirements of the tasks, the tasks are sorted as Task 1, Task 2, and Task 3. According to the configuration of the nodes, the nodes can be sorted as follows: new node, node A, node B, and node C. According to the order of the tasks, the tasks are arranged in the distributed nodes in the order of the nodes: Task 1 is arranged to the new node, Task 2 is arranged to the new node, and Task 3 is arranged to Node A.

[0036] S103, monitor the utilization of nodes and release redundant nodes to avoid waste of resources.

[0037] Obtain the resource utilization of each node and determine whether there are inefficient nodes; when the average resource utilization of a node is lower than the threshold, it is marked as an inefficient node; select the target node based on task requirements and other node load conditions, and migrate the tasks on the inefficient node to the target node; gradually migrate tasks until there are no active tasks running on the inefficient node; if the inefficient node still has task data or status information, place it in external storage; retract the inefficient node to the distributed node resource pool, and update the node list and resource overview information in high-performance computing.

[0038] For example, the system detects resource utilization node by node, and sets the utilization rate below 15% to be an inefficient node. If the node with ID A is detected, and its GPU utilization rate is continuously below 15%, node A is marked as an inefficient node, and the task migration process is started. There are 3 tasks running on node A, which are marked as A1, A2, and A3. Task A1 requires 2 GPUs for 3 hours to complete, task A2 takes 4 GPUs for 2 hours, and task A3 takes 3 GPUs for 1 hour. In order to release node A as soon as possible, tasks A1 and A2 are migrated to nodes B and C, where the current GPU utilization rate is low. Since node D has completed the remaining tasks and its 6 GPUs are idle, task A3 is migrated to node D to complete resource scheduling optimization. After all tasks are migrated, the GPU resources on node A are completely idle. Node A also stores the intermediate result data of the task, so the data is migrated to the object storage server for permanent storage. Finally, the GPU resources of node A are recovered and returned to the distributed node resource pool to reduce resource waste. Update the node list and resource overview information in the high-performance computing system.

[0039] S104. Determine the node operation status through heartbeat detection to achieve real-time detection of faulty nodes.

[0040] A fixed heartbeat detection mechanism is set up, and heartbeat packets are sent regularly between the node and the monitoring center; the heartbeat response time and response status are obtained to determine whether the node is currently reachable; when the heartbeat does not respond for more than the preset number of times, the node status is determined to be abnormal and the connection is lost; according to the node connection information in the heartbeat packet and the time of the node's last response, the failure time point is calculated; the fault type is determined, including hardware failure, network failure, software failure, and permission failure; according to the importance of the node and the scope of the fault impact, the fault level is determined, which is divided into fatal failure, serious failure, and general failure; if it is determined to be a fatal level failure, a fault alarm is issued immediately; other levels of failures are logged, sorted according to the fault level, and enter the fault queue for processing.

[0041] For example, the heartbeat frequency is set to 1 time per minute, and the heartbeat data packet contains the node ID, operating status, and node connection information. Under normal circumstances, each node sends a heartbeat to the monitoring center at an interval of 30 seconds. It is detected that the heartbeat packet of node A is suddenly interrupted, and the system records the time when the heartbeat of node A is last received as 09:05. In the time window of 2 minutes thereafter, no heartbeat response from node A is received. It is determined that node A fails at about 09:07 and loses connection. For the failure of node A, first analyze the infrastructure such as hardware cabinets, power cables, and network cables, and then check the process status of the operating system kernel, system services, and application software. According to the on-site status investigation, it is determined that node A has a network interface hardware failure and cannot communicate normally with the monitoring center. The tasks running on node A belong to the critical business category, and the impact of the failure is assessed to be serious. Therefore, the alarm information of the serious failure is immediately triggered, and the operation and maintenance personnel are notified to handle it. Other levels of failures are logged, sorted according to the failure level, and enter the failure queue for processing.

[0042] S105. After a fault is discovered, the importance is calculated to determine the order in which the tasks of the faulty node are restored.

[0043] For all running tasks, set key task identifiers and task weights, and the weights are determined according to the business line to which the tasks belong. Once a faulty node is found, obtain information on all affected tasks. Evaluate the task running progress and determine the cost and impact of task recalculation. Analyze task dependencies and determine subsequent related tasks that are directly dependent on the affected tasks. Calculate the comprehensive importance of tasks based on business weights and the degree of task loss. Determine the priority of task recovery based on task importance, and recover the most important tasks first. At the same time, obtain the complexity of task recalculation, and recover the tasks with the lowest complexity in advance.

[0044] For example, when submitting a task, you need to fill in the task name, task type, business attributes, priority and other information. The priority is divided into 1-5 levels, and the larger the number, the higher the priority. There are three types of business attributes: core business, important business, and ordinary business. After discovering the failure of computing node A, obtain all task information running on node A. Task 1 is a core business, with a priority of 4 and a running progress of 30%; Task 2 is a general business, with a priority of 3 and a running progress of 80%. The comprehensive weight of computing task 1 is the weight of the core business attribute 3x priority 4=12; the comprehensive weight of task 2 is the weight of the general business 1x priority 3=3. The running progress of task 2 is higher, and if it is recalculated, it will lose more. The complexity of its recalculation is lower than that of task 1. Therefore, the determined task recovery priority is task 2>task 1. That is, first restore task 2 to other available computing nodes to continue running, and then start the recalculation of task 1.

[0045] S106. Recover the task from the failure point according to the priority, and repeat the detection until the task is completed.

[0046] Check the resource status of the standby node to ensure that it meets the recovery task requirements; match tasks according to priority and available resources, and schedule the recovery of failed tasks; restart and resume tasks that can be directly restarted; set migration strategies and new node information for tasks that need to be migrated, and execute migration; monitor the recovery status of failed tasks in real time to detect whether the recovery is successful; if there are still tasks that fail to recover after all tasks have been recovered, re-formulate a scheduling plan until all tasks are recovered; detect whether the original failed node is repaired, and automatically schedule new tasks to the repaired node after the repair.

[0047] For example, it has been confirmed that Task 1 has the highest priority, followed by Task 2. Currently, Node B and Node C are available for recovery. A recovery plan is then formulated: Try to restart Task 1 directly on Node B. If it fails, migrate Task 1 to Node C for restart; monitor the recovery status of Task 1. After Task 1 is successfully restored, migrate Task 2 to the idle Node B for restart. Execute the above recovery plan and obtain the real-time status of each task by monitoring the real-time information of high-performance computing. Task 1 is successfully restarted directly on Node B, and Task 1 starts to run normally. The status and intermediate data of Task 2 are migrated to Node B, and Task 2 is successfully started. It is now confirmed that all faulty tasks have been successfully restored and the business system is running stably. At the same time, the fault detection module is bound to continuously detect the repair status of the original faulty node. After detecting that the fault of Node A is repaired, the newly created Task 3 is automatically scheduled directly to Node A for execution.

[0048] S107: If the recovery time exceeds the specified time, the recovery is terminated and the node resources are released.

[0049] Set the task recovery deadline based on task priority and business attributes; monitor the duration and number of task recovery attempts; terminate the recovery when the cumulative recovery duration exceeds the task recovery deadline or the number of attempts exceeds the preset threshold; roll back the state of the rollbackable tasks that have terminated recovery to the last checkpoint before the failure; obtain the intermediate state and data of the non-rollbackable tasks and save them to external storage; release the failed node and return it to the distributed node resource pool.

[0050] For example, a 1-hour recovery period is set for core business tasks, and the threshold for the number of attempts is 5. The actual recovery time and the number of repeated recovery times of Task 1 are counted. The monitoring center detects that the cumulative recovery time of Task 1 has exceeded 2 hours. The current recovery attempt is the third time and all have failed. Due to the recovery timeout policy, the subsequent recovery operation of Task 1 is terminated. Since Task 1 has run 30% and has saved intermediate states, it can be directly rolled back to the most recent checkpoint to minimize repeated calculations. For tasks that cannot be rolled back, their intermediate states and data will be obtained and saved to external storage for subsequent analysis and processing. Finally, the faulty node will be released and returned to the distributed node resource pool for use by other tasks or businesses. Generate an evaluation report for this fault recovery, analyze the duration of the fault impact, the degree of business loss, the rollback of the stored intermediate states, etc., and provide a reference for improving the recovery mechanism next time.

[0051] S108. When the amount of newly added tasks changes unevenly, evaluate the load status of each node.

[0052] Record the relevant characteristic information of the newly added tasks, including task type, required resources, and business attributes; collect real-time operation and load data of each computing node; monitor the CPU, memory, and disk utilization of each node. When the node load limit is exceeded, the node is considered to be overloaded; monitor the load changes of each node. When all nodes exceeding the preset threshold are overloaded, the high-performance computing is determined to be overloaded and expansion is triggered; when all nodes exceeding the preset threshold are overloaded, the high-performance computing is determined to be in a high-load state, and traffic control and frequency limiting strategies are initiated.

[0053] For example, when recording the relevant characteristic information of a newly added task, the type of the newly added task is an image processing task, the required resources include 8 CPU cores, 16GB memory size, 500GB disk space, and high priority business attributes. When collecting the real-time operation and load data of each computing node, the current CPU utilization of a computing node is 90%, the memory utilization is 75%, and the disk utilization is 80%. The node load limit is 75%, so the node is considered to be overloaded. When monitoring the load changes of each node, there are 10 nodes, of which 9 nodes have loads exceeding the node load limit, that is, the proportion of overloaded nodes is 90%. When more than 85% of the nodes are overloaded, it is determined that high-performance computing is overloaded and expansion needs to be triggered. When more than 70% of the nodes are overloaded, it is determined that high-performance computing is in a high-load state, and traffic control and frequency limiting strategies need to be started.

[0054] S109. When high-performance computing is overloaded, a scheduling plan is formed for tasks based on the load, and tasks are remapped to nodes based on the scheduling plan to improve resource utilization efficiency.

[0055] Collect historical resource usage and historical task load data of each node in high-performance computing, and build a task load data set; establish a task load model based on autoregressive integrated moving average, and use the task load data set to train the task load model to obtain a trained task load model; input the current resource usage and task load data of each node into the task load model to evaluate the load contribution of each node in executing tasks; obtain the migration difficulty and cost of different task types, obtain the upstream and downstream dependencies of tasks, and build a task execution logic chain; form a task queue for all tasks to be run and in operation; generate a task scheduling plan based on the task queue and the task execution logic chain, and the scheduling plan includes several nodes, and each node is assigned several tasks in the task queue; map all tasks according to the task scheduling plan, allocate them in turn according to the load contribution of each task in high-performance computing, and perform migration; monitor the execution effect of task migration, and adjust the tasks assigned to each node until the load of each node tends to be balanced.

[0056] For example, collect the historical resource usage and task load data of each node in high-performance computing and build a task load data set. Establish a task load model based on autoregressive integrated moving average, and use the task load data set to train the task load model. Use the autoregressive integrated moving average model ARIMA (1,0,1) to model the CPU utilization of node X, and the model parameters obtained by training are 7 and 3. Input the current resource usage and task load data of each node into the task load model to evaluate the load contribution of each node in executing tasks. According to the task load data of node X and node Y, the average task execution time of node X is calculated to be 20 seconds, and the average task execution time of node Y is 15 seconds. The CPU utilization of node X is 75%, and the CPU utilization of node Y is 70%. Considering the task execution time and resource usage, it can be evaluated that the load contribution of node X in executing tasks is 0.75*20=15, and the load contribution of node Y in executing tasks is 0.70*15=10.5. Obtain the migration difficulty and cost of different task types, obtain the upstream and downstream dependencies of tasks, and build a task execution logic chain. The migration difficulty of task type A is 5, and the migration difficulty of task type B is 10. Task A is the upstream task of task B. The task execution logic chain is represented as A->B. All tasks to be run and running form a task queue. The task queue to be run includes task A and task B. A task scheduling scheme is generated based on the task queue and the task execution logic chain. The scheduling scheme includes several nodes, and each node is assigned several tasks in the task queue. According to the task execution logic chain, task A is assigned to node 1 and task B is assigned to node 2. All tasks are mapped according to the task scheduling scheme, and they are assigned in turn according to the load contribution of each task in high-performance computing, and then migration is performed. The load contribution of node X is 15, and the load contribution of node Y is 10.5. The nodes are sorted as A->B. Task A is migrated to node X, and task B is migrated to node Y. Monitor the execution effect of task migration and adjust the tasks assigned to each node until the load of each node tends to be balanced. Monitor the load of nodes X and Y. If the load of node X is too high, adjust the task scheduling plan and migrate some tasks from node X to node Y until the loads of the two nodes are balanced.

[0057] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A distributed automatic expansion method in high performance computing, characterized in that: The method comprises: Predict future computing resource requirements based on the rate of task increase and determine whether current resources can meet the needs; when it is predicted that resources will be insufficient, apply for new nodes in advance; monitor node utilization and release redundant nodes to avoid resource waste; determine node operation status through heartbeat detection and implement real-time detection of faulty nodes; after a fault is discovered, determine the order of task recovery for the faulty node based on calculation importance; recover tasks from the fault point based on priority and repeat detection until the task is completed; if the recovery time exceeds the specified time, terminate recovery and release node resources; when the amount of new tasks changes unevenly, evaluate the load status of each node; when high-performance computing is overloaded, form a scheduling plan for tasks based on the load, and remap tasks to nodes based on the scheduling plan to improve resource utilization efficiency; After the fault is found, the importance is calculated to determine the task recovery order of the faulty node, including: For all running tasks, set key task identifiers and task weights, and the weights are determined according to the business line to which the tasks belong; once a faulty node is found, obtain information on all affected tasks; evaluate the progress of task running, determine the cost and impact of task recalculation; analyze task dependencies, and determine subsequent related tasks that directly depend on the affected tasks; calculate the comprehensive importance of tasks based on business weights and task loss levels; determine the priority of task recovery based on task importance, and recover the most important tasks first; at the same time, obtain the complexity of task recalculation, and recover the tasks with the lowest complexity in advance; When the high performance computing is overloaded, a scheduling scheme is formed for the tasks according to the load, and the tasks are remapped to the nodes according to the scheduling scheme to improve resource utilization efficiency, including: Collect historical resource usage and historical task load data of each node in high-performance computing, and build a task load data set; establish a task load model based on autoregressive integrated moving average, and use the task load data set to train the task load model to obtain a trained task load model; input the current resource usage and task load data of each node into the task load model to evaluate the load contribution of each node in executing tasks; obtain the migration difficulty and cost of different task types, obtain the upstream and downstream dependencies of tasks, and build a task execution logic chain; form a task queue for all tasks to be run and in operation; generate a task scheduling plan based on the task queue and the task execution logic chain, and the scheduling plan includes several nodes, each node is assigned several tasks in the task queue; map all tasks according to the task scheduling plan, allocate them in turn according to the load contribution of each task in high-performance computing, and perform migration; monitor the execution effect of task migration, and adjust the tasks assigned to each node until the load of each node tends to be balanced.

2. The method according to claim 1, wherein: The predicting of future computing resource requirements according to the task increase rate and judging whether current resources can meet the requirements include: According to the historical task data of all nodes in the high-performance computing system, the task quantity, task type and task completion time information are obtained to construct a historical task data set; a task increase rate prediction model is established according to the linear regression method, and the task increase rate prediction model is trained using the historical task data set until the accuracy of the task increase rate prediction model reaches the preset threshold, and the training is terminated to obtain a trained task increase rate prediction model; the task increase rate prediction model is used to predict real-time data to obtain the task growth rate; the required resource occupancy is calculated based on the task growth rate, task type and average task completion time; the current resource capacity information is obtained to determine the available amount of each type of resource; it is determined whether the resource occupancy exceeds the resource capacity. When the resource occupancy exceeds the resource capacity, an application is made to add a distributed node.

3. The method according to claim 1, wherein: When it is predicted that resources will be insufficient, applying for adding new nodes in advance includes: Based on the predicted resource requirements, the configuration requirements of the new node are determined, including the number of CPU and GPU cores, memory size, and storage capacity; apply for new nodes from the distributed node resource pool, obtain resource pool information, and determine the types and quantities of resources that can be applied for at present. Resource types include GPU, CPU, and storage; according to resource requirements, task priority, and node configuration, tasks and nodes are sorted respectively to obtain the task order and node order, and tasks are arranged in the distributed nodes in the node order according to the task order.

4. The method according to claim 1, wherein: The monitoring node utilization and releasing redundant nodes to avoid resource waste include: Obtain the resource utilization of each node and determine whether there are inefficient nodes; when the average resource utilization of a node is lower than the threshold, it is marked as an inefficient node; select the target node based on the task requirements and the load of other nodes, and migrate the tasks on the inefficient node to the target node; gradually migrate tasks until there are no active tasks running on the inefficient node; if the inefficient node still has task data or status information, place it in external storage; Reclaim inefficient nodes to the distributed node resource pool, and update the node list and resource overview information in high-performance computing.

5. The method according to claim 1, wherein: The node operation status is judged by heartbeat detection to realize real-time detection of faulty nodes, including: Set up a fixed heartbeat detection mechanism, and periodically send heartbeat packets between the node and the monitoring center; Get the heartbeat response time and response status to determine whether the node is currently reachable; When the heartbeat does not respond for more than the preset number of times, the node is judged to be abnormal and the connection is lost; the failure time point is calculated based on the node connection information in the heartbeat packet and the time when the node last responded; Determine the fault type, including hardware fault, network fault, software fault and permission fault; determine the fault level according to the importance of the node and the scope of fault impact, and divide it into fatal fault, serious fault and general fault; If a fatal fault is identified, a fault alarm will be issued immediately; other faults will be logged and sorted according to the fault level before entering the fault queue to be processed.

6. The method according to claim 1, wherein: The step of recovering the task from the failure point according to the priority and repeating the detection until the task is completed includes: Check the resource status of the standby node to ensure that it meets the recovery task requirements; match tasks according to priority and available resources, and schedule the recovery of failed tasks; restart and resume tasks that can be directly restarted; set migration strategies and new node information for tasks that need to be migrated, and perform migration; monitor the recovery status of failed tasks in real time to detect whether the recovery is successful; if there are still tasks that fail to recover after all tasks have been recovered, re-plan the scheduling plan until all tasks are recovered; Check whether the original faulty node is repaired, and automatically schedule new tasks to the repaired node after repair.

7. The method according to claim 1, wherein: If the recovery time exceeds the specified time, the recovery is terminated and the node resources are released, including: Set the task recovery deadline according to the task priority and business attributes; monitor the duration and number of attempts of task recovery; terminate the recovery when the cumulative recovery duration exceeds the task recovery deadline or the number of attempts exceeds the preset threshold; roll back the status of the rollback tasks that have been terminated and restored to the last checkpoint before the failure; obtain the intermediate status and data of the tasks that cannot be rolled back and save them to external storage; Release the faulty node and return it to the distributed node resource pool.

8. The method according to claim 1, wherein: When the amount of newly added tasks changes unevenly, the load status of each node is evaluated, including: Record the relevant characteristic information of the newly added tasks, including the task type, required resources and business attributes; collect the real-time operation and load data of each computing node; monitor the CPU, memory and disk utilization of each node. When the node load limit is exceeded, the node is considered to be overloaded; monitor the load changes of each node. When all nodes exceeding the preset threshold are overloaded, the high-performance computing is determined to be overloaded and the expansion is triggered; when all nodes exceeding the preset threshold are overloaded, the high-performance computing is determined to be in a high-load state, and the traffic control and frequency limiting strategies are initiated.

Images