Large model training task arrangement method based on heterogeneous resource pool

Abstract

The invention provides a big model training task arrangement method based on a heterogeneous resource pool. The method comprises the following steps: constructing a heterogeneous resource pool, and obtaining a current resource state and historical performance data of each computing node; determining a computing resource demand feature of each sub-task of the large model training task; according to the current resource state and the historical performance data of each computing node and the computing resource demand characteristics of each sub-task, matching the computing node for each sub-task by comprehensively considering the current load and the historical performance of the computing node, the predicted energy consumption for executing the corresponding sub-task and the computing resource demand characteristics of the sub-task; in the process of executing the subtask based on the computing node, if it is detected that the resource state of the computing node does not meet the preset requirement or the execution progress of the subtask is abnormal, the computing node is matched for the subtask again, and the subtask continues to be executed; and after the subtasks are completed, summarizing and verifying the large model training tasks, and releasing the occupied computing nodes. According to the invention, the problems of low heterogeneous computing resource utilization rate and low energy efficiency ratio of the existing large model training based on heterogeneous computing resources can be solved.

Description

Technical Field

[0001] This invention belongs to the field of distributed computing and artificial intelligence technology, and more specifically, relates to a method for orchestrating large model training tasks based on heterogeneous resource pools. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, large-scale model training has become an important research direction. Large-scale model training involves higher computational demands and memory consumption, placing greater demands on computing and storage resources. To meet the needs of large-scale model training, heterogeneous computing platforms have emerged. A heterogeneous computing platform is a computing platform that integrates different types of processors (such as CPUs, GPUs, NPUs, etc.), which can fully leverage the advantages of different processors in terms of computation and storage, thereby improving computational efficiency. In large-scale model training, heterogeneous computing platforms can simultaneously handle large-scale data and highly complex calculations, effectively improving training speed and accuracy.

[0003] However, with the scaling up of large models, the training process places higher demands on the coordination of heterogeneous computing resources. Existing methods for training large models based on heterogeneous computing resources mainly face the following problems: Imbalance in the utilization of heterogeneous hardware resources: The existing fixed allocation and scheduling strategy for computing resources cannot dynamically adapt to the differences in hardware characteristics (such as GPUs being good at matrix operations and NPUs being good at inference acceleration), resulting in lower-than-expected utilization of dedicated chips and CPU nodes often being overloaded due to the accumulation of I / O tasks. Low energy efficiency: The lack of a global energy efficiency optimization mechanism and the fixed allocation and scheduling strategy of computing resources lead to uneven computing resource load, resulting in low energy efficiency for training large models. Summary of the Invention

[0004] The purpose of this invention is to solve the problems of low utilization rate and low energy efficiency of heterogeneous computing resources in the training of large models based on heterogeneous computing resources.

[0005] To achieve the above objectives, this invention provides a method for orchestrating large model training tasks based on heterogeneous resource pools. This method includes the following steps: Construct a heterogeneous resource pool containing different types of computing nodes, monitor the current resource status of each computing node, and record the historical performance data of each computing node. The large model training task is decomposed into several corresponding sub-tasks, and the computational resource requirements of each sub-task are determined. Based on the current resource status and historical performance data of each computing node, as well as the computing resource requirements of each subtask, a computing node is matched for each subtask by comprehensively considering the current load, historical performance, expected energy consumption of executing the corresponding subtask, and computing resource requirements of the subtask. If, during the execution of a subtask based on a corresponding computing node, it is detected that the resource status of the computing node does not meet the predetermined requirements or the execution progress of the subtask is abnormal, then a computing node is re-matched for the subtask and the subtask continues to be executed based on the re-matched computing node. After all subtasks are completed, the large model training task is summarized and validated, and the occupied computing nodes are released.

[0006] Optionally, the step of constructing a heterogeneous resource pool containing different types of computing nodes further includes: Configure a predetermined number of CPU computing nodes, GPU computing nodes, and NPU computing nodes; High-speed networks are used to interconnect various CPU computing nodes, GPU computing nodes and NPU computing nodes to form a heterogeneous resource pool.

[0007] Optionally, the steps of monitoring the current resource status of each computing node and recording the historical performance data of each computing node further include: Set up a central management node and deploy a monitoring agent on each computing node; The central management node collects the current resource status and historical performance data of the corresponding computing nodes from the various deployed monitoring agents.

[0008] Optionally, the step of decomposing the large model training task into several corresponding sub-tasks and determining the computational resource requirements of each sub-task further includes: The large model training task is decomposed according to task type, computational complexity, or data scale to obtain several corresponding sub-tasks. Each subtask is labeled with its corresponding computing resource requirement characteristics to form a subtask list. The computing resource requirement characteristics include CPU-dependent, GPU-accelerated, and NPU-dedicated types.

[0009] Alternatively, if the task type is used as the dimension for task decomposition, the large model training task is decomposed into a data preprocessing subtask, a model training subtask, and a model evaluation subtask. And / or, If computational complexity is used as a dimension for task decomposition, then large model training tasks are decomposed into computationally intensive subtasks and I / O-intensive subtasks.

[0010] Optionally, the step of matching a computing node for each subtask by comprehensively considering the current load, historical performance, expected energy consumption of executing the corresponding subtask, and the computing resource requirements of the subtask further includes: Establish a task-resource matching model based on heuristic rules and machine learning algorithms; Based on the task-resource matching model, a computing node is matched for each subtask, and a subtask-node mapping table is generated. The subtask-node mapping table includes the mapping relationship between the subtask and the matched computing node, the subtask ID, the computing node IP, and the resource quota.

[0011] Alternatively, the task-resource matching model is configured to follow the following principles: For CPU-dependent subtasks, priority should be given to assigning them to CPU computing nodes with better network bandwidth. For GPU-accelerated subtasks, priority is given to assigning them to GPU computing nodes with historically better performance. For NPU-specific subtasks, priority is given to assigning them to NPU computing nodes with historically good performance. For computing nodes of the same type, load balancing of computing resources should be considered. Under the premise of meeting the task execution time requirements, select computing nodes or combinations of computing nodes with lower energy consumption.

[0012] Optionally, the step of re-matching a computing node for the sub-task and continuing to execute the sub-task based on the re-matched computing node further includes: If the resource status of the computing node does not meet the predetermined requirements or the execution progress of the sub-task is abnormal during the execution of the sub-task based on the corresponding computing node, The pre-deployed monitoring agent detects the resource status and task execution progress of the computing nodes executing subtasks at a predetermined frequency and uploads the relevant information to the central management node. If the central management node detects that the resource utilization rate of the computing node exceeds the threshold or the execution time of the subtask exceeds the expected total execution time to a predetermined standard, the central management node sends a task stop execution instruction to the corresponding computing node and freezes the task process. Based on the task-resource matching model, the computing nodes for the subtask are rematched, and the task status data of the original computing node is migrated to the rematched computing node based on RDMA technology so that the subtask can continue to be executed.

[0013] Optionally, the steps of summarizing and validating the large model training task after all subtasks are completed, and releasing the occupied computing nodes, further include: The central management node sends process termination instructions to each computing node, releasing the computing resources of each computing node. It also destroys the task containers on the CPU computing nodes and reclaims memory and CPU cores through the Docker container management system.

[0014] Optionally, the steps of summarizing and validating the large model training task after all subtasks are completed, and releasing the occupied computing nodes, further include: Update the status list of the heterogeneous resource pool, mark the released compute nodes as available, and trigger a health check of the heterogeneous resource pool.

[0015] The beneficial effects of this invention are as follows: The large model training task orchestration method based on heterogeneous resource pools of the present invention, after obtaining the current resource status and historical performance data of each computing node, as well as the computing resource requirement characteristics of each subtask, comprehensively considers the current load, historical performance, expected energy consumption of executing the corresponding subtask, and computing resource requirement characteristics of the computing node, and matches a suitable computing node for each subtask of the large model training based on a multi-objective optimization scheduling strategy, and then executes the corresponding subtask based on the corresponding computing node.

[0016] On the one hand, this invention significantly improves the utilization of computing resources through precise task-resource matching, increasing the utilization of dedicated acceleration chips such as GPUs / NPUs by more than 30% and the I / O processing capacity utilization of CPU nodes by more than 20%, thus avoiding the resource idleness or overload problems caused by existing fixed allocation and scheduling strategies for computing resources. On the other hand, this invention determines the computing nodes corresponding to subtasks based on a load balancing strategy. This setting enables global energy efficiency optimization, ensuring that idle computing resources are fully utilized, thereby improving the energy efficiency ratio of large model training.

[0017] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0018] The present invention can be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which the same or similar reference numerals are used throughout the drawings to denote the same or similar parts.

[0019] Figure 1 A flowchart illustrating the implementation of a large model training task orchestration method based on a heterogeneous resource pool according to an embodiment of the present invention is shown. Figure 2A flowchart of a large model training task orchestration method based on a heterogeneous resource pool according to an embodiment of the present invention is shown. Detailed Implementation

[0020] To enable those skilled in the art to more fully understand the technical solutions of the present invention, exemplary embodiments of the present invention will be described more comprehensively and in detail below with reference to the accompanying drawings. Obviously, the one or more embodiments of the present invention described below are merely one or more specific ways to implement the technical solutions of the present invention, and are not exhaustive. It should be understood that other ways belonging to a general inventive concept can be used to implement the technical solutions of the present invention, and should not be limited to the embodiments described exemplary. Based on one or more embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] Example: Figure 1 A flowchart illustrating the implementation of a large model training task orchestration method based on a heterogeneous resource pool, according to an embodiment of the present invention, is shown. (Refer to...) Figure 1 The large model training task orchestration method based on heterogeneous resource pools in this embodiment of the invention includes the following steps: Step S100: Construct a heterogeneous resource pool containing different types of computing nodes, monitor the current resource status of each computing node, and record the historical performance data of each computing node. Step S200: Decompose the large model training task into several corresponding sub-tasks and determine the computational resource requirements of each sub-task. Step S300: Based on the current resource status and historical performance data of each computing node, as well as the computing resource requirements of each subtask, a computing node is matched for each subtask by comprehensively considering the current load, historical performance, expected energy consumption of executing the corresponding subtask, and computing resource requirements of the subtask. Step S400: During the execution of a subtask based on a corresponding computing node, if it is detected that the resource status of the computing node does not meet the predetermined requirements or the execution progress of the subtask is abnormal, then a computing node is re-matched for the subtask and the subtask continues to be executed based on the re-matched computing node. Step S500: After all subtasks are completed, summarize and verify the large model training task, and release the occupied computing nodes.

[0022] Furthermore, in step S100 of this embodiment of the invention, constructing a heterogeneous resource pool containing different types of computing nodes further includes the following steps: Configure a predetermined number of CPU computing nodes, GPU computing nodes, and NPU computing nodes; High-speed networks are used to interconnect various CPU computing nodes, GPU computing nodes and NPU computing nodes to form a heterogeneous resource pool.

[0023] Furthermore, in step S100 of this embodiment of the invention, monitoring the current resource status of each computing node and recording the historical performance data of each computing node further includes the following steps: Set up a central management node and deploy a monitoring agent on each computing node; The central management node collects the current resource status and historical performance data of the corresponding computing nodes from the various deployed monitoring agents.

[0024] Specifically, in this embodiment of the invention, the constructed heterogeneous resource pool includes CPU computing nodes for general-purpose computing, GPU computing nodes for matrix operation acceleration, and NPU computing nodes for AI-specific acceleration, wherein the NPU computing nodes are implemented using the Ascend AI processor. The computing nodes within the heterogeneous resource pool are interconnected via the infiniteBand high-speed network. A monitoring agent is deployed on each computing node, which collects real-time resource status information of each computing node, including but not limited to CPU core utilization, memory bandwidth, disk I / O throughput, GPU memory utilization, temperature, and the load and power consumption of the Ascend AI processor. Simultaneously, it records historical performance data for each computing node, such as task execution data from the past 30 days.

[0025] Furthermore, in this embodiment of the invention, step S200, which involves decomposing the large model training task into several corresponding sub-tasks and determining the computational resource requirements of each sub-task, further includes the following steps: The large model training task is decomposed according to task type, computational complexity, or data scale to obtain several corresponding sub-tasks. Each subtask is labeled with its corresponding computing resource requirement characteristics to form a subtask list. The computing resource requirement characteristics include CPU-dependent, GPU-accelerated, and NPU-dedicated.

[0026] Furthermore, in this embodiment of the invention, when decomposing a large model training task, if the task type is used as the dimension of task decomposition, the large model training task is decomposed into a data preprocessing subtask, a model training subtask, and a model evaluation subtask. When decomposing large model training tasks, if computational complexity is used as the dimension of task decomposition, the large model training task is decomposed into computationally intensive subtasks and I / O-intensive subtasks. For computationally intensive subtasks (such as matrix operations in training the model), they are preferentially allocated to dedicated acceleration chips such as GPUs or Ascend to make full use of their parallel computing capabilities and improve training speed. For I / O-intensive subtasks (such as data loading), they are preferentially allocated to CPU resources, while taking into account the impact of network bandwidth to ensure the efficiency of data transmission.

[0027] Specifically, in this embodiment of the invention, the large model training task is decomposed into subtasks according to factors such as task type (e.g., data preprocessing, model training, model evaluation, etc.), computational complexity, and data scale. Each subtask is labeled with its computational resource requirements; for example, the data preprocessing subtask relies more on the general computing power of the CPU, while the model training subtask may have higher requirements for the parallel computing power of the GPU or Ascend.

[0028] Furthermore, in step S300 of this embodiment of the invention, the step of matching a computing node for each subtask by comprehensively considering the current load, historical performance, expected energy consumption of executing the corresponding subtask, and computing resource requirements of the subtask further includes the following steps: Establish a task-resource matching model based on heuristic rules and machine learning algorithms; Based on the task-resource matching model, a computing node is matched for each subtask, and a subtask-node mapping table is generated. The subtask-node mapping table includes the mapping relationship between the subtask and the matched computing node, the subtask ID, the computing node IP, and the resource quota.

[0029] Furthermore, in this embodiment of the invention, the task-resource matching model is configured to follow the following principles: For CPU-dependent subtasks, priority should be given to assigning them to CPU computing nodes with better network bandwidth. For GPU-accelerated subtasks, priority is given to assigning them to GPU computing nodes with historically better performance. For NPU-specific subtasks, priority is given to assigning them to NPU computing nodes with historically good performance. For computing nodes of the same type, load balancing of computing resources should be considered. Under the premise of meeting the task execution time requirements, select computing nodes or combinations of computing nodes with lower energy consumption.

[0030] Furthermore, step S400 of this embodiment of the invention further includes the following steps: The pre-deployed monitoring agent detects the resource status and task execution progress of the computing nodes executing subtasks at a predetermined frequency and uploads the relevant information to the central management node. If the central management node detects that the resource utilization of the computing node exceeds the threshold or the execution time of the subtask exceeds the expected total execution time to a predetermined standard, the central management node sends a task stop execution command to the corresponding computing node and freezes the task process. The task-resource matching model is used to re-match computing nodes for subtasks, and the task status data of the original computing nodes is migrated to the re-matched computing nodes based on RDMA technology so that the subtasks can continue to be executed.

[0031] Specifically, in this embodiment of the invention, during the execution of subtasks, the resource status of each computing node and the execution progress of the subtasks are monitored in real time. When changes in resource status occur (such as insufficient GPU memory) or task execution anomalies occur (such as subtasks taking longer than expected), a dynamic adjustment mechanism is triggered. The dynamic adjustment mechanism re-matches and re-arranges tasks and resources based on the new resource status and task execution situation, and adjusts the execution nodes of subtasks in a timely manner to ensure the smooth execution of the entire training task.

[0032] Furthermore, step S500 of this embodiment of the invention further includes the following steps: The central management node sends process termination instructions to each computing node, releasing the computing resources of each computing node, and destroying the task containers on the CPU computing nodes through the Docker container management system, and reclaiming memory and CPU cores. Update the status list of the heterogeneous resource pool, mark the released compute nodes as available, and trigger a health check of the heterogeneous resource pool.

[0033] Figure 2 A flowchart illustrating the large model training task orchestration method based on heterogeneous resource pools according to an embodiment of the present invention is shown. The following is in conjunction with... Figure 2 A more detailed description of the large model training task orchestration method based on heterogeneous resource pools in this embodiment of the invention is provided below: Reference Figure 2 The large model training task orchestration method based on heterogeneous resource pools in this invention includes five core steps: Resource status collection, that is, the monitoring agent periodically collects the real-time load (CPU utilization, GPU memory usage, etc.) and historical performance data (average execution latency and energy consumption of various tasks over a period of time) of the computing nodes and stores them in the database; Task decomposition and labeling: Based on the TensorFlow Serving framework, large model training tasks are decomposed into sub-tasks such as data preprocessing (CPU-dependent), model forward propagation (GPU-accelerated), and gradient calculation (Ascend-specific), generating a task list with resource requirement tags; Intelligent matching algorithm: The scheduling engine generates the optimal allocation scheme based on the task list and resource status using a multi-objective optimization algorithm, and outputs the "Subtask-Node Mapping Table"; Dynamically triggered adjustment: When the GPU node memory usage exceeds 90% or the subtask execution times out (exceeding the expected time by 150%), a rescheduling process is triggered to regenerate the mapping table and migrate the task. Resource release and reclamation: After the task is completed, the central management node sends a completion command to the computing nodes, releases the occupied resources, and updates the availability status of the resource pool.

[0034] In practical applications, a heterogeneous resource pool can consist of multiple servers of different types. CPU servers provide general-purpose computing capabilities, while GPU servers and Ascend servers provide accelerated computing capabilities based on GPU and Ascend chips, respectively. These servers are connected via a high-speed network to form a unified resource pool. A monitoring agent is deployed on each computing node to collect resource status information in real time and send the information to a central management node.

[0035] The monitoring agent can use existing system monitoring tools, such as nvidia-smi, to obtain information such as GPU memory usage, temperature, and power consumption. For Ascend chip load information, it can be obtained through the dedicated monitoring interface provided by the Ascend CANN toolchain. Historical performance data is stored in a time-series database. The data tables include a series of metrics such as node_id, type, value, and timestamp, stored in the database of the central management node. Typically, the most recent 30 days of historical data are retained for subsequent task-resource matching and orchestration.

[0036] Large model training tasks can be decomposed using existing task decomposition tools and frameworks. During decomposition, the task is divided into subtasks such as data preprocessing, model forward propagation, back propagation, and parameter updating, taking into account the architecture and training process of the large model. The computational complexity and resource requirements of each subtask are then analyzed. For example, the data preprocessing task includes text segmentation, word embedding generation, data reading, cleaning, and transformation, which places certain demands on CPU computing power and memory capacity. Meanwhile, the model forward and back propagation subtasks involve numerous large matrix operations, requiring high parallel computing capabilities from GPUs or Ascend.

[0037] The task-resource matching model employs a rule-based heuristic algorithm, setting resource selection priority rules for different types of subtasks. For computationally intensive subtasks, the priority rule prioritizes GPUs or Ascend nodes with lower current load and better historical performance; for I / O-intensive subtasks, it prioritizes CPU nodes with lower CPU load and sufficient network bandwidth. Simultaneously, combined with an energy consumption model, the estimated energy consumption for executing subtasks on each computing node is calculated. Deep reinforcement learning algorithms are used to train the model to automatically learn the optimal task-resource matching strategy, adapting to different training tasks and resource environments.

[0038] The dynamic adjustment mechanism is implemented through a combination of timed monitoring and event triggering. The timed monitoring interval is set according to actual conditions, typically checking resource status and task execution progress every 10 seconds. A dynamic adjustment event is triggered when the resource utilization of a computing node exceeds a threshold (e.g., GPU node memory usage exceeds 90%) or the execution time of a subtask exceeds 150% of the estimated time (the estimated time is calculated based on a 95% confidence interval of historical execution times for similar tasks). During dynamic adjustment, the central management node sends a TaskSuspend command to the target node, freezing the task process while preserving the GPU / RAM state, and migrating the task status data to a new node that meets the conditions using RDMA technology. On the new node, the task process is resumed, and the execution plan for the remaining tasks is recalculated based on the latest resource status. Subsequently, the Subtask-Node Mapping Table is updated, and the monitoring agent is notified to switch data collection targets.

[0039] Once the training task is complete, the central management node sends a process termination command to each computing node, releasing the occupied memory, GPU memory, and other resources. It then uses the Docker container management system to destroy the task containers on the CPU nodes, reclaiming memory and CPU cores. The resource pool's status list is updated, marking the released nodes as Available, and triggering a resource pool health check to facilitate subsequent task scheduling and orchestration.

[0040] This invention presents a large-scale model training task orchestration method based on heterogeneous resource pools. It utilizes multi-dimensional state modeling of heterogeneous resources to construct a two-layer state awareness system encompassing real-time resource load (CPU / GPU / Ascend utilization, video memory / memory usage, network bandwidth) and historical performance data (task execution time, energy consumption indicators). This overcomes the limitations of traditional single real-time state monitoring and provides a data foundation for intelligent scheduling. The refined task feature annotation system is based on a three-dimensional task decomposition annotation method using task type (data preprocessing / model training / evaluation), computational characteristics (computation-intensive / I / O-intensive), and resource requirements (general CPU requirements, GPU parallel computing / Ascend dedicated acceleration), achieving accurate description of task resource requirements. The task-resource matching model integrates resource load balancing, task execution efficiency, and energy consumption control. Through a combination of heuristic rules and machine learning algorithms, it dynamically generates the optimal task-resource matching strategy, addressing the shortcomings of traditional scheduling methods that only focus on a single performance indicator. The dynamic elastic adjustment mechanism is designed based on a real-time rescheduling algorithm triggered by thresholds (exceeding resource utilization limits / abnormal task time), which enables dynamic migration of task execution nodes during training and ensures the robustness of the system in the event of resource fluctuations or task anomalies.

[0041] The large model training task orchestration method based on heterogeneous resource pools in this invention has the following beneficial effects: 1. Significantly improved resource utilization: Through precise task-resource matching, the utilization rate of computing resources of dedicated acceleration chips such as GPU / Ascend is increased by more than 30%, and the utilization rate of I / O processing capability of CPU nodes is increased by more than 20%, avoiding resource idleness or overload problems caused by fixed allocation mode.

[0042] 2. Training efficiency is greatly improved. For computationally intensive model training tasks, the parallel computing advantages of GPU / Ascend, combined with dynamic load balancing strategies, can shorten training time by 20%-40%. For I / O intensive tasks, data preprocessing speed is improved by more than 30% through CPU node optimized scheduling.

[0043] 3. Energy consumption costs are effectively reduced. The node selection strategy based on the energy consumption model can reduce training energy consumption by 15%-25% while meeting the task execution time requirements. The power cost savings are particularly significant in large-scale distributed training scenarios.

[0044] 4. Enhanced system robustness: The dynamic adjustment mechanism enables the system to automatically respond to resource failures (such as GPU memory overflow) or task anomalies (such as data loading delays), reducing the risk of training interruption by more than 60% and ensuring the stability of long-term large model training.

[0045] While one or more embodiments of the present invention have been described above, those skilled in the art will recognize that the present invention can be implemented in any other form without departing from its spirit and scope. Therefore, the embodiments described above are illustrative and not restrictive, and many modifications and substitutions will be apparent to those skilled in the art without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for orchestrating large model training tasks based on heterogeneous resource pools, characterized in that, include: Construct a heterogeneous resource pool containing different types of computing nodes, monitor the current resource status of each computing node, and record the historical performance data of each computing node. The large model training task is decomposed into several corresponding sub-tasks, and the computational resource requirements of each sub-task are determined. Based on the current resource status and historical performance data of each computing node, as well as the computing resource requirements of each subtask, a computing node is matched for each subtask by comprehensively considering the current load, historical performance, expected energy consumption of executing the corresponding subtask, and computing resource requirements of the subtask. If, during the execution of a subtask based on a corresponding computing node, it is detected that the resource status of the computing node does not meet the predetermined requirements or the execution progress of the subtask is abnormal, then a computing node is re-matched for the subtask and the subtask continues to be executed based on the re-matched computing node. After all subtasks are completed, the large model training task is summarized and validated, and the occupied computing nodes are released.

2. The method for orchestrating large model training tasks based on heterogeneous resource pools according to claim 1, characterized in that, The step of constructing a heterogeneous resource pool containing different types of computing nodes further includes: Configure a predetermined number of CPU computing nodes, GPU computing nodes, and NPU computing nodes; High-speed networks are used to interconnect various CPU computing nodes, GPU computing nodes and NPU computing nodes to form a heterogeneous resource pool.

3. The method for orchestrating large model training tasks based on heterogeneous resource pools according to claim 2, characterized in that, The steps of monitoring the current resource status of each computing node and recording the historical performance data of each computing node further include: Set up a central management node and deploy a monitoring agent on each computing node; The central management node collects the current resource status and historical performance data of the corresponding computing nodes from the various deployed monitoring agents.

4. The method for orchestrating large model training tasks based on heterogeneous resource pools according to claim 3, characterized in that, The step of decomposing the large model training task into several corresponding sub-tasks and determining the computational resource requirements of each sub-task further includes: The large model training task is decomposed according to task type, computational complexity, or data scale to obtain several corresponding sub-tasks. Each subtask is labeled with its corresponding computing resource requirement characteristics to form a subtask list. The computing resource requirement characteristics include CPU-dependent, GPU-accelerated, and NPU-dedicated types.

5. The method for orchestrating large model training tasks based on heterogeneous resource pools according to claim 4, characterized in that, If task type is used as the dimension for task decomposition, then the large model training task is decomposed into data preprocessing subtask, model training subtask, and model evaluation subtask. And / or, If computational complexity is used as a dimension for task decomposition, then large model training tasks are decomposed into computationally intensive subtasks and I / O-intensive subtasks.

6. The method for orchestrating large model training tasks based on heterogeneous resource pools according to claim 3, characterized in that, The step of matching a computing node to each subtask by comprehensively considering the current load, historical performance, expected energy consumption of executing the corresponding subtask, and the computing resource requirements of the subtask further includes: Establish a task-resource matching model based on heuristic rules and machine learning algorithms; Based on the task-resource matching model, a computing node is matched for each subtask, and a subtask-node mapping table is generated. The subtask-node mapping table includes the mapping relationship between the subtask and the matched computing node, the subtask ID, the computing node IP, and the resource quota.

7. The method for orchestrating large model training tasks based on heterogeneous resource pools according to claim 6, characterized in that, The task-resource matching model is configured to follow the following principles: For CPU-dependent subtasks, priority should be given to assigning them to CPU computing nodes with better network bandwidth. For GPU-accelerated subtasks, priority is given to assigning them to GPU computing nodes with historically better performance. For NPU-specific subtasks, priority is given to assigning them to NPU computing nodes with historically better performance. For computing nodes of the same type, consider load balancing of computing resources. Under the premise of meeting the task execution time requirements, select computing nodes or combinations of computing nodes with lower energy consumption.

8. The method for orchestrating large model training tasks based on heterogeneous resource pools according to claim 7, characterized in that, The step of re-matching a computing node for the subtask and continuing to execute the subtask based on the re-matched computing node during the execution of the subtask on the corresponding computing node further includes: The pre-deployed monitoring agent detects the resource status and task execution progress of the computing nodes executing subtasks at a predetermined frequency and uploads the relevant information to the central management node. If the central management node detects that the resource utilization rate of the computing node exceeds the threshold or the execution time of the subtask exceeds the expected total execution time to a predetermined standard, the central management node sends a task stop execution instruction to the corresponding computing node and freezes the task process. Based on the task-resource matching model, the computing nodes for the subtask are rematched, and the task status data of the original computing node is migrated to the rematched computing node based on RDMA technology so that the subtask can continue to be executed.

9. The method for orchestrating large model training tasks based on heterogeneous resource pools according to claim 8, characterized in that, The steps of summarizing and validating the large model training task after all subtasks are completed, and releasing the occupied computing nodes, further include: The central management node sends process termination instructions to each computing node, releasing the computing resources of each computing node. It also destroys the task containers on the CPU computing nodes and reclaims memory and CPU cores through the Docker container management system.

10. The method for orchestrating large model training tasks based on heterogeneous resource pools according to claim 9, characterized in that, The steps of summarizing and validating the large model training task after all subtasks are completed, and releasing the occupied computing nodes, further include: Update the status list of the heterogeneous resource pool, mark the released compute nodes as available, and trigger a health check of the heterogeneous resource pool.

Images