A server GPU computing power intelligent allocation method for large model training

Abstract

The application discloses a server GPU computing power intelligent allocation method for large model training, and relates to the technical field of computer resource management. The method comprises the following steps: S1: determining the dependency relationship between each stage in the large model research and development workflow according to the large model research and development workflow defined by a directed acyclic graph, and obtaining a corresponding resource demand portrait matrix; S2: setting a dynamic priority resource reservation pool according to the resource demand portrait matrix, and generating a corresponding resource supply curve; S3: determining the short-term predicted demand corresponding to each task according to the actual resource consumption and the dynamic resource demand curve of each task on the critical path in the large model research and development workflow, comparing the short-term predicted demand with the resource supply curve, and dynamically adjusting the dynamic priority resource reservation pool. The application can adapt to the fluctuation and uncertainty of the training task operation, guarantee the progress of the critical path, and flexibly handle the excess resource demand.

Description

Technical Field

[0001] This invention relates to the field of computer resource management technology, specifically to a method for intelligent allocation of server GPU computing power for large model training. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, ultra-large-scale neural networks, represented by Large Language Models (LLMs) and multimodal large models, have made breakthroughs in many fields such as natural language processing, computer vision, and scientific computing. These large models typically contain billions or even hundreds of billions of parameters, and their training process demands enormous computing resources, especially the computing power of high-performance graphics processing units (GPUs). A single complete training session often requires hundreds or thousands of high-end GPUs to work together for weeks or even months, consuming a large amount of electrical energy and hardware resources.

[0003] To support such high-intensity training tasks, data centers generally adopt a GPU cluster architecture consisting of multiple servers. However, in actual operation, due to the high heterogeneity (such as different model structures, batch sizes, and communication modes), dynamism (such as elastic training and fault-tolerant restarts), and resource contention (multi-user or multi-task shared clusters) of large model training tasks, traditional GPU resource allocation strategies (such as static partitioning, round-robin scheduling, and priority-based preemption) are difficult to efficiently utilize limited hardware resources.

[0004] Although existing research has attempted to improve resource management capabilities by introducing containerization (such as Kubernetes + GPU plugins), job scheduling systems (such as Slurm and YARN), or dedicated frameworks (such as Ray and DeepSpeed), most of these solutions are based on rule-driven or simple heuristic strategies, lack deep modeling of the characteristics of large model training loads, and have failed to achieve joint optimization allocation of multi-dimensional resources such as computing power, GPU memory, and network bandwidth.

[0005] Chinese invention patent application CN114035941A discloses a resource scheduling system, method, and computing device. The resource scheduling system includes a central scheduler, a management component, and at least one node component. Each node component can detect local resource usage information. The management component can calculate the resource capacity of specified resources in each node component based on its resource usage information. When the central scheduler receives a scheduling request for a container group, it can determine the target node component corresponding to the scheduling request based on the resource capacity of specified resources in each node component and schedule the container group to the target node component. In this scenario, the resource usage information of each node component can be used to calculate the resource capacity of specified resources in real time. Allocated but unused resources from various types of resources in the node components can be reclaimed and reused for container group scheduling, providing more resource capacity for container group scheduling, meeting the resource capacity requirements of various tasks, and enabling more flexible resource scheduling of each node component.

[0006] However, the above-mentioned and similar technical solutions still have the following shortcomings: In the large-scale model development process, tasks usually exist in the form of directed acyclic graphs containing multi-stage dependencies, covering data preprocessing, model training, hyperparameter search, evaluation and verification, and model export. However, when allocating resources in a cluster scheduling system, each stage is easily treated as an independent discrete task and processed in stages. This leads to a significant increase in the end-to-end completion time of the development pipeline due to discontinuous scheduling. At the same time, the cluster resource utilization rate fluctuates drastically between different stages, thus ignoring the dependencies between stages and changes in resource requirements. Summary of the Invention

[0007] The purpose of this invention is to provide a method for intelligent allocation of server GPU computing power for training large models, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for intelligent allocation of server GPU computing power for large model training, comprising:

[0009] S1: Workflow parsing profile construction: Based on the large model R&D workflow defined by a directed acyclic graph, determine the dependencies between each stage in the large model R&D workflow, and obtain the corresponding resource requirement profile matrix based on the historical execution data of each stage.

[0010] S2: Continuous scheduling plan generation: Based on the resource demand profile matrix, a dynamic priority resource reservation pool is set up, and a corresponding resource supply curve is generated based on the directed acyclic graph structure corresponding to the large model R&D workflow, the dependency relationship between each stage, and the dynamic priority resource reservation pool.

[0011] S3: Dynamic execution of elastic management: Based on the actual resource consumption and dynamic resource demand curves of each task on the critical path in the large model R&D workflow, the short-term predicted demand for each task is determined. At the same time, the short-term predicted demand is compared with the resource supply curve, and the dynamic priority resource reservation pool is dynamically adjusted according to the comparison results.

[0012] Furthermore, obtain the corresponding resource demand profile matrix, including:

[0013] S1.1: Workflow parsing: Using a workflow description file defined in a structured format, the corresponding graph data structure library is determined, including node objects, edge objects, and graph objects;

[0014] S1.2: Archive Retrieval: Collect the corresponding task execution data through the data acquisition agent set on the computing node, and obtain the corresponding historical execution archive based on the task execution data. At the same time, perform a similarity search in the historical execution archive based on the static declaration and context features corresponding to the new task stage to determine the corresponding historical task execution data.

[0015] S1.3: Matrix Construction: Using the historical task execution data, the set time series prediction model is trained to obtain the trained time series prediction model. At the same time, the configuration parameters corresponding to the new task stage are used as the input of the trained time series prediction model, and the corresponding dynamic resource demand curve and confidence interval are output. Simultaneously, according to the user declared resource requests corresponding to the new task stage, the corresponding static declaration baseline is set. According to the task type, model configuration and data characteristics corresponding to the new task stage, the corresponding context fingerprint is set. The dynamic resource demand curve, confidence interval, static declaration baseline and context fingerprint are encapsulated to obtain the corresponding resource demand profile matrix.

[0016] Furthermore, by using the data acquisition agent set on the computing node, task execution data is collected and the corresponding resource consumption time-series data and context execution metadata are determined. At the same time, the resource consumption time-series data and context execution metadata are associated according to the task ID-stage ID combination to obtain the corresponding historical execution file.

[0017] Furthermore, the corresponding resource supply curves are generated, including:

[0018] S2.1: Joint Modeling: Through the resource demand profile matrix, the directed acyclic graph structure, and the dependencies between each stage, resource-time joint extrapolation is performed;

[0019] S2.2: Aggregation Detection: Based on the resource-time joint extrapolation results and the time window-based overlap detection algorithm, parallelizable task pairs with time overlap are identified. At the same time, based on the discrete time points and the start and end times of each task, the execution tasks corresponding to the discrete time points are identified. The predicted total demand corresponding to the discrete time points is determined through a two-dimensional array of time-resource planes. Based on the predicted total demand, a global predicted resource demand curve is constructed.

[0020] S2.3: Path Synthesis: Based on the time series corresponding to the discrete time points, the dynamic resource demand curves of each task on the critical path in the directed acyclic graph are spliced ​​together to obtain the corresponding resource supply curve, and a priority resource reservation pool is created.

[0021] Furthermore, resource-time joint simulations are conducted, including:

[0022] S2.1.1: Timeline Establishment: Based on the resource demand profile matrix, determine the corresponding expected execution time, and through the directed acyclic graph structure, determine the start time of all initial tasks without upstream dependencies. At the same time, combine the expected execution time and the start time to determine the earliest end time of the initial task, and based on the earliest end time and the dependency relationship between each stage, determine the start time and earliest end time of each task.

[0023] S2.1.2: Constraint Modeling: Based on the directed acyclic graph structure and the dependencies between stages, set time constraints between tasks, and add preset delay time and / or preset cache time to the time constraints according to preset delay time and / or preset cache time.

[0024] S2.1.3: Demand Mapping: Based on the planning time range corresponding to the large model R&D workflow, set discrete time points, and determine the resource demand corresponding to each discrete time point through the resource demand profile matrix. At the same time, construct a two-dimensional array of time-resource plane based on the discrete time points and the corresponding resource demand.

[0025] Furthermore, a priority resource reservation pool is created, including:

[0026] S2.3.1: Path Determination: Using the directed acyclic graph structure, all tasks without downstream dependencies are identified. Based on the start time and earliest end time of each task, the latest start time and latest end time of each task are set. At the same time, based on the earliest end time and latest end time of each task, the corresponding end time difference is determined. Based on the dependency relationship between node objects, all tasks with an end time difference of 0 are connected in sequence to obtain the corresponding critical path.

[0027] S2.3.2: Curve Synthesis: Through the critical path, determine the dynamic resource demand curve corresponding to each task in the critical path, and set the start time and earliest end time of the dynamic resource demand curve according to the start time and earliest end time of each task. At the same time, the dynamic resource demand curves are spliced ​​in chronological order to obtain the corresponding resource supply curve. Set time-resource blocks to determine the quantity and type of resources guaranteed for each time interval. Update the real-time status of all physical resources in the cluster, and combine all the real-time updated physical resources to construct the corresponding dynamic priority resource reservation pool.

[0028] Furthermore, based on the earliest end time corresponding to the end task, the latest end time corresponding to the end task is set, and the latest end time and the expected execution time are combined to determine the latest start time corresponding to the end task. At the same time, based on the latest start time and the dependency relationship between each stage, the latest start time and the latest end time corresponding to each task are determined.

[0029] Furthermore, the dynamic priority resource reservation pool is dynamically adjusted, including:

[0030] S3.1: Conflict Diagnosis: Based on discrete time points, the global predicted resource demand curve and the resource supply curve are superimposed and compared to determine the predicted total demand and non-critical path reserved demand corresponding to the discrete time points. Simultaneously, the predicted total demand is compared with the total physical resources to determine the corresponding resource conflict points, specifically:

[0031] When the predicted total demand is not greater than the total physical resources, the corresponding discrete time point is not a resource conflict point; otherwise, the corresponding discrete time point is a resource conflict point, and the corresponding excess amount is determined based on the resource difference between the predicted total demand and the total physical resources.

[0032] S3.2: Cost assessment: Based on the resource conflict points, determine the corresponding non-critical path execution tasks and obtain all replacement strategies corresponding to the non-critical path execution tasks;

[0033] S3.3: Strategy Determination: Compare the costs of all strategies corresponding to the resource conflict points, determine the minimum cost, and the replacement strategy corresponding to the minimum cost is the optimal executable strategy. At the same time, the dynamic priority resource reservation pool is dynamically adjusted according to the optimal executable strategy.

[0034] Furthermore, based on the critical path reserved demand, non-critical path reserved demand, and predicted total demand corresponding to the resource conflict point, the critical path demand ratio and non-critical path demand ratio corresponding to the resource conflict point are determined. At the same time, the critical path demand ratio and non-critical path demand ratio are compared to determine the one with the larger ratio, which is the main conflict corresponding to the resource conflict point.

[0035] Furthermore, obtain all replacement strategies corresponding to the non-critical path execution tasks, including:

[0036] S3.2.1: Delayed Scheduling: Based on the resource requirement profile matrix corresponding to each task on the non-critical path, the corresponding total floating time is extracted. Simultaneously, through the global scheduling plan, the corresponding task delay time is obtained. The feasibility of the global scheduling plan is determined by comparing the total floating time and the task delay time. Specifically:

[0037] When the task delay time is greater than the extracted total floating time, the corresponding global scheduling plan is not feasible; otherwise, the corresponding global scheduling plan is feasible, and the delay scheduling cost corresponding to the global scheduling plan is the corresponding task delay time.

[0038] S3.2.2: Resource Condensation: Using a preset performance predictor, obtain the updated estimated execution time of the non-critical path execution task corresponding to the resource conflict point, and determine the corresponding task performance cost based on the time difference between the updated estimated execution time and the expected execution time.

[0039] S3.2.3: Preemptive Migration: Combine the checkpoint saving time, task start and stop time and data migration time of the non-critical path execution task corresponding to the resource conflict point to obtain the corresponding effective loss time and determine the corresponding preemptive migration cost.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] Firstly, during the execution phase of the plan, this invention can monitor the actual resource consumption of critical path tasks in real time and compare it with the prediction curve to make short-term demand predictions. At the same time, when it detects that the resource demand exceeds the reserved supply, it automatically diagnoses the conflict point and dynamically adjusts the dynamic priority resource reservation pool for non-critical path tasks by using cost evaluation. This enables the system to adapt to the fluctuations and uncertainties during the training task, ensuring the progress of the critical path while flexibly handling the excess resource demand.

[0042] Secondly, this invention analyzes the large model development workflow and combines it with historical execution data from each task stage to construct a resource demand profile matrix that includes dynamic resource demand curves, confidence intervals, static declaration baselines, and context fingerprints. This enables the system to accurately predict the resource consumption patterns of future tasks, rather than relying solely on users' rough static declarations, thus laying the foundation for refined scheduling.

[0043] Thirdly, this invention identifies the critical path that determines the overall workflow completion time by performing resource-time joint simulation of the workflow. When generating scheduling plans, resources are reserved for tasks on the critical path in advance, forming a resource supply curve. This ensures that the resource supply of critical tasks is not disturbed, reduces pipeline blockage caused by resource competition or scheduling discontinuity, and effectively shortens the end-to-end completion time of the entire R&D process.

[0044] Fourthly, this invention analyzes the overlap of resource requirements of all tasks in the workflow over time by aggregation detection, and constructs a global predictive resource requirement curve. This enables the system to coordinate resource allocation from a global perspective, which helps to smooth cluster load, avoid drastic fluctuations in resource requirements between different task stages, and thus reduce resource fragmentation and idleness, and improve the overall utilization of expensive computing resources such as GPUs. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the intelligent allocation method for server GPU computing power in this invention.

[0046] Figure 2 This is a flowchart illustrating the portrait construction method for workflow parsing in this invention;

[0047] Figure 3 This is a schematic diagram of the resource-time joint extrapolation process in this invention;

[0048] Figure 4 This is a flowchart illustrating the continuous scheduling plan generation method in this invention.

[0049] Figure 5 This is a schematic diagram illustrating the creation process of the priority resource reservation pool in this invention;

[0050] Figure 6 This is a flowchart illustrating the dynamic execution method of flexible management in this invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] refer to Figure 1 This embodiment provides a method for intelligent allocation of server GPU computing power for large model training. The method specifically includes the following steps:

[0053] Step S1: Workflow Analysis and Portfolio Construction. This involves determining the dependencies between different stages (such as data preprocessing and model training) of the large-scale model development workflow, based on a directed acyclic graph (DAG). Specifically, each node in the DAG represents a corresponding development stage, and edges represent dependencies and data flow relationships between stages. In other words, the static resource requirements and dependencies of each stage are determined through the nodes and edges set in the DAG.

[0054] Furthermore, based on the historical execution data corresponding to each stage, a corresponding resource demand profile matrix is ​​obtained. It is worth noting that the resource demand profile matrix in this embodiment includes basic resource demand, resource demand change pattern functions, and dynamic resource demand curves.

[0055] Step S2: Continuous scheduling plan generation. This involves setting up a corresponding dynamic priority resource reservation pool based on the obtained resource demand profile matrix. Simultaneously, based on the directed acyclic graph structure corresponding to the large-scale R&D workflow, the dependencies between stages, and the set dynamic priority resource reservation pool, dependency-aware resource-time joint extrapolation is performed to generate the corresponding resource supply curve.

[0056] Step S3: Dynamic Execution of Elastic Management. This involves obtaining the corresponding short-term forecasted demand based on the actual resource consumption and dynamic resource demand curves for each task on the critical path of the large-scale model development workflow. Simultaneously, the obtained short-term forecasted demand is compared with the resource supply curve generated in Step S2 to dynamically adjust the set dynamic priority resource reservation pool.

[0057] In this embodiment, a resource requirement profile matrix for each stage is obtained by analyzing the static resource requirements and dependencies corresponding to each stage of the large-scale model development workflow (defined as a directed acyclic graph). (Reference) Figure 2 This embodiment provides a workflow parsing profile construction method, which specifically includes the following steps:

[0058] Step S1.1: Workflow Parsing. This involves reading the workflow description file defined in a structured format (e.g., JSON / YAML) (and defining the development process of the workflow description file using a directed acyclic graph), and determining the corresponding graph data structure library, which includes node objects, edge objects, and graph objects. Specifically, each node object in the graph data structure library corresponds to a specific development stage (e.g., "Data Preprocessing - Node A", "Model Training - Node B"), and each node object stores a node ID, node type, and node attributes. Node attributes include the identifier of the corresponding stage, the executable program / script path, and configuration parameters. Simultaneously, each edge object in the graph data structure library stores a source node ID and a target node ID, corresponding to the dependencies between stages (e.g., "Node A → Node B" means that Node B depends on the output of Node A). The graph objects in the graph data structure library are used to manage all node and edge objects.

[0059] Furthermore, based on the node objects and their corresponding edge objects in the graph data structure library, the dependency relationships between two different node objects are determined, including direct dependencies and transitive dependencies. Direct dependencies are the direct connections between node objects, while transitive dependencies are the reachability between node objects. For example, if "node A → node B, node B → node C", then node C is transitively dependent on node A.

[0060] Furthermore, by analyzing the node attributes of each node object, the corresponding resource declaration fields are extracted and converted into the corresponding resource metric representation (for example, resolving "8" and "500GiB" in the resource declaration field resources:{gpu:8, memory:"500GiB"} into numerical values ​​and units). Simultaneously, the converted resource declaration fields are appended to the corresponding node object as the corresponding static declaration baseline.

[0061] Step S1.2: File Retrieval. This involves using a data acquisition agent set up on each compute node (e.g., server and container) to collect corresponding task execution data from operating system performance counters, GPU drivers (e.g., NVIDIA's nvidia-smi tool), running containers (e.g., Docker), and application frameworks (e.g., PyTorch's Profiler and TensorBoard). Simultaneously, based on the collected task execution data, the corresponding resource consumption time-series data and context execution metadata are determined. The resource consumption time-series data includes timestamps, task IDs, stage IDs, and corresponding GPU utilization, memory usage, CPU utilization, and network traffic. The context execution metadata includes the task ID of completed task stages, stage type, code version, hyperparameters (e.g., batch_size=32, learning_rate=0.001), input data characteristics (e.g., data volume and feature dimensions), start / end time, final status (success / failure), and queuing time. Based on the task ID-stage ID combination, the determined resource consumption time-series data and context execution metadata are associated to obtain the corresponding historical execution files.

[0062] Furthermore, based on the static declarations and contextual features corresponding to the new task stage, a similarity search is performed on the acquired historical execution archives to determine the corresponding historical task execution data. Specifically, through the static declarations and contextual features corresponding to the new task stage, relevant categorical features (including task type (e.g., "pre-training", "fine-tuning", "evaluation"), model framework (e.g., "PyTorch", "TensorFlow"), model architecture family (e.g., "ViT", "ResNet", "BERT"), numerical features (including hyperparameters (e.g., batch_size, learning_rate, num_epochs), input data scale (e.g., number of dataset samples, feature dimension)) and identifier features (including dataset name (e.g., "ImageNet-1K", "SQuAD 2.0"), specific version hash of the code repository, and base model checkpoint ID). Simultaneously, by using an embedding lookup table or word embedding model, the extracted categorical and identifier features are converted into corresponding dense numerical vectors. The extracted numerical features are then normalized or standardized (e.g., Z-score normalization, Min-Max normalization). The transformed categorical and identifier features are then concatenated or aggregated with the processed numerical features to obtain a high-dimensional feature vector corresponding to the new task stage. In this embodiment, the obtained high-dimensional feature vector is used as the query vector, and an approximate nearest neighbor search algorithm is used to determine the corresponding adjacent historical task feature vectors in the historical execution archive.

[0063] Step S1.3: Matrix Construction. This involves training the established time series prediction model (e.g., LSTM, TCN, Transformer model) using the adjacent historical task feature vectors determined in Step S1.2, obtaining the trained time series prediction model. Simultaneously, the configuration parameters corresponding to the new task stage are used as input to the trained time series prediction model, outputting the corresponding dynamic resource demand curve and confidence interval. Furthermore, a corresponding static declaration baseline is set based on the user-declared resource requests corresponding to the new task stage. Finally, a corresponding context fingerprint is set based on the task type, model configuration, and data characteristics corresponding to the new task stage.

[0064] Furthermore, by pre-setting a data structure, the obtained dynamic resource demand curves, confidence intervals, static declarative baselines, and context fingerprints are encapsulated to obtain a resource demand profile matrix corresponding to the new task stage. It is worth noting that the context fingerprint in the resource demand profile matrix is ​​equipped with an inverted index, for example, indexes are created for fields such as model_arch, dataset_name, and batch_size range.

[0065] In this embodiment, a dependency-aware resource-time joint extrapolation is performed using a resource demand profile matrix, the directed acyclic graph structure corresponding to the large model development workflow, and the dependencies between each stage, generating a corresponding resource supply curve. (Reference) Figures 3-5 This embodiment provides a method for generating continuous scheduling plans, which specifically includes the following steps:

[0066] Step S2.1: Joint Modeling. This involves performing dependency-aware resource-time joint modeling by using the resource requirement profile matrix, the directed acyclic graph structure corresponding to the large-scale model development workflow, and the dependencies between each stage. Specifically:

[0067] Step S2.1.1: Timeline Establishment. Based on the resource demand profile matrix obtained in Step S1.3, determine the corresponding expected execution time, i.e., the total span of the dynamic resource demand curve. Simultaneously, using the directed acyclic graph structure corresponding to the large-scale model development workflow, identify all initial tasks without upstream dependencies. Set the start time of each initial task to 0, and combine the start time of each initial task with its corresponding expected execution time to determine the earliest end time for each initial task. Also, based on the directed acyclic graph structure corresponding to the large-scale model development workflow, determine the start time and earliest end time for the remaining tasks. The start time of the remaining tasks is the earliest end time of their upstream tasks.

[0068] Step S2.1.2: Constraint Modeling. This involves determining the corresponding time constraints based on the directed acyclic graph structure corresponding to the large-scale R&D workflow and the dependencies between each stage. In other words, based on the dependencies between tasks (e.g., "Node A → Node B" means Node B depends on the output of Node A), time constraints are set between tasks, meaning the start time of task B corresponding to Node B cannot be less than the completion time of task B corresponding to Node A. It is worth noting that during the time constraint setting process, preset delay times and / or preset cache times can be used to implement corresponding time constraints. That is, preset delay times and / or preset cache times are added to the time constraints between tasks.

[0069] Step S2.1.3: Demand Mapping. This involves setting corresponding discrete time points based on the planned time range of the large-scale R&D workflow. Simultaneously, based on the resource demand profile matrix for each task, the resource demand for each discrete time point is determined. In other words, a corresponding two-dimensional time-resource plane array is constructed based on each discrete time point and its corresponding resource demand. The row index in the time-resource plane array represents the discrete time point, and the column vector represents the resource demand.

[0070] Step S2.2: Aggregation Detection. Based on the start and end times of each task determined in Step S2.1.1, the time constraints set in Step S2.1.2, and the two-dimensional time-resource plane array constructed in Step S2.1.3, an overlap detection algorithm based on time windows is used to compare each task in the large model development workflow pairwise, identify task pairs with time overlap, and mark these task pairs as parallelizable task pairs.

[0071] Furthermore, based on the discrete time points set in step S2.1.3 and the start and end times corresponding to each task, the execution tasks corresponding to each discrete time point are determined. That is, the start and end times of each execution task contain the corresponding discrete time points. Simultaneously, using a two-dimensional time-resource plane array, the predicted demand corresponding to each execution task at each discrete time point is obtained, and the predicted demand corresponding to each execution task is combined to determine the total predicted demand for each discrete time point. In other words, based on each discrete time point and its corresponding total predicted demand, a corresponding global predicted resource demand curve is constructed.

[0072] Step S2.3: Path Synthesis. This involves stitching together the dynamic resource demand curves of each task on the critical path of the directed acyclic graph based on the time series corresponding to discrete time points to obtain the corresponding resource supply curve. Simultaneously, a corresponding priority resource reservation pool is created based on the obtained resource supply curve. Details are as follows:

[0073] Step S2.3.1: Path Determination. This involves identifying all terminating tasks without downstream dependencies using the directed acyclic graph structure corresponding to the large-scale model's R&D workflow. Based on the start and earliest finish times of each task determined in Step S2.1.1, a latest finish time is set for each terminating task; that is, the earliest finish time determined in Step S2.1.1 is set as the corresponding latest finish time. Simultaneously, the latest finish time of each initial task is combined with its expected execution time to determine the latest start time for each terminating task. The difference between the latest finish time and the expected execution time is the corresponding latest start time. Furthermore, based on the directed acyclic graph structure corresponding to the large-scale model's R&D workflow, the latest finish and latest start times for the remaining tasks are determined. The latest finish time of the remaining tasks is the latest start time for their downstream tasks.

[0074] Furthermore, based on the earliest and latest finish times for each task, the difference in finish time for each task is determined. Then, based on these differences, the task with a finish time difference of 0 is identified—that is, the task with a total float of zero. Finally, based on the dependencies between node objects (i.e., tasks), all tasks with a total float of zero are connected to obtain the corresponding critical path.

[0075] Step S2.3.2: Curve Synthesis. Based on the critical path obtained in Step S2.3.1, determine the dynamic resource demand curve corresponding to each task in the critical path. Simultaneously, based on the start time and earliest end time of each task, set the start time and earliest end time of the corresponding dynamic resource demand curve. In other words, set the start time of the task to the start time of the corresponding dynamic resource demand curve, and set the earliest end time of the task to the earliest end time of the corresponding dynamic resource demand curve.

[0076] Furthermore, based on the start time and earliest end time corresponding to each dynamic resource demand curve, these curves are sequentially pieced together in chronological order. It's worth noting that during this sequential pairing process, when multiple tasks have resource demands at the same point in time, these demands are combined to obtain the total resource demand for that point in time. In other words, the corresponding resource supply curve is obtained through the sequentially pieced dynamic resource demand curves.

[0077] Furthermore, based on the obtained resource supply curve, multiple time-resource blocks are set. That is, by setting the time-resource blocks, the quantity and type of resources to be guaranteed for each time interval are determined (for example, in the interval [10:00, 10:30], 8 A100 GPUs need to be continuously reserved, and each GPU has no less than 40GB of memory). Simultaneously, based on the real-time status (including idle, allocated, and reserved) of all physical resources in the cluster (such as servers, GPUs, and memory), the real-time status of the corresponding physical resources is updated according to the resource quantity and type corresponding to the time-resource blocks, and the corresponding reservation effective time is set for the updated real-time status according to the time interval corresponding to the time-resource blocks. Finally, all physical resources with updated real-time statuses are combined to construct a corresponding dynamic priority resource reservation pool.

[0078] In this embodiment, the short-term predicted demand for each task on the critical path and the task demand for each task on the non-critical path are combined and compared with the resource supply curve obtained in step S2.3.2 to dynamically adjust the set dynamic priority resource reservation pool. (Reference) Figure 6 This embodiment provides a dynamic execution method for flexible management, which specifically includes the following steps:

[0079] Step S3.1: Conflict Diagnosis. This involves overlaying and comparing the global predicted resource demand curve from Step S2.2 with the resource supply curve obtained in Step S2.3.2, based on the set discrete time points. Specifically, the resource supply curve is used to obtain the critical path reserved demand for each discrete time point. Simultaneously, the global predicted resource demand curve is used to obtain the predicted total demand for each discrete time point. In other words, the non-critical path reserved demand for each discrete time point is determined by the demand difference between the predicted total demand and the critical path reserved demand.

[0080] Furthermore, by obtaining the total physical resources of the cluster, the predicted total demand at each discrete time point on the global predicted resource demand curve is detected. In other words, the predicted total demand at each discrete time point is compared with the total physical resources to determine the corresponding resource conflict points. Specifically:

[0081] When the predicted total demand is no greater than the total physical resources, the corresponding discrete time point is not a resource conflict point. Conversely, when the predicted total demand is greater than the total physical resources, the corresponding discrete time point is a resource conflict point. Simultaneously, the excess amount is determined based on the resource difference between the predicted total demand and the total physical resources.

[0082] Furthermore, based on the critical path reserved demand, non-critical path reserved demand, and projected total demand corresponding to each resource conflict point, the critical path demand ratio and non-critical path demand ratio for each resource conflict point are determined. These determined critical path demand ratios and non-critical path demand ratios are then compared to identify the one with the larger ratio. The one with the larger ratio is the primary conflict for the resource conflict point. In other words, when the critical path demand ratio is larger, the critical path demand is the primary conflict for the resource conflict point. When the non-critical path demand ratio is larger, the non-critical path demand is the primary conflict for the resource conflict point.

[0083] Step S3.2: Cost Assessment. Based on the resource conflict points identified in Step S3.1, determine the non-critical path execution tasks corresponding to the resource conflict points, obtain all replacement strategies for the non-critical path execution tasks, and determine the optimal replacement strategy from all replacement strategies. Specifically:

[0084] Step S3.2.1: Delayed Scheduling. This involves extracting the total float time based on the resource requirement profile matrix for each task on the non-critical path. Simultaneously, the corresponding task delay time is obtained through the set global scheduling plan. The obtained task delay time and the extracted total float time are compared, and the feasibility of the global scheduling plan is determined based on the comparison result. Specifically:

[0085] If the obtained task delay time is greater than the extracted total floating time, the corresponding global scheduling plan is infeasible and will be deleted. Conversely, if the obtained task delay time is not greater than the extracted total floating time, the corresponding global scheduling plan is feasible and will be retained.

[0086] In other words, based on the retained global scheduling plans, the task delay time corresponding to each retained global scheduling plan is obtained, and the corresponding task delay time is used as the delay scheduling cost of the corresponding global scheduling plan.

[0087] Step S3.2.2: Resource Condensation. This involves using a pre-defined performance predictor (based on historical benchmark data, scaling laws such as Amdahl's Law, or a trained regression model) to predict the execution time of tasks on non-critical paths corresponding to resource conflict points, thus determining the updated expected execution time for these tasks. Simultaneously, the updated expected execution time for non-critical path tasks is compared with the anticipated execution time to determine the corresponding time difference, which represents the corresponding task performance cost.

[0088] Step S3.2.3: Preemptive Migration. This involves determining the required time to save the complete checkpoint to persistent storage based on the runtime environment of the non-critical path task corresponding to the resource conflict point; this is the checkpoint saving time. Simultaneously, by analyzing the historical execution archives of the non-critical path, similar historical task execution data for the non-critical path task is identified, and the corresponding task start and stop times are obtained based on this data. Furthermore, based on the executable file size of the non-critical path task and the available bandwidth of the current cluster network, the network transmission time for migrating the executable file from the resource conflict point to the target node is obtained, and the corresponding data migration time is set based on the obtained network transmission time.

[0089] Furthermore, by combining the obtained checkpoint retention time, task start / stop time, and data migration time, the corresponding effective loss time is obtained, and based on the obtained effective loss time, the corresponding preemptive migration cost is determined.

[0090] Step S3.3: Strategy Determination. Based on the delay scheduling cost obtained in Step S3.2.1, determine the delay scheduling cost corresponding to each delay strategy. Based on the task performance cost obtained in Step S3.2.2, determine the task performance cost corresponding to each compaction strategy. Based on the preemption migration cost obtained in Step S3.2.3, determine the preemption migration cost corresponding to each preemption strategy.

[0091] Furthermore, the costs of all strategies corresponding to the resource conflict point (i.e., delay scheduling cost, task performance cost, and preemption migration cost) are compared to determine the minimum cost. The strategy corresponding to the minimum cost is the optimal executable strategy for the resource conflict point. In other words, when the strategy corresponding to the minimum cost is a delay strategy, the optimal executable strategy is a delay strategy. When the strategy corresponding to the minimum cost is a compact strategy, the optimal executable strategy is a compact strategy. When the strategy corresponding to the minimum cost is a preemption strategy, the optimal executable strategy is a preemption strategy. Simultaneously, based on the determined optimal executable strategy, the dynamic priority resource reservation pool constructed in step S2.3.2 is dynamically adjusted.

[0092] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended embodiments and their equivalents.

Claims

1. A method for intelligent allocation of server GPU computing power for large model training, characterized in that, Including: S1: Workflow parsing profile construction: Based on the large model R&D workflow defined by a directed acyclic graph, determine the dependencies between each stage in the large model R&D workflow, and obtain the corresponding resource requirement profile matrix based on the historical execution data of each stage. S2: Continuous scheduling plan generation: Based on the resource demand profile matrix, a dynamic priority resource reservation pool is set up, and a corresponding resource supply curve is generated based on the directed acyclic graph structure corresponding to the large model R&D workflow, the dependency relationship between each stage, and the dynamic priority resource reservation pool. S3: Dynamic Execution of Elastic Management: Based on the actual resource consumption and dynamic resource demand curves corresponding to each task on the critical path of the large-scale model R&D workflow, the short-term predicted demand for each task is determined. Simultaneously, the short-term predicted demand is compared with the resource supply curve, and based on the comparison results, the dynamic priority resource reservation pool is dynamically adjusted, including: S3.1: Conflict Diagnosis: Based on discrete time points, the global predicted resource demand curve and the resource supply curve are superimposed and compared to determine the predicted total demand and non-critical path reserved demand corresponding to the discrete time points. Simultaneously, the predicted total demand is compared with the total physical resources to determine the corresponding resource conflict points, specifically: When the predicted total demand is not greater than the total physical resources, the corresponding discrete time point is not a resource conflict point; otherwise, the corresponding discrete time point is a resource conflict point, and the corresponding excess amount is determined based on the resource difference between the predicted total demand and the total physical resources. S3.2: Cost assessment: Based on the resource conflict points, determine the corresponding non-critical path execution tasks and obtain all replacement strategies corresponding to the non-critical path execution tasks; S3.3: Strategy Determination: Compare the costs of all strategies corresponding to the resource conflict points, determine the minimum cost, and the replacement strategy corresponding to the minimum cost is the optimal executable strategy. At the same time, the dynamic priority resource reservation pool is dynamically adjusted according to the optimal executable strategy.

2. The intelligent allocation method for server GPU computing power for large model training according to claim 1, characterized in that, Obtain the corresponding resource demand profile matrix, including: S1.1: Workflow parsing: Using a workflow description file defined in a structured format, the corresponding graph data structure library is determined, including node objects, edge objects, and graph objects; S1.2: Archive Retrieval: Collect the corresponding task execution data through the data acquisition agent set on the computing node, and obtain the corresponding historical execution archive based on the task execution data. At the same time, perform a similarity search in the historical execution archive based on the static declaration and context features corresponding to the new task stage to determine the corresponding historical task execution data. S1.3: Matrix Construction: Using the historical task execution data, the set time series prediction model is trained to obtain the trained time series prediction model. At the same time, the configuration parameters corresponding to the new task stage are used as the input of the trained time series prediction model, and the corresponding dynamic resource demand curve and confidence interval are output. Simultaneously, according to the user declared resource requests corresponding to the new task stage, the corresponding static declaration baseline is set. According to the task type, model configuration and data characteristics corresponding to the new task stage, the corresponding context fingerprint is set. The dynamic resource demand curve, confidence interval, static declaration baseline and context fingerprint are encapsulated to obtain the corresponding resource demand profile matrix.

3. The intelligent allocation method for server GPU computing power for large model training according to claim 2, characterized in that, By using the data acquisition agent set on the computing node, task execution data is collected and the corresponding resource consumption time-series data and context execution metadata are determined. At the same time, the resource consumption time-series data and context execution metadata are associated according to the task ID-stage ID combination to obtain the corresponding historical execution file.

4. The intelligent allocation method for server GPU computing power for large model training according to claim 1, characterized in that, Generate the corresponding resource supply curves, including: S2.1: Joint Modeling: Through the resource demand profile matrix, the directed acyclic graph structure, and the dependencies between each stage, resource-time joint extrapolation is performed; S2.2: Aggregation Detection: Based on the resource-time joint extrapolation results and the time window-based overlap detection algorithm, parallelizable task pairs with time overlap are identified. At the same time, based on the discrete time points and the start and end times of each task, the execution tasks corresponding to the discrete time points are identified. The predicted total demand corresponding to the discrete time points is determined through a two-dimensional array of time-resource planes. Based on the predicted total demand, a global predicted resource demand curve is constructed. S2.3: Path Synthesis: Based on the time series corresponding to the discrete time points, the dynamic resource demand curves of each task on the critical path in the directed acyclic graph are spliced ​​together to obtain the corresponding resource supply curve, and a priority resource reservation pool is created.

5. The intelligent allocation method for server GPU computing power for large model training according to claim 4, characterized in that, Conduct resource-time joint simulations, including: S2.1.1: Timeline Establishment: Based on the resource demand profile matrix, determine the corresponding expected execution time, and through the directed acyclic graph structure, determine the start time of all initial tasks without upstream dependencies. At the same time, combine the expected execution time and the start time to determine the earliest end time of the initial task, and based on the earliest end time and the dependency relationship between each stage, determine the start time and earliest end time of each task. S2.1.2: Constraint Modeling: Based on the directed acyclic graph structure and the dependencies between stages, set time constraints between tasks, and add preset delay time and / or preset cache time to the time constraints according to preset delay time and / or preset cache time. S2.1.3: Demand Mapping: Based on the planning time range corresponding to the large model R&D workflow, set discrete time points, and determine the resource demand corresponding to each discrete time point through the resource demand profile matrix. At the same time, construct a two-dimensional array of time-resource plane based on the discrete time points and the corresponding resource demand.

6. The intelligent allocation method for server GPU computing power for large model training according to claim 4, characterized in that, Create a priority resource reservation pool, including: S2.3.1: Path Determination: Using the directed acyclic graph structure, all tasks without downstream dependencies are identified. Based on the start time and earliest end time of each task, the latest start time and latest end time of each task are set. At the same time, based on the earliest end time and latest end time of each task, the corresponding end time difference is determined. Based on the dependency relationship between node objects, all tasks with an end time difference of 0 are connected in sequence to obtain the corresponding critical path. S2.3.2: Curve Synthesis: Through the critical path, determine the dynamic resource demand curve corresponding to each task in the critical path, and set the start time and earliest end time of the dynamic resource demand curve according to the start time and earliest end time of each task. At the same time, the dynamic resource demand curves are spliced ​​in chronological order to obtain the corresponding resource supply curve. Set time-resource blocks to determine the quantity and type of resources guaranteed for each time interval. Update the real-time status of all physical resources in the cluster, and combine all the real-time updated physical resources to construct the corresponding dynamic priority resource reservation pool.

7. The intelligent allocation method for server GPU computing power for large model training according to claim 6, characterized in that, Based on the earliest end time corresponding to the end task, the latest end time corresponding to the end task is set, and the latest end time and the expected execution time are combined to determine the latest start time corresponding to the end task. At the same time, based on the latest start time and the dependency relationship between each stage, the latest start time and the latest end time corresponding to each task are determined.

8. The intelligent allocation method for server GPU computing power for large model training according to claim 1, characterized in that, Based on the critical path reserved demand, non-critical path reserved demand, and predicted total demand corresponding to the resource conflict point, the critical path demand ratio and non-critical path demand ratio corresponding to the resource conflict point are determined. At the same time, the critical path demand ratio and non-critical path demand ratio are compared to determine the one with the larger ratio. The one with the larger ratio is the main conflict corresponding to the resource conflict point.

9. The intelligent allocation method for server GPU computing power for large model training according to claim 1, characterized in that, Obtain all replacement strategies corresponding to the non-critical path execution task, including: S3.2.1: Delayed Scheduling: Based on the resource requirement profile matrix corresponding to each task on the non-critical path, the corresponding total floating time is extracted. Simultaneously, through the global scheduling plan, the corresponding task delay time is obtained. The feasibility of the global scheduling plan is determined by comparing the total floating time and the task delay time. Specifically: When the task delay time is greater than the extracted total floating time, the corresponding global scheduling plan is not feasible; otherwise, the corresponding global scheduling plan is feasible, and the delay scheduling cost corresponding to the global scheduling plan is the corresponding task delay time. S3.2.2: Resource Condensation: Using a preset performance predictor, obtain the updated estimated execution time of the non-critical path execution task corresponding to the resource conflict point, and determine the corresponding task performance cost based on the time difference between the updated estimated execution time and the expected execution time. S3.2.3: Preemptive Migration: Combine the checkpoint saving time, task start and stop time and data migration time of the non-critical path execution task corresponding to the resource conflict point to obtain the corresponding effective loss time and determine the corresponding preemptive migration cost.

Images