A method for elastic resource adjustment in distributed data parallel training

Abstract

The present invention relates to a method for elastic resource adjustment in distributed data parallel training, comprising: step one: using a performance monitoring module to monitor job performance in real time; step two: comprehensively evaluating the resource status of the current cluster based on a preset fitness function evaluation and a reallocation penalty mechanism; when it is determined through the fitness function that not rescheduling will seriously affect the cluster performance, quickly starting the rescheduling process; step three: judging which jobs need to be subjected to which elastic operations based on continuous monitoring and evaluation of the fitness function; at the same time, using an intelligent node screening algorithm to select the most suitable nodes from an available node pool based on elastic adjustment targets; step four: executing an elastic scheduling strategy to actually perform elastic resource adjustment operations; and for different elastic operations of different jobs, implementing dynamic adjustment and optimal configuration of resources through an incremental communication topology reconstruction mechanism and a gradient cache and interpolation compensation mechanism.

Description

Technical Field

[0001] The present invention relates to the fields of distributed computing and deep learning optimization, and in particular to a method for adjusting elastic resources in distributed data parallel training. Background Art

[0002] In recent years, with the continuous growth of deep learning model parameter size and the exponential expansion of training data, distributed data parallel training has become an inevitable choice for accelerating model iteration. GPU clusters, with their parallel computing advantages, have become the core infrastructure supporting large-scale training tasks. Large-scale training tasks based on GPU clusters typically rely on multi-node collaborative computing, distributing the computing load to heterogeneous hardware through data parallel strategies. However, in cloud computing and multi-tenant scenarios, training tasks often face dynamic changes in resource supply: hardware failures, bid instance recycling, or load balancing strategies may trigger the real-time addition or exit of computing nodes. This elasticity requirement is in sharp conflict with the inherent strong state consistency requirements of distributed data training, exposing the key bottlenecks of traditional architectures.

[0003] Existing data distributed training frameworks are often designed with static resource configurations in mind, and their communication topologies and synchronization mechanisms lack adaptability to dynamic node changes. When a cluster scales up or down, the system must restructure inter-node communication paths and redistribute the computational load. This process not only interrupts the training process but also triggers cascading issues with the gradient synchronization mechanism. Specific issues with traditional elastic adjustment solutions in dynamic scaling scenarios include:

[0004] 1) Excessive topology reconstruction latency: Traditional communication architectures rely on a static communication ring topology design. Dynamic node changes (such as scaling or fault replacement) require global reconstruction of communication links. This process involves complex operations such as node discovery, path negotiation, and link reconfiguration. This forces the training process to be interrupted, resulting in idle computing resources and extended task execution cycles, severely limiting the cluster's elastic scalability.

[0005] 2) Insufficient state continuity: Existing solutions overly crudely handle training status and unfinished gradients at node exit, failing to fully consider differences in computational stages (such as interruptions during backpropagation) and the spatiotemporal correlation of gradients. Strategies that simply discard or average local gradients can easily cause parameter updates to drift, leading to fluctuations in model convergence and even the risk of divergence.

[0006] 3) Rigid resource allocation mechanisms: In a multi-tenant, multi-task cloud computing environment, traditional solutions struggle to achieve refined resource management. They typically allocate resources based on fixed ratios or priorities, failing to flexibly adjust to the actual operational status of jobs and the dynamic needs of tenants. This not only leads to irrational resource allocation but can also cause some jobs to starve due to insufficient resources, while others experience an oversupply, reducing overall resource utilization and tenant satisfaction. Summary of the Invention

[0007] In response to the shortcomings of the existing technology, the present invention provides an innovative elastic resource adjustment method in deep learning distributed data parallel training.

[0008] By proposing a dynamic resource allocation optimization mechanism based on the overall state of the cluster, combined with an incremental communication topology reconstruction mechanism, gradient caching, and asynchronous interpolation compensation, this approach effectively addresses many issues with existing technologies in dynamic elastic scaling. This enables efficient utilization of cluster resources and stable job execution, significantly improving training efficiency and model performance.

[0009] This invention mainly realizes the dynamic adjustment of resources in data distributed deep learning training by utilizing the following core technologies:

[0010] 1) Dynamic Communication Topology Reconstruction Method: To address the high latency caused by global reconstruction of the communication ring in traditional algorithms, a dynamic communication topology reconstruction method based on a topology cost model and an optimal insertion algorithm driven by reinforcement learning is proposed. Through progressive parameter synchronization, this method achieves rapid and efficient reconstruction of the communication topology when adding nodes, reducing training interruptions and latency and improving training efficiency.

[0011] 2) Gradient caching and asynchronous interpolation aggregation mechanism: To address the problem of inconsistent model parameter updates that may occur when exiting nodes during node reduction operations, a mechanism based on a gradient cache pool and gradient interpolation aggregation is proposed. By saving historical gradient data and using the gradient interpolation aggregation method to dynamically compensate for incomplete gradient calculations, training bias is avoided and the convergence and accuracy of the model are ensured.

[0012] 3) Resource Allocation Mechanism Based on Overall Cluster Status: To address the rigidity of traditional resource allocation mechanisms in multi-tenant, multi-tasking scenarios, we propose a resource allocation mechanism based on dynamic cluster performance evaluation. By monitoring the key performance indicators of each node in the cluster in real time, we dynamically adjust resource allocation strategies, enabling refined resource management and improving resource utilization.

[0013] The technical solution of the present invention is:

[0014] A method for elastic resource adjustment in distributed data parallel training, comprising:

[0015] Step 1: Use the performance monitoring module to monitor job performance in real time and collect key performance indicators including computing load, communication bandwidth, and GPU utilization;

[0016] Step 2: Based on the preset fitness function evaluation and the reallocation penalty mechanism, a comprehensive assessment is conducted on the current cluster resource status. If the fitness function determines that not rescheduling will seriously affect cluster performance, the rescheduling process is quickly initiated to begin resource elasticity adjustments.

[0017] Step 3: Based on continuous monitoring and evaluation of the adaptability function, determine which jobs require which elasticity operations. Simultaneously, based on the elasticity adjustment target, use an intelligent node selection algorithm to select the most suitable nodes from the available node pool.

[0018] Step 4: Execute the elastic scheduling strategy and perform elastic resource adjustments. Dynamic resource adjustment and optimal configuration are achieved through an incremental communication topology reconstruction mechanism and a gradient caching and interpolation compensation mechanism for different elastic operations for different jobs.

[0019] A method for elastic resource adjustment in distributed data parallel training, specifically comprising:

[0020] Step S100: The scheduling period arrives; according to the preset scheduling period, the resource scheduling evaluation process is periodically triggered;

[0021] Step S200: Job rescheduling evaluation: Use the adaptability function to quantitatively evaluate the resource allocation status of the current job; by calculating the gain ratio of each job, determine the efficiency of the current resource allocation and provide a quantitative basis for possible rescheduling operations;

[0022] Step S300: Determine whether the indicator exceeds the threshold; comprehensively consider traditional performance detection indicators and fitness function evaluation results, conduct a comprehensive threshold breakthrough judgment, and decide whether to perform rescheduling;

[0023] If the core performance indicators of the job exceed the preset threshold range, or if the adaptability function determines that not rescheduling will seriously affect the cluster performance, step S400 is executed to perform elastic adjustment through rescheduling; otherwise, the original allocation plan is maintained without rescheduling. The core performance indicators are traditional performance detection indicators, including: GPU memory occupancy, GPU computing utilization, communication delay and bandwidth bottleneck indicators;

[0024] The core performance indicators of the job exceed the preset threshold range, including:

[0025] The GPU memory usage remains above 90% for more than N evaluation cycles.

[0026] Alternatively, the GPU computing utilization rate is continuously monitored to be less than 30%, and the overall throughput of the training job on this GPU continues to decline;

[0027] Alternatively, the communication delay is more than twice the normal value, or the bandwidth utilization is less than 60%;

[0028] Step S400: Rescheduling is performed. Based on the evaluation result of the adaptability function, it is determined what elastic adjustment method should be applied to the rescheduled jobs. For jobs with high priority and urgent resource requirements, a scale-out operation will be implemented.

[0029] High priority refers to jobs that are explicitly marked in the scheduling policy as requiring priority for completion. Urgent resource requirements refer to jobs with significant performance constraints, as evidenced by: GPU utilization consistently exceeding 90%, a significant increase in mini-batch processing time, and a job's fitness function gain under existing resources being less than 85% of what would be achieved under the cluster's current optimal resource allocation.

[0030] For jobs with a gain ratio significantly lower than the cluster average and low resource utilization, the Scale In operation is performed. A gain ratio significantly lower than the cluster average means that the job's gain ratio under the current resource allocation is at least 40% lower than the average for jobs in the cluster. A low resource utilization means that the actual utilization level of the GPU resources used by the job is in a low load state for a long time.

[0031] Step S500: Scale Out operation: Using the incremental communication topology reconstruction mechanism, based on the topology cost model and reinforcement learning algorithm, the optimal communication path of the new node is determined to achieve efficient reconstruction of the communication topology;

[0032] Step S600: Scale In operation; using a gradient cache and asynchronous interpolation compensation mechanism, by saving historical gradient data and using an interpolation algorithm to dynamically compensate for incomplete gradient calculations.

[0033] According to a preferred embodiment of the present invention, the job rescheduling evaluation includes:

[0034] Step S201: Generate candidate allocation schemes; regularly analyze the overall resource allocation status of the cluster, including the number of available GPUs, , number of jobs , run the search algorithm according to the predetermined scheduling cycle to generate a series of candidate allocation matrices , each matrix represents a resource allocation scheme, and each resource allocation scheme must satisfy:

[0035] (1);

[0036] in, Indicates homework In the candidate allocation matrix The number of GPUs allocated in;

[0037] Step S202: Throughput model estimation: Establish a throughput prediction model for each job to predict the given resource configuration Based on historical data and real-time operation data, supervised learning is used to capture the complex mapping relationship between resource configuration parameters and cluster dynamic status;

[0038] Step S203: Calculate the job gain ratio; Based on the prediction results of the throughput model, calculate the gain ratio of each job under different resource configurations; For each job in the cluster, the gain ratio Defined as using a given resource allocation Throughput and fair resource allocation Throughput The ratio is used to quantitatively evaluate the efficiency of resource allocation; the gain ratio calculation formula is as follows:

[0039] (2);

[0040] Step S204: Redistribution penalty is imposed; when evaluating the fitness function, a penalty is imposed on jobs that need to reallocate resources; a penalty factor is set , based on the historical reallocation frequency of the job, the future reallocation cost is predicted using the following formula:

[0041] (3);

[0042] in, For homework Current running time; For homework The number of reallocations so far; It is the cost of redistribution, which is reflected in the delay time;

[0043] The penalty factor discounts the gain ratio of the current running configuration, namely:

[0044] (4);

[0045] Step S205: Adaptability function evaluation; for each candidate allocation matrix , through multiple iterations of calculating the fitness function for evaluation, continuously optimizing the resource allocation plan, and finally selecting the allocation matrix with the highest score to achieve the optimal configuration of cluster resources; the fitness function is defined as the generalized average value of the gain ratio of each job, and resource allocation is determined by maximizing the fitness function; the fitness function formula is for:

[0046] (5);

[0047] set up As a fairness regulator, by setting Dynamically balance the fairness and efficiency of resource allocation based on value; Indicates the total number of jobs.

[0048] Furthermore, for each candidate allocation scheme , perform resource configuration mapping; resource configuration mapping means: in each candidate allocation scheme Next, for each job Allocate the corresponding number of GPUs ;

[0049] For homework , load the optimal configuration from historical tuning records according to its task type , Indicates homework The batch size per GPU under the optimal parameter configuration; Indicates homework The optimal number of gradient accumulation steps;

[0050] At the same time, the performance data obtained in real time by the cluster monitoring system is used for dynamic parameter injection.

[0051] Furthermore, the optimized configuration and parameters are input into the throughput prediction model, batch inference is performed, and the throughput of each job under each configuration is predicted in parallel. This includes:

[0052] The throughput prediction model is expressed as:

[0053] ;

[0054] Indicates the throughput of the job under configuration; Represents the throughput prediction model, using a lightweight regression model;

[0055] For each job and each candidate allocation configuration , construct input features ;

[0056] Use the parallel inference framework to input the trained throughput prediction model , thus obtaining the predicted throughput of each job under each candidate configuration .

[0057] According to a preferred embodiment of the present invention, the Scale Out operation includes:

[0058] Step S501: elastic expansion is triggered;

[0059] Step S502: Extended resource evaluation: After receiving the extension request, a comprehensive evaluation is performed on the current cluster's resource status, including GPU usage, video memory usage, and network bandwidth utilization. The GPU with the highest compatibility with the current job and sufficient idle resources is determined as the new GPU.

[0060] Step S503: non-stop expansion initialization: after determining the newly added GPU resources, perform initialization operations;

[0061] Step S504: context preparation; context preparation for the newly added GPU node includes data preparation and environment configuration;

[0062] In the data preparation phase, based on the job's established data distribution strategy, some unprocessed data shards are rationally distributed so that they can be accurately transmitted to the newly added GPU nodes.

[0063] Environment configuration, including: deploying dynamic libraries and software dependencies on newly added GPU nodes;

[0064] Step S505: Reconstruct the communication topology; use the incremental communication topology construction mechanism to build a new communication topology for the computing nodes of the job;

[0065] Step S506: Progressive gradient synchronization: When adding a new GPU node to synchronize parameters, the local parameters of the new GPU node are synchronized. and global model parameters ; Generate difference parameters :

[0066] (6);

[0067] Step S507: The training job is resumed.

[0068] Preferably, according to the present invention, a new communication topology is constructed for the computing nodes of the job using an incremental communication topology construction mechanism; including:

[0069] Step S5051: Resource isolation and pre-check: Isolate the selected expanded GPU, memory, and network bandwidth resources, and perform node health checks to eliminate abnormal devices.

[0070] Step S5052: topology exploration: calling the NCCL topology discovery interface to obtain hardware topology information, including the physical connection relationship of all nodes in the cluster, GPU architecture, and network interface information;

[0071] Step S5053: Communication status awareness; real-time node acquisition through distributed probes Communication performance data, including the amount of data transferred in a single synchronization , real-time effective bandwidth , fixed communication delay and real-time depth of the communication queue ;

[0072] Step S5054: Communication cost modeling; based on the acquired hardware topology information and communication performance data, construct a topology cost model to define nodes Communication cost It is a multidimensional function consisting of data transmission time, fixed delay and instantaneous congestion. The formula is as follows:

[0073] (7);

[0074] in, is the dynamic weight coefficient, satisfying the constraint and ; The design logic is as follows:

[0075] (8);

[0076] Specifically, by minimizing the communication cost function The actual end-to-end delay collected in real time by distributed probes The mean square error of ;

[0077] Step S5055: Optimal insertion decision; using the optimal insertion algorithm driven by reinforcement learning, with the current communication topology and node information as the state, the location selection of the new node as the action, and the reduction of the communication cost as the reward function, the insertion point with the least global impact is selected from the candidate positions, that is, the lowest total cost increment; for the existing nodes in the topology , quantified cost increment Defined as inserting a new node The difference between the new link cost after disconnection and the communication cost before the original link is disconnected is calculated as follows:

[0078] (9);

[0079] Step S5056: Connectivity check: After the new communication topology is established, a simulated transmission test is performed to ensure that there are no loops or isolated nodes in the topology;

[0080] If there is no connectivity problem, the communication topology is switched directly; otherwise, step S1055 is executed again to re-build the optimal communication topology.

[0081] Furthermore, for the existence The topology of the nodes, the new node The optimal insertion position Determined by the following formula:

[0082] (10).

[0083] Preferably, according to the present invention, a gradient cache and asynchronous interpolation compensation mechanism is used to dynamically compensate for unfinished gradient calculations by saving historical gradient data and using an interpolation algorithm; including:

[0084] Step S601: elastic shrinkage triggering;

[0085] Step S602: Exit node screening; after receiving the reduction request, a comprehensive assessment of the current cluster's resource status is conducted, including indicators such as GPU utilization, video memory occupancy, and network bandwidth utilization; when selecting exit nodes, select those with low computing power utilization and low data migration costs as exit nodes; low computing power utilization means that the average GPU utilization rate during the statistical period is lower than the set threshold; this is defined as: GPU utilization is less than 30% and video memory occupancy is less than 40%; low data migration cost means that the communication cost and waiting time required to migrate unfinished data batches or model copies on a node to other nodes are short; specifically, this means that the data migration volume is less than 500MB and the network transmission delay is less than 10ms, or the node only participates in the training of low-priority, pausable tasks;

[0086] Step S603: consistency check: after determining the exit node, check whether the training job is at a gradient synchronization safety point, and verify the consistency of the calculation progress of the node to be removed with the global model version;

[0087] If yes, then execute step S604 and the node exits directly; otherwise, execute step S605 and calculate the compensation gradient of the exiting node;

[0088] Step S604: the node exits directly;

[0089] Step S605: Compensate gradient calculation: When the exiting node is in the process of calculating the gradient of the current mini-batch, the gradient interpolation compensation mechanism is activated. By fusing the historical cached gradient of the exiting node with the real-time gradient of the cluster, a compensation value is generated to replace the unfinished gradient calculation.

[0090] Step S606: asynchronous aggregation; after the compensation gradient is generated, the compensation gradient is quickly distributed and synchronized with the parameter update through a non-blocking communication mechanism;

[0091] Step S607: The training job is resumed.

[0092] Further preferably, the compensation gradient calculation includes:

[0093] Step S6051: Gradient cache acquisition; establish a gradient cache queue for each computing node, always save the latest and most likely to be called gradient data; obtain the latest cached gradient in the gradient cache pool of the exit node ;

[0094] Step S6052: real-time gradient aggregation; The current mini-batch gradient of each node Collect and calculate the gradient mean ;

[0095] Step S6053: Dynamic weight correction; introduction As a weight to balance historical gradients and real-time gradients; specifically, by setting the original progress value of the exit node at the current mini-batch Perform gradient similarity correction and adjust weight distribution:

[0096] (11);

[0097] in, , is the gradient difference vector; correction term Quantify the consistency between the historical gradient of the exiting node and the real-time gradient of the remaining nodes: Represents the L2 norm of the vector, which measures the magnitude of the gradient change;

[0098] Step S6054: Compensation gradient fusion; generate compensation gradient by weighted fusion of historical gradient information of exit node and current gradient information of remaining nodes; calculate compensation gradient by the following formula :

[0099] (12).

[0100] According to the preferred embodiment of the present invention, asynchronous aggregation includes:

[0101] Step S6061: Parallelization is triggered; execution permissions are allocated, and an independent thread pool is created to ensure parallel execution of gradient distribution and training threads;

[0102] Step S6062: Topology reconstruction; constructing a minimum spanning tree topology based on the remaining nodes, optimizing the gradient synchronization path; using incremental link switching, updating only the communication subgraph affected by the exiting node;

[0103] Step S6063: Compensate for gradient broadcast; use The interface will compensate for gradient sharding broadcasts; set up redundant transmission channels, and transmit in parallel on the primary and backup links;

[0104] Step S6064: data migration and backup;

[0105] Step S6065: resource decoupling and release;

[0106] Step S6066: Gradient aggregation. After receiving the compensation gradient, each remaining node immediately joins the gradient aggregation queue. The node aggregates the gradient calculated by itself and the received compensation gradient to form complete gradient information.

[0107] Step S6067: Parameter update and verification: Based on the aggregated gradients, each node updates the global model parameters;

[0108] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the elastic resource adjustment method in the above-mentioned distributed data parallel training when executing the computer program.

[0109] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the elastic resource adjustment method in the above-mentioned distributed data parallel training.

[0110] Compared with the prior art, the present invention has the following beneficial effects:

[0111] 1. Globally Optimal Dynamic Resource Allocation: By analyzing the overall cluster state in real time and introducing an adaptability function, this method dynamically optimizes resource allocation between tasks, ensuring optimal overall system throughput and performance. Compared to traditional resource allocation methods, this approach not only improves cluster efficiency but also ensures stable job execution.

[0112] 2. Efficient Reconfiguration of Dynamic Communication Topologies: By integrating a real-time communication cost model with a reinforcement learning-driven incremental insertion strategy, optimal topological path selection is achieved when nodes are dynamically added. Compared to the global reconstruction mechanism of traditional static communication architectures, this invention significantly shortens topology reconstruction latency, reduces training interruptions caused by link switching, and improves cluster expansion agility and resource utilization.

[0113] 3. Gradient Compensation Ensures Convergence Stability: Based on a gradient cache pool and Bayesian interpolation fusion mechanism, the weights of historical and real-time gradients are balanced when nodes are removed, generating a smooth transition of compensation gradients. This design effectively avoids sudden changes in parameter update direction, ensuring model convergence stability during dynamic resource adjustments, and overcoming the risk of model divergence caused by gradient loss or extensive compensation in traditional solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0115] Figure 1 This is an overall flow chart of the elastic resource adjustment method in distributed data parallel training according to an embodiment of the present invention;

[0116] Figure 2 This is a flow chart of job rescheduling evaluation based on the overall state of the cluster according to an embodiment of the present invention;

[0117] Figure 3 This is a flowchart of dynamic node expansion based on incremental topology reconstruction according to an embodiment of the present invention;

[0118] Figure 4 Schematic diagram of the incremental communication topology reconstruction process according to an embodiment of the present invention

[0119] Figure 5 This is a flowchart of a node dynamic reduction based on asynchronous gradient interpolation compensation according to an embodiment of the present invention;

[0120] Figure 6 This is a schematic diagram of the compensation gradient calculation process according to an embodiment of the present invention;

[0121] Figure 7 This is a schematic diagram of the asynchronous gradient aggregation process according to an embodiment of the present invention;

[0122] Figure 8 This is a diagram of the experimental results of the deep learning training task in an embodiment of the present invention. DETAILED DESCRIPTION

[0123] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0124] Explanation of terms:

[0125] 1. Scale-out: This refers to a resource expansion method used during distributed training to increase overall computing power or training throughput by adding computing nodes (such as GPUs or servers). Scale-out is typically used to address the increased computing demands of large-scale models or datasets. The key lies in properly scheduling the newly added nodes and reconfiguring the communication topology to maintain training efficiency and convergence speed.

[0126] 2. Scale-in (horizontal contraction): This refers to the process of releasing some of the previously allocated compute nodes during a training job based on changes in resource utilization, task load, or system policies, thereby reducing resource usage and scheduling costs. Scale-in typically requires dynamic adjustments to parameter synchronization mechanisms and communication structures to ensure training consistency and performance stability during node contraction.

[0127] 3. NCCL topology: This refers to the logical topology structure for efficient communication between GPUs, built and managed by the NVIDIA Collective Communications Library (NCCL). NCCL supports multiple topology construction modes, such as Ring, Tree, and Hierarchical. The system automatically selects or manually configures the optimal communication path based on the hardware layout (such as the number of GPUs and PCIe / IB ​​topology). A reasonable NCCL topology is crucial for improving bandwidth utilization and reducing communication latency in multi-GPU training.

[0128] 4. Mini-batch: In deep learning model training, mini-batch refers to dividing the complete training data into multiple smaller subsets, using one of these subsets for forward and backward propagation in each iteration. The mini-batch mechanism improves training efficiency and reduces memory consumption while ensuring model convergence. In distributed training, multiple devices can process different mini-batches in parallel, and model updates are completed through parameter synchronization.

[0129] 5. NCCL ibcast interface: This is the communication interface in the NCCL library used to implement non-blocking broadcast operations. ibcast allows data from a single source node to be broadcast to multiple destination nodes, returning control immediately after the call, enabling asynchronous overlapping of computation and communication. This interface is commonly used to broadcast model parameters, optimizer status, and other functions in distributed training. When combined with asynchronous communication mechanisms, it can significantly improve overall training performance.

[0130] Example 1

[0131] A method for elastic resource adjustment in distributed data parallel training, comprising:

[0132] Step 1: Use the performance monitoring module to monitor job performance in real time, collecting key performance indicators including computing load, communication bandwidth, and GPU utilization; providing data support for subsequent resource evaluation and adjustment decisions.

[0133] Step 2: Based on the preset fitness function evaluation and the reallocation penalty mechanism, a comprehensive assessment is conducted on the current cluster resource status. If the fitness function determines that not rescheduling will seriously affect cluster performance, the rescheduling process is quickly initiated to begin resource elasticity adjustments.

[0134] Step 3: Based on continuous monitoring and evaluation of the adaptability function, scientifically determine which jobs require which elasticity operations. Simultaneously, based on the elasticity adjustment target, use an intelligent node screening algorithm to select the most suitable nodes from the available node pool, ensuring the efficiency and rationality of the adjustment operations.

[0135] Step 4: Execute the elastic scheduling strategy and perform elastic resource adjustments. For different elastic operations for different jobs, dynamic resource adjustment and optimal configuration are achieved through incremental communication topology reconstruction mechanisms and gradient caching and interpolation compensation mechanisms, ensuring efficient and stable model training.

[0136] Example 2

[0137] The difference between the elastic resource adjustment method in distributed data parallel training described in Example 1 is that:

[0138] refer to Figure 1 , a method for elastic resource adjustment in distributed data parallel training, specifically comprising:

[0139] Step S100: The scheduling period arrives; according to the preset scheduling period, the resource scheduling evaluation process is periodically triggered to provide an opportunity for subsequent resource allocation decisions.

[0140] Step S200: Job rescheduling evaluation: Quantitatively evaluate the resource allocation status of the current job using the adaptability function; by calculating the gain ratio of each job, the system can scientifically determine the efficiency of the current resource allocation and provide a quantitative basis for possible rescheduling operations;

[0141] Step S300: Determine whether the indicator exceeds the threshold; comprehensively consider traditional performance detection indicators (such as memory usage) and fitness function evaluation results, perform a comprehensive threshold violation judgment, and decide whether to perform rescheduling;

[0142] If the core performance indicators of the job exceed the preset threshold range, or if the adaptability function determines that not rescheduling will seriously affect the cluster performance, step S400 is executed to perform elastic adjustment through rescheduling; otherwise, the original allocation plan is maintained without rescheduling. The core performance indicators are traditional performance detection indicators, including: GPU memory occupancy, GPU computing utilization, communication delay and bandwidth bottleneck indicators;

[0143] The core performance indicators of the job exceed the preset threshold range, including:

[0144] The GPU memory usage rate reflects the model's memory usage on the current GPU device. If the GPU memory usage rate remains above 90% (a configurable threshold, set to 90% by default) for more than N evaluation cycles (e.g., 5 minutes), it indicates resource constraints, which may affect task stability or cause overflow interrupts.

[0145] Alternatively, if the GPU computing utilization rate is continuously monitored to be less than 30% (indicating that computing resources are not fully utilized) and the overall throughput of the training job on this GPU continues to decline, it is considered that there is resource waste.

[0146] Alternatively, consider communication latency and bandwidth bottleneck indicators: Use NCCL or the underlying RDMA monitoring interface to collect the average latency and bandwidth utilization of the AllReduce phase. If communication latency is more than twice the normal value or bandwidth utilization is less than 60% during training, this indicates that the communication topology may no longer be suitable for the current scale and needs to be adjusted.

[0147] During the scheduling process, a rescheduling evaluation is performed at the end of each scheduling cycle. If the calculated fitness function value of the new allocation matrix generated based on the latest resource status and job requirements in this evaluation round improves by more than 30% compared to the fitness function value of the finalized solution in the previous round—that is, the fitness function value obtained in the new evaluation round is significantly better than the fitness function value of the actual execution solution in the previous round—then the current resource allocation solution is no longer adequately supported. Failure to reschedule will lead to degraded cluster performance or significantly limited resource utilization efficiency. Therefore, this situation is judged as "failure to reschedule will seriously affect cluster performance," and the elastic scheduling process should be triggered immediately. This judgment indicates that the current allocation solution has significant room for improvement. Maintaining the original allocation will cause the overall system scheduling strategy to lag, fail to fully respond to resource changes and load variations, and thus affect cluster throughput and job completion efficiency.

[0148] Step S400: Rescheduling is performed. Based on the evaluation results of the adaptability function, a scientific determination is made as to what elastic adjustment method should be applied to the rescheduled jobs. For high-priority jobs with urgent resource requirements, a scale-out operation is implemented to meet their computing needs.

[0149] High priority refers to jobs that are explicitly marked in the scheduling policy as requiring priority for completion. Urgent resource requirements refer to jobs with significant performance constraints, as evidenced by: GPU utilization consistently exceeding 90%, a significant increase in mini-batch processing time (exceeding 30% of the average processing time in the previous evaluation cycle), and a job's fitness function gain under existing resources being less than 85% of that achieved under the cluster's current optimal resource allocation.

[0150] For jobs with a gain ratio significantly lower than the cluster average and low resource utilization, Scale In is performed to optimize resource allocation. A gain ratio significantly lower than the cluster average means that the job's gain ratio under the current resource allocation is at least 40% lower than the average for all jobs in the cluster. Low resource utilization means that the actual utilization level of the GPU resources used by the job is in a low-load state for a long time. Specific judgment criteria are as follows: the average core utilization rate of the GPU used is less than 30%, the memory utilization rate is less than 40%, and there are long idle periods or data waiting bottlenecks during the job's phased execution.

[0151] Step S500: Scale-Out Operation: During distributed deep learning training, as training tasks progress, workloads may gradually increase, necessitating dynamic node expansion to meet computing resource demands. During the scale-out process, an incremental communication topology reconstruction mechanism, based on a topology cost model and reinforcement learning algorithms, rapidly determines the optimal communication path for new nodes, enabling efficient reconstruction of the communication topology and minimizing training interruptions and delays.

[0152] Step S600: Scale-In Operation. During distributed deep learning training, as training tasks progress, the workload may gradually decrease. Dynamically scaling down nodes is necessary to optimize resource utilization. During the scale-in process, a gradient cache and asynchronous interpolation compensation mechanism are employed to save historical gradient data and dynamically compensate for incomplete gradient calculations using an interpolation algorithm. This prevents training bias and ensures model convergence and accuracy.

[0153] After completing the fitness function evaluation and rescheduling decision, the resource allocation status of the jobs currently running in the cluster is updated. The final resource adjustment results include:

[0154] (1) For some high-priority jobs with urgent resource requirements, scale-out operations were performed, and the number of GPUs used was increased to ensure the computing power required for their job execution.

[0155] (2) Some low-priority jobs with low resource utilization and poor fitness function performance were scaled in. Their allocated GPU resources were reduced, their number of parallel processes was reduced, and their computing resources were compressed to release high-quality GPU resources.

[0156] (3) The resource distribution matrix of the entire cluster is updated, and the GPU allocation list corresponding to each job is regenerated. The scheduling system will adjust the communication topology based on the new allocation results when distributing job data and control signals.

[0157] (4) Under the updated resource allocation state, the system automatically triggers the incremental communication topology reconstruction mechanism and gradient compensation mechanism to adapt to the communication structure changes and training consistency requirements brought about by resource changes, ensuring that each job can continue training seamlessly under the adjusted resource conditions.

[0158] The specific process of job rescheduling evaluation is as follows: Figure 2 As shown, job rescheduling evaluation includes:

[0159] Step S201: Generate candidate allocation schemes; regularly analyze the overall resource allocation status of the cluster, including the number of available GPUs, , number of jobs , run the search algorithm according to the predetermined scheduling cycle to generate a series of candidate allocation matrices , using a heuristic search algorithm (such as a genetic algorithm) to generate several resource allocation combinations in the feasible solution space. These candidate solutions will serve as the basis for fitness function evaluation and resource optimization in subsequent steps. As the basis for subsequent evaluation and optimization, each matrix represents a resource allocation solution, and each resource allocation solution must meet the following requirements:

[0160] (1);

[0161] in, Indicates homework In the candidate allocation matrix The number of GPUs allocated in;

[0162] Step S202: Throughput model estimation: Establish a throughput prediction model for each job to predict the given resource configuration Based on historical data and real-time operation data, supervised learning is used to capture resource configuration parameters (number of GPUs) , batch size , the number of cumulative gradient steps ) and the complex mapping relationship between the dynamic state of the cluster. Providing data support for resource allocation decisions.

[0163] Step S203: Calculate the job gain ratio; Based on the prediction results of the throughput model, calculate the gain ratio of each job under different resource configurations; For each job in the cluster, the gain ratio Defined as using a given resource allocation Throughput and fair resource allocation Throughput The ratio is used to quantitatively evaluate the efficiency of resource allocation; the gain ratio calculation formula is as follows:

[0164] (2);

[0165] Step S204: Redistribution penalty is imposed; in order to avoid the delay and performance degradation caused by frequent resource reallocation, a reallocation penalty mechanism is introduced into the scheduling strategy. Penalize jobs that need to reallocate resources. The system will only choose to reschedule if not rescheduling will cause the optimal value of the scheduling target to drop significantly. When evaluating the fitness function, impose a penalty on jobs that need to reallocate resources; set the penalty factor , based on the historical reallocation frequency of the job, the future reallocation cost is predicted using the following formula:

[0166] (3);

[0167] in, For homework Current running time; For homework The number of reallocations so far; It is the cost of redistribution, which is reflected in the delay time;

[0168] The penalty factor discounts the gain ratio of the current running configuration, namely:

[0169] (4);

[0170] For a job that frequently requests resource reallocation, the penalty factor The job's performance will gradually increase, causing its gain ratio to significantly decrease after discounting. During the fitness function evaluation process, the job's competitiveness will drop significantly, and the system will try to avoid reallocating it. This mechanism effectively prevents unnecessary frequent resource changes, ensuring stable cluster operation and continuous job execution.

[0171] Step S205: Adaptability function evaluation; for each candidate allocation matrix , through multiple iterations of calculating the fitness function for rigorous evaluation, continuously optimizing the resource allocation plan, and finally selecting the allocation matrix with the highest score to achieve the optimal configuration of cluster resources; the fitness function is defined as the generalized (power) average of each job gain ratio, and resource allocation is determined by maximizing the fitness function; the fitness function formula is for:

[0172] (5);

[0173] set up As a fairness regulator, by flexibly setting Dynamically balance the fairness and efficiency of resource allocation based on value; Indicates the total number of jobs. hour, is the arithmetic mean of all job gain ratios. At this time, resource allocation tends to prioritize jobs that can significantly improve efficiency, so that resources are tilted towards jobs that are efficiently utilized to maximize the overall performance improvement rate. hour, It tends to minimize the difference in gain ratios between tasks and emphasizes the fairness of resource allocation.

[0174] For each candidate allocation , perform resource configuration mapping; resource configuration mapping means: in each candidate allocation scheme Next, for each job Allocate the corresponding number of GPUs ;

[0175] For homework , load the optimal configuration from historical tuning records according to its task type , Indicates homework The batch size per GPU under the optimal parameter configuration (“optimal” means the highest throughput under this parameter configuration); Indicates homework The optimal number of gradient accumulation steps; represents the number of forward steps accumulated by the model before backpropagation, often used to reduce communication load or improve memory utilization.

[0176] At the same time, the performance data obtained in real time by the cluster monitoring system is used for dynamic parameter injection.

[0177] Input the tuning configuration and parameters into the throughput prediction model, perform batch inference, and predict the throughput of each job under each configuration in parallel. This includes:

[0178] The throughput prediction model is a performance regression model whose core goal is to: for each job, based on its input features (number of GPUs , batch size , the number of cumulative gradient steps ), predicting the number of samples processed per unit time (throughput) under this configuration is expressed as:

[0179] ;

[0180] Indicates that the job is configured (the number of GPUs , the batch size is , the number of gradient accumulation steps is ) throughput under This represents a throughput prediction model that uses a lightweight regression model (MLP), balancing prediction accuracy and batch inference speed.

[0181] For each job and each candidate allocation configuration , construct input features ;

[0182] Use the parallel inference framework to input the trained throughput prediction model , thus obtaining the predicted throughput of each job under each candidate configuration .Will Storage is an important basis for subsequent adaptive function evaluation; if the actual scheduling results deviate significantly from the prediction, feedback is used to update the model parameters and continuously improve the prediction accuracy.

[0183] The throughput prediction model uses an existing lightweight linear regression model and historical throughput data to provide a rough estimate, rather than a new model specifically designed for prediction. Throughput estimation is intended to provide a more accurate reference for fitness function evaluation in the early stages of cluster operation, when throughput data is scarce. If accuracy is not a priority in the early stages, it can be assumed that job throughput increases exponentially with the number of allocated GPU resources. As the cluster runs longer, sufficient throughput data (i.e., the throughput of different jobs with different numbers of GPUs) can be collected, and throughput estimation is no longer necessary.

[0184] The specific process of Scale Out operation is as follows Figure 3 Shown: Scale Out operation; including:

[0185] Step S501: elastic expansion triggering; based on the resource allocation decision, the scheduling system generates an expansion decision and triggers the node elastic expansion mechanism.

[0186] Step S502: Extended resource evaluation: After receiving the extension request, a comprehensive evaluation of the current cluster's resource status is performed. Specifically, the cluster resource evaluation dynamically obtains the GPU utilization, memory occupancy, and network bandwidth status information of the unused resources of the current job through the monitoring system, and determines resource availability based on the set thresholds (a GPU core utilization rate of less than 30% and a memory occupancy rate of less than 20% is considered "resource idle"; if the current network congestion rate exceeds the threshold (such as a link utilization rate greater than 80%), cross-node expansion will be restricted). At the same time, the GPU hardware specifications are considered, and GPUs with the same type as the currently used GPU, the shortest communication distance, and the lightest load are preferentially selected for resource expansion, including GPU utilization rate, memory occupancy, and network bandwidth utilization indicators. When selecting a new GPU, the GPU hardware specifications and the current load status are comprehensively considered. Generally, the GPU with the highest compatibility with the current job and sufficient idle resources will be given priority in the selection range to determine the new GPU.

[0187] Step S503: non-stop extended initialization; after determining the newly added GPU resources, start preparing context preparation S504 and communication topology construction S505 for the newly added GPU working nodes to perform initialization operations;

[0188] Initialization operations include:

[0189] Context preparation initialization operation: start the container that the training job depends on; load the current model structure and parameter status; load the current optimizer status (such as gradient, momentum); initialize the batch data pipeline to synchronize with the main task.

[0190] Initialization of communication topology construction: Communication status collection and pre-warming are performed. A lightweight communication performance probe is launched to pre-collect key performance parameters in the background, including inter-node bandwidth, latency, and queue depth, and initialize the communication status perception module. Communication cost modeling is prepared by loading historical communication data and a trained LSTM model to initialize the dynamic weight estimation mechanism in the communication cost function, providing a basis for subsequent optimal insertion location selection. Data pre-loading and topology pre-calculation are initiated for subsequent parallel operations to ensure that training jobs are not interrupted during expansion.

[0191] Step S504: context preparation; context preparation for the newly added GPU node includes data preparation and environment configuration;

[0192] The data preparation phase is based on the job's established data distribution strategy. This strategy refers to the data segmentation and allocation logic used by the system during the initial training task distribution phase. This strategy typically involves partitioning based on sample order, class balance, or feature space, aiming to ensure balanced training load and model generalization. Specifically, if the job uses a data parallelization strategy based on mini-batch partitioning, newly added nodes will be assigned several mini-batches of data that have not yet entered the training process and their local data loaders will be rebuilt. Unprocessed data shards will be properly distributed so that they can be accurately transferred to the newly added GPU nodes. This process involves data repartitioning and efficient transmission, using RDMA high-speed data transmission technology and bandwidth isolation to prevent data migration from interfering with training communications.

[0193] The environment configuration phase includes: deploying necessary dynamic libraries and software dependencies on the newly added GPU nodes; strictly following the job's operating environment requirements, and ensuring that the runtime environment of the newly added nodes is compatible with the global model through containerized image synchronization.

[0194] Step S505: Reconstructing the communication topology; while executing step S504, using the incremental communication topology construction mechanism to construct a new communication topology for the computing nodes of the job;

[0195] Step S506: Progressive gradient synchronization; To solve the parameter synchronization delay problem when nodes are dynamically added, when a new node synchronizes parameters, a differential transmission mechanism is used, which only transmits the historical state differences that the new node lacks. When a new GPU node synchronizes parameters, the local parameters of the new GPU node are used. and global model parameters ; Local parameters : Refers to the model parameter copy currently held on the newly added GPU node. Since the node is newly added, its parameters are usually from the default initialization value or an early snapshot, and have not been iterated and updated for the current global training rounds, so there is a significant deviation from the main training process. Global model parameters : Refers to the latest model parameter set maintained and synchronized by all existing nodes in the current cluster after several rounds of training. This parameter represents the overall learning status of the current training process and is the basis for training consistency. Generate difference parameters :

[0196] (6);

[0197] To achieve parallelization of gradient synchronization and computing tasks, a double buffering strategy is adopted. The background synchronization thread periodically calculates global parameters, and the foreground training thread continuously calculates gradients and updates local parameters. The difference pipeline mechanism is used to asynchronously push parameter differences and incrementally update to the latest parameters.

[0198] Step S507: Training job resumes. After the global communication topology switches to the new structure and the gradient synchronization path is updated, all nodes are based on the latest global parameters. Continue to perform subsequent mini-batch training to maintain computational efficiency and model consistency.

[0199] The specific process of communication topology reconstruction is as follows: Figure 4 As shown: Using the incremental communication topology construction mechanism, a new communication topology is built for the computing nodes of the job; including:

[0200] Step S5051: Resource isolation and pre-check: Isolate the selected expanded GPU, memory, and network bandwidth resources to prevent them from being preempted by other tasks; perform node health checks and remove abnormal devices.

[0201] Step S5052: topology exploration: calling the NCCL topology discovery interface to obtain hardware topology information, including the physical connection relationship of all nodes in the cluster, GPU architecture, and network interface information;

[0202] Step S5053: Communication status awareness; real-time node acquisition through distributed probes Communication performance data between nodes, including the amount of data transferred in a single synchronization (such as the size of the gradient tensor) , real-time effective bandwidth , fixed communication delay and real-time depth of the communication queue ; Provide basic data for subsequent topology reconstruction.

[0203] Step S5054: Communication cost modeling; Based on the acquired hardware topology information and communication performance data, a topology cost model (Formula (7)) is constructed; In distributed deep learning training, the communication cost between nodes has an important impact on the overall training efficiency. The present invention constructs a topology cost model that comprehensively considers the data transmission volume, inter-node bandwidth and fixed delay factors to accurately evaluate the communication cost between nodes and provide a basis for subsequent communication topology reconstruction. Through this model, the communication efficiency between nodes can be quantified, so that when adding nodes, the optimal insertion position can be selected to avoid the decline in training efficiency due to communication bottlenecks. Define nodes Communication cost It is a multidimensional function consisting of data transmission time, fixed delay and instantaneous congestion. The formula is as follows:

[0204] (7);

[0205] in, is a dynamic weight coefficient obtained through historical communication data training and dynamically adjusted based on the LSTM prediction model to meet the constraints and ; The design logic is as follows:

[0206] (8);

[0207] Specifically, by minimizing the communication cost function The actual end-to-end delay collected in real time by distributed probes The mean square error of ;

[0208] Step S5055: Optimal insertion decision; using the optimal insertion algorithm driven by reinforcement learning, with the current communication topology and node information as the state, the location selection of the new node as the action, and the reduction of the communication cost as the reward function, the insertion point with the least global impact is selected from the candidate positions, that is, the lowest total cost increment; for the existing nodes in the topology , quantified cost increment Defined as inserting a new node The difference between the new link cost after disconnection and the communication cost before the original link is disconnected is calculated as follows:

[0209] (9);

[0210] This strategy can automatically select the optimal insertion location based on the current communication topology and node status, thereby quickly and efficiently reconstructing the communication topology when adding nodes, reducing training interruptions and delays, and improving training efficiency.

[0211] Step S5056: Connectivity check: After the new communication topology is established, a simulated transmission test is performed to ensure that there are no loops or isolated nodes in the topology;

[0212] If there is no connectivity problem, the communication topology is switched directly; otherwise, step S1055 is executed again to re-build the optimal communication topology.

[0213] For existence The topology of the nodes, the new node The optimal insertion position Determined by the following formula:

[0214] (10).

[0215] This strategy automatically selects the best insertion location based on the current communication topology and node status, thereby quickly and efficiently reconstructing the communication topology when adding nodes.

[0216] The specific process of Scale In operation is as follows Figure 5 As shown: Adopting gradient caching and asynchronous interpolation compensation mechanism, by saving historical gradient data and using interpolation algorithm to dynamically compensate for unfinished gradient calculations; including:

[0217] Step S601: elastic scaling-in triggering; based on the resource allocation decision, the scheduler generates a scaling-in decision, triggering the elastic scaling-in operation of the node.

[0218] Step S602: Screening of exiting nodes; after receiving the reduction request, a comprehensive assessment of the resource status of the current cluster is conducted; including: real-time collection of the following performance indicators of each GPU node and statistics of its status: GPU utilization, GPU memory occupancy, and network bandwidth utilization. Sort and screen the most suitable nodes for exit based on the following two key parameters: low computing power utilization and low data migration cost; including GPU utilization, memory occupancy, and network bandwidth utilization indicators; when selecting exiting nodes, comprehensive consideration is given to node performance, load balancing, and data migration cost factors. Normally, nodes with low computing power utilization and low data migration cost are preferred as exiting nodes; low computing power utilization means: the average GPU utilization rate during the statistical period is lower than the set threshold; it is defined as: GPU utilization rate is less than 30% and memory occupancy rate is less than 40%; low data migration cost means: the communication cost and waiting time required to migrate unfinished data batches or model copies on the node to other nodes are short; specifically, it means: the data migration volume is less than 500MB and the network transmission delay is less than 10ms, or the node only participates in the training of low-priority, pausable tasks;

[0219] For nodes that are forced to exit due to hardware failure or other reasons, they are regarded as a special elastic reduction case, and the abnormal nodes are regarded as exit nodes.

[0220] Step S603: consistency check: after determining the exit node, check whether the training job is at a gradient synchronization safety point, and verify the consistency of the calculation progress of the node to be removed with the global model version; including:

[0221] The global model refers to the unified set of model parameters that are collaboratively updated by all active nodes in the current training task through gradient synchronization. Verify that the local model parameters of the node to be removed are synchronized with the global model. This ensures that the models on all participating compute nodes are consistent, thus determining whether gradient compensation is necessary.

[0222] Method for verifying the consistency of the calculation progress: Each training node maintains the version number of its local model parameters. By pulling the synchronization version number and comparing it, it can be determined whether it is in the synchronization "safe point". For example: suppose the version number of the local model parameters of the training node is , the global model also maintains a synchronized version number .like , indicating that the node has completed the most recent global synchronization and the status is consistent; if , indicating that the node has not completed the most recent synchronization and there is a parameter lag that needs to be compensated.

[0223] If yes, then execute step S604 and the node exits directly; otherwise, execute step S605 and calculate the compensation gradient of the exiting node;

[0224] Step S604: The node exits directly. If the training job is at a gradient synchronization safety point, there is no need to consider the gradient loss problem. The unprocessed data is backed up and the exit node can be directly removed.

[0225] Step S605: Compensate gradient calculation. When the exiting node is in the process of calculating the gradient of the current mini-batch, the gradient interpolation compensation mechanism is activated to avoid the parameter update deviation problem caused by directly discarding the calculation results. The historical cached gradient of the exiting node is integrated with the real-time gradient of the cluster to generate a compensation value to replace the unfinished gradient calculation.

[0226] Step S606: Asynchronous aggregation; after the compensation gradient is generated, the compensation gradient is quickly distributed and synchronized with the parameter update through a non-blocking communication mechanism; the training thread can continue to execute without waiting for the resources of the exit node to be released, greatly reducing the interruption time.

[0227] Step S607: Training job resumes. After completing the above steps, the remaining nodes continue the training task, improving resource utilization. The resources released by the exiting nodes are reallocated and optimized to improve overall performance.

[0228] The specific process of compensation gradient calculation is as follows Figure 6 As shown, the compensation gradient calculation includes:

[0229] In order to cope with the problems caused by dynamic changes in nodes during distributed deep learning training, this paper designs a gradient cache pool mechanism. Each computing node maintains a gradient cache queue with a fixed capacity of n, using an LRU (least recently used) cache strategy to save the gradient tensors of the latest n mini-batches. , ensuring that the cache always holds the latest and most likely to be called gradient data.

[0230] In this way, when a node exits, the gradient data in the cache can be used in combination with the current calculation progress for compensation, avoiding the inconsistent model parameter updates caused by directly discarding the calculation results of the exiting node, thereby ensuring the convergence and accuracy of the model. The core mechanism of the gradient cache pool is:

[0231] Cache update: After completing the calculation of a mini-batch, the gradient Add to the tail of the queue. If the queue is full, remove the oldest gradient at the head.

[0232] Data integrity: Cached gradients are accompanied by timestamps and computational stage metadata (such as forward propagation completion, number of backpropagation layers), as well as batch identifiers to ensure metadata association.

[0233] Step S6051: Gradient cache acquisition; establish a gradient cache queue for each computing node, always save the latest and most likely to be called gradient data; obtain the latest cached gradient in the gradient cache pool of the exit node ;

[0234] Step S6052: real-time gradient aggregation; The current mini-batch gradient of each node Collect and calculate the gradient mean ;

[0235] Step S6053: Dynamic weight correction; to achieve interpolation balance, introduce As a weight to balance historical gradients and real-time gradients; specifically, by setting the original progress value of the exit node at the current mini-batch Perform gradient similarity correction and adjust weight distribution:

[0236] (11);

[0237] in, , is the gradient difference vector; correction term The consistency between the historical gradient of the exiting node and the real-time gradient of the remaining nodes is quantified: the closer the correction term is to 1, the more similar the direction and magnitude of the two are, and the historical data can be trusted; otherwise, there is a significant deviation, and the real-time gradient is more dependent. Represents the L2 norm of the vector, which measures the magnitude of the gradient change;

[0238] Step S6054: Compensation gradient fusion; when performing the dynamic node removal operation, the exiting node may be in the process of gradient calculation of the current mini-batch. In order to avoid the parameter update deviation problem caused by directly discarding the calculation results, the present invention proposes a gradient interpolation compensation method. By fusing the historical cache gradient of the exiting node with the real-time gradient of the cluster, a compensation value is generated to replace the unfinished gradient calculation. By weightedly fusing the historical gradient information of the exiting node with the current gradient information of the remaining nodes, a compensation gradient is generated; the model parameter update is smoother and more stable, thereby ensuring the convergence and accuracy of the model. In addition, gradient interpolation compensation can also effectively deal with the gradient loss problem when the node is removed, thereby improving the robustness and reliability of the system. The compensation gradient is calculated by the following formula :

[0239] (12).

[0240] The specific process of asynchronous aggregation is as follows Figure 7 As shown in the figure, asynchronous aggregation; after the compensation gradient is generated, the compensation gradient is quickly distributed and synchronized with the parameter update through a non-blocking communication mechanism, so that the training thread can continue to execute without waiting for the resources of the exit node to be released, greatly reducing the interruption time. It includes:

[0241] Step S6061: Parallelization is triggered; execution permissions are allocated, and an independent thread pool is created to ensure parallel execution of gradient distribution and training threads;

[0242] Step S6062: Topology reconstruction; constructing a minimum spanning tree topology based on the remaining nodes, optimizing the gradient synchronization path; using incremental link switching, updating only the communication subgraph affected by the exiting node;

[0243] Step S6063: Compensate for gradient broadcast; use The interface will compensate for gradient sharding broadcasts; set up redundant transmission channels, with primary and backup links transmitting in parallel; and give priority to low-latency paths.

[0244] Step S6064: Data migration and backup: At the same time, the exiting node begins data backup and resource release. The exiting node organizes and backs up the unfinished data and migrates it to other appropriate nodes or storage devices according to the data management policy of the job.

[0245] Step S6065: Resource decoupling and release; at the same time point as the broadcast trigger, a resource release instruction is sent to the exiting node that has completed data backup, gradually releasing its GPU memory occupancy and network connection. The release process is completely decoupled from the training calculations of the remaining nodes.

[0246] Step S6066: Gradient aggregation. After receiving the compensation gradient, each remaining node immediately joins the gradient aggregation queue. The node aggregates the gradient calculated by itself and the received compensation gradient to form complete gradient information.

[0247] Step S6067: Parameter update and verification; based on the aggregated gradient, each node updates the global model parameters; after the parameter update is completed, a posteriori verification is performed to compare the updated parameter hash values ​​of all nodes to ensure global consistency.

[0248] Through the above asynchronous mechanism, the synchronous waiting delay is eliminated, so that the training task can maintain a fast interrupt response in the event of node removal, and maintain the continuity and stability of the parameter update pipeline.

[0249] In a typical deep learning training task, a ResNet-50 model was trained on NVIDIA Tesla T4 GPUs using distributed data. The job used the CIFAR-10 dataset (50,000 training samples, 32×32 color images) and the SGD optimizer, with a fixed batch size of 64 and 50 training epochs. Dynamic resource adjustments were simulated during model training, with nodes randomly joining (scale-out) or leaving (scale-in). Data was collected through multiple repeated experiments to ensure reliable results. The experimental group settings were as follows:

[0250] (1) Ablation group: Remove the elastic resource adjustment module, cancel the gradient cache and interpolation compensation, and adopt the traditional global checkpoint rollback strategy - when a node changes, pause training and roll back to the most recently saved global checkpoint, and resume training after rebuilding the communication topology.

[0251] (2) Control group: The complete elastic resource adjustment module is enabled, including two core mechanisms:

[0252] a) Incremental communication topology reconstruction: Real-time perception of node changes, dynamic update of multi-GPU communication groups, and avoidance of global restarts;

[0253] b) Gradient caching and interpolation compensation mechanism: Cache unsynchronized gradient data before node exit, perform linear interpolation based on global parameter gradients, and compensate for gradient offset.

[0254] The experiment used the standard deviation of the training loss as the primary evaluation metric, reflecting the fluctuation of the model's loss during each training round. A lower standard deviation indicates a more stable training process, indicating that resource changes will minimize disruptions to the continuity of model training.

[0255] Result analysis: The experimental results are as follows: Figure 8As shown, the training loss standard deviation curve clearly demonstrates the significant difference in training stability between the two groups. Overall, the control group's training error curve exhibits smaller fluctuations and a more stable downward trend. Even during dynamic resource changes, the control group maintains low training perturbations, demonstrating the effectiveness of the gradient caching and interpolation compensation mechanisms in mitigating training shocks caused by resource adjustments.

[0256] Data trends show that in the initial stage, when no nodes change, the loss standard deviation curves of the two groups almost overlap, with consistent fluctuation trends, indicating that the two scheduling methods have similar training stability under static resource conditions. However, as node scaling events are triggered, differences between the two groups quickly become apparent. This is because traditional checkpoint rollback strategies inevitably introduce training interruptions and gradient inconsistencies due to parameter reloading and communication group reconstruction. The elastic resource adjustment module reduces data loss through gradient caching and maintains parameter continuity through interpolation compensation, effectively reducing the interference of dynamic resource adjustments on the training process.

[0257] Example 3

[0258] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the elastic resource adjustment method in distributed data parallel training described in Example 1 or 2.

[0259] Example 4

[0260] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the elastic resource adjustment method in distributed data parallel training described in Example 1 or 2 are implemented.

[0261] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for adjusting elastic resources in distributed data parallel training, characterized in that: include: Step 1: Use the performance monitoring module to monitor job performance in real time and collect key performance indicators including computing load, communication bandwidth, and GPU utilization; Step 2: Based on the preset fitness function evaluation and the reallocation penalty mechanism, a comprehensive assessment is conducted on the current cluster resource status. If the fitness function determines that not rescheduling will seriously affect cluster performance, the rescheduling process is quickly initiated to begin resource elasticity adjustments. Step 3: Based on continuous monitoring and evaluation of the adaptability function, determine which jobs require which elasticity operations. Simultaneously, based on the elasticity adjustment target, use an intelligent node selection algorithm to select the most suitable nodes from the available node pool. Step 4: Execute the elastic scheduling strategy and perform elastic resource adjustments. Dynamic resource adjustment and optimal configuration are achieved through an incremental communication topology reconstruction mechanism and a gradient caching and interpolation compensation mechanism for different elastic operations for different jobs. Utilize the incremental communication topology construction mechanism to build a new communication topology for the job's compute nodes. This includes: Step S5051: Resource isolation and pre-check: Isolate the selected expanded GPU, memory, and network bandwidth resources, and perform node health checks to eliminate abnormal devices. Step S5052: topology exploration: calling the NCCL topology discovery interface to obtain hardware topology information, including the physical connection relationship of all nodes in the cluster, GPU architecture, and network interface information; Step S5053: Communication status perception; collect communication performance data between nodes (i, j) in real time through distributed probes, including the data transmission volume D of a single synchronization, the real-time effective bandwidth B i，j , fixed communication delay L i，j and communication queue real-time depth Queue i，j ; Step S5054: Communication cost modeling: Based on the acquired hardware topology information and communication performance data, a topology cost model is constructed to define the communication cost C(i, j) between nodes (i, j) as a multidimensional function consisting of a data transmission time term, a fixed delay term, and an instantaneous congestion term. The formula is as follows: Among them, α, β, and γ are dynamic weight coefficients, satisfying the constraints α+β+γ=1 and α, β, γ≥0.

1. The design logic is as follows: Specifically, by minimizing the communication cost function C(i, j) and the actual end-to-end delay collected by the distributed probes in real time The mean square error of α, β, and γ are obtained; Step S5055: Optimal insertion decision; using the optimal insertion algorithm driven by reinforcement learning, with the current communication topology and node information as the state, the location selection of the new node as the action, and the reduction in communication cost as the reward function, the insertion point with the least global impact is selected from the candidate locations, that is, the lowest total cost increment; for an existing node k in the topology, the quantified cost increment ΔC(k) is defined as the difference between the newly added link cost after inserting the new node new and the communication cost before the original link was disconnected, as follows: ΔC(k)=C(k,new)+C(new,k+1)-C(k,k+1) (9); Step S5056: Connectivity check: After the new communication topology is established, a simulated transmission test is performed to ensure that there are no loops or isolated nodes in the topology; If there is no connectivity problem, the communication topology is switched directly; otherwise, step S1055 is executed again to re-build the optimal communication topology; Adopting gradient caching and asynchronous interpolation compensation mechanism, it saves historical gradient data and uses interpolation algorithm to dynamically compensate for incomplete gradient calculations; including: Step S601: elastic shrinkage triggering; Step S602: Exit node screening; after receiving the reduction request, a comprehensive assessment of the current cluster's resource status is conducted, including indicators such as GPU utilization, video memory occupancy, and network bandwidth utilization; when selecting an exit node, select one with low computing power utilization and low data migration cost as the exit node; low computing power utilization means that the average GPU utilization rate during the statistical period is lower than the set threshold; it is defined as: GPU utilization is less than 30% and video memory occupancy is less than 40%; low data migration cost means that the communication cost and waiting time required to migrate unfinished data batches or model copies on a node to other nodes are short; specifically, it means that the data migration volume is less than 500MB and the network transmission delay is less than 10ms, or the node only participates in the training of low-priority, pausable tasks; Step S603: consistency check: after determining the exit node, check whether the training job is at a gradient synchronization safety point, and verify the consistency of the calculation progress of the node to be removed with the global model version; If yes, then execute step S604 and the node exits directly; otherwise, execute step S605 and calculate the compensation gradient of the exiting node; Step S604: the node exits directly; Step S605: Compensate gradient calculation: When the exiting node is in the process of calculating the gradient of the current mini-batch, the gradient interpolation compensation mechanism is activated. By fusing the historical cached gradient of the exiting node with the real-time gradient of the cluster, a compensation value is generated to replace the unfinished gradient calculation. Step S606: asynchronous aggregation; after the compensation gradient is generated, the compensation gradient is quickly distributed and synchronized with the parameter update through a non-blocking communication mechanism; Step S607: Resuming the training job; Compensated gradient calculation; including: Step S6051: Gradient cache acquisition; establish a gradient cache queue for each computing node, always save the latest and most likely to be called gradient data; obtain the latest cached gradient g in the gradient cache pool of the exit node cached ; Step S6052: real-time gradient aggregation; the current mini-batch gradients {g1, g2, ..., g n } to collect and calculate the gradient mean Step S6053: Dynamic weight correction; introduce p as the weight to balance the historical gradient and the real-time gradient; specifically, by setting the original progress value p of the exit node in the current mini-batch progres Perform gradient similarity correction and adjust weight distribution: Where Δg = g cached -g avg , is the gradient difference vector; correction term Quantify the consistency between the historical gradient of the exiting node and the real-time gradient of the remaining nodes: ||.|| represents the L2 norm of the vector, which measures the magnitude of the gradient change; Step S6054: Compensation gradient fusion; generate compensation gradient by weighted fusion of historical gradient information of exit node and current gradient information of remaining nodes; calculate compensation gradient g by the following formula comp : g comp =p×g cached +(1-p)×g avg (12)。 2. The elastic resource adjustment method in distributed data parallel training according to claim 1, characterized in that: The elastic resource adjustment method in distributed data parallel training includes: Step S100: The scheduling period arrives; according to the preset scheduling period, the resource scheduling evaluation process is periodically triggered; Step S200: Job rescheduling evaluation: Use the adaptability function to quantitatively evaluate the resource allocation status of the current job; by calculating the gain ratio of each job, determine the efficiency of the current resource allocation and provide a quantitative basis for possible rescheduling operations; Step S300: Determine whether the indicator exceeds the threshold; comprehensively consider traditional performance detection indicators and fitness function evaluation results, conduct a comprehensive threshold breakthrough judgment, and decide whether to perform rescheduling; If the core performance indicators of the job exceed the preset threshold range, or if the adaptability function determines that not rescheduling will seriously affect the cluster performance, step S400 is executed to perform elastic adjustment through rescheduling; otherwise, the original allocation plan is maintained without rescheduling. The core performance indicators are traditional performance detection indicators, including: GPU memory occupancy, GPU computing utilization, communication delay and bandwidth bottleneck indicators; The core performance indicators of the job exceed the preset threshold range, including: The GPU memory usage remains above 90% for more than N evaluation cycles; Alternatively, the GPU computing utilization rate is continuously monitored to be less than 30%, and the overall throughput of the training job on this GPU continues to decline; Alternatively, the communication delay is more than twice the normal value, or the bandwidth utilization is less than 60%; Step S400: Rescheduling is performed. Based on the evaluation result of the adaptability function, it is determined what elastic adjustment method should be applied to the rescheduled jobs. For jobs with high priority and urgent resource requirements, a scale-out operation will be implemented. High priority refers to jobs explicitly marked in the scheduling policy as requiring priority. Urgent resource requirements refer to jobs with significant performance constraints, as evidenced by: GPU utilization consistently exceeding 90%, a significant increase in mini-batch processing time, and a job's fitness function gain under existing resources being less than 85% of what would be achieved under the cluster's current optimal resource allocation. For jobs whose gain ratio is significantly lower than the cluster average and whose resource utilization is low, the Scale In operation is performed. A gain ratio significantly lower than the cluster average means that the job's gain ratio under the current resource allocation is at least 40% lower than the average for all jobs in the cluster. A low resource utilization means that the actual utilization level of the GPU resources used by the job is in a low load state for a long time. Step S500: Scale Out operation: Using the incremental communication topology reconstruction mechanism, based on the topology cost model and reinforcement learning algorithm, the optimal communication path of the new node is determined to achieve efficient reconstruction of the communication topology; Step S600: Scale In operation; using a gradient cache and asynchronous interpolation compensation mechanism, by saving historical gradient data and using an interpolation algorithm to dynamically compensate for incomplete gradient calculations.

3. The elastic resource adjustment method in distributed data parallel training according to claim 2, characterized in that: Job rescheduling assessment; including: Step S201: Generate candidate allocation schemes; regularly analyze the overall resource allocation status of the cluster, including the number of available GPUs N according to the cluster status. GPU , the number of jobs J, runs the search algorithm according to the predetermined scheduling cycle, and generates a series of candidate allocation matrices {A (1) , A (2) ,…,A (K) }, each matrix represents a resource allocation scheme, and each resource allocation scheme must satisfy: in, Represents job i in the candidate allocation matrix A (k) The number of GPUs allocated in; Step S202: Throughput model estimation: establish a throughput prediction model for each job to predict the job throughput under a given resource configuration A; based on historical data and real-time operation data, use supervised learning to capture the complex mapping relationship between resource configuration parameters and cluster dynamic state; Step S203: Calculate the job gain ratio; Based on the prediction results of the throughput model, calculate the gain ratio of each job under different resource configurations; For each job in the cluster, the gain ratio Gain i (A i ) is defined as using the given resource allocation A i Throughput i (A i ) and using fair resource allocation A average Throughput i (A average ) is used to quantitatively evaluate the efficiency of resource allocation; the gain ratio calculation formula is as follows: Step S204: reallocation penalty is imposed; when evaluating the fitness function, a penalty is imposed on jobs that need to reallocate resources; a penalty factor r is set i , based on the historical reallocation frequency of the job, the future reallocation cost is predicted using the following formula: Among them, T i is the current running time of job i; N i is the number of reallocations of job i so far; S is the cost of reallocation, which is reflected in the delay time; The penalty factor discounts the gain ratio of the current running configuration, namely: Gain i (A i )’=Gain i (A i )×r i (4); Step S205: Adaptability function evaluation; for each candidate allocation matrix A (K) , through multiple iterations of calculating the fitness function for evaluation, continuously optimizing the resource allocation scheme, and finally selecting the allocation matrix with the highest score to achieve the optimal configuration of cluster resources; the fitness function is defined as the generalized average of the gain ratio of each job, and resource allocation is determined by maximizing the fitness function; the fitness function formula Adaption(A) is: Set ρ∈(-∞,1] as the fairness regulator to dynamically balance the fairness and efficiency of resource allocation by setting the ρ value; J represents the total number of jobs.

4. The elastic resource adjustment method in distributed data parallel training according to claim 3 is characterized in that: For each candidate allocation plan A (k) , perform resource configuration mapping; resource configuration mapping means: in each candidate allocation scheme A (k) Under this condition, the corresponding number of GPUs is allocated to each job i For job i, load the optimal configuration from the historical tuning records based on its task type represents the batch size per GPU under the optimal parameter configuration for job i; represents the cumulative number of optimal gradient steps for job i; At the same time, the performance data obtained in real time by the cluster monitoring system is used for dynamic parameter injection.

5. The elastic resource adjustment method in distributed data parallel training according to claim 3, characterized in that: Input the tuning configuration and parameters into the throughput prediction model, perform batch inference, and predict the throughput of each job under each configuration in parallel. This includes: The throughput prediction model is expressed as: Throughput(n,m,s)=f θ (n,m,s); Throughput (n, m, s) represents the throughput of the job under configuration; f θ (n,m,s) represents the throughput prediction model, which uses a lightweight regression model; For each job i and each candidate allocation configuration Construct input features (n, m, s); Use the parallel inference framework to input the trained throughput prediction model f θ (·), thus obtaining the predicted throughput (n, m, s) of each job under each candidate configuration.

6. The elastic resource adjustment method in distributed data parallel training according to claim 2, characterized in that: Scale Out operations; include: Step S501: elastic expansion is triggered; Step S502: Extended resource evaluation: After receiving the extension request, a comprehensive evaluation is performed on the current cluster's resource status, including GPU usage, video memory usage, and network bandwidth utilization. The GPU with the highest compatibility with the current job and sufficient idle resources is determined as the new GPU. Step S503: non-stop expansion initialization: after determining the newly added GPU resources, perform initialization operations; Step S504: context preparation; context preparation for the newly added GPU node includes data preparation and environment configuration; In the data preparation phase, based on the job's established data distribution strategy, some unprocessed data shards are rationally distributed so that they can be accurately transmitted to the newly added GPU nodes. Environment configuration, including: deploying dynamic libraries and software dependencies on newly added GPU nodes; Step S505: Reconstruct the communication topology; use the incremental communication topology construction mechanism to build a new communication topology for the computing nodes of the job; Step S506: Progressive gradient synchronization: When a new GPU node is added to synchronize parameters, the local parameters θ of the new GPU node are synchronized. local and the global model parameter θ glo{al ; Generate the difference parameter Δθ: Δθ=θ glo{al -θ local (6); Step S507: The training job is resumed.

7. The elastic resource adjustment method in distributed data parallel training according to claim 1, characterized in that: For a topology with n nodes, the optimal insertion position k for the new node new is * Determined by the following formula: k * =argmin k∈[1,6] [ΔC(k)] (10)。 8. The elastic resource adjustment method in distributed data parallel training according to claim 1, characterized in that: Asynchronous aggregation; including: Step S6061: Parallelization is triggered; execution permissions are allocated, and an independent thread pool is created to ensure parallel execution of gradient distribution and training threads; Step S6062: Topology reconstruction; constructing a minimum spanning tree topology based on the remaining nodes, optimizing the gradient synchronization path; using incremental link switching, updating only the communication subgraph affected by the exiting node; Step S6063: broadcast the compensation gradient; broadcast the compensation gradient slices using the NCCL ibcast interface; set up redundant transmission channels, and transmit in parallel on the primary and backup links; Step S6064: data migration and backup; Step S6065: resource decoupling and release; Step S6066: Gradient aggregation. After receiving the compensation gradient, each remaining node immediately joins the gradient aggregation queue. The node aggregates the gradient calculated by itself and the received compensation gradient to form complete gradient information. Step S6067: Parameter update and verification: Based on the aggregated gradient, each node updates the global model parameters.

Images