Large language model distributed training method and system based on dynamic resource scheduling

Abstract

The present invention provides a large language model distributed training method and system based on dynamic resource scheduling. By collecting resource status data of each computing node at periodic time intervals, the training task of the large language model is divided into multiple subtasks in the current training batch. Based on the resource status data of each computing node and the task description data of each subtask, a reinforcement learning strategy is used to allocate each subtask to the optimal computing node in an optimal proportion. In addition, a gradient compression algorithm is used to compress the gradient data generated on the computing node, and the compression rate of the gradient data is dynamically adjusted in combination with the current network bandwidth utilization of the computing node. The parameter server then performs weighted fusion on the compressed gradient data of different computing nodes based on the parameter synchronization interval, updates the global model parameters based on the fusion result, and broadcasts the global model parameters to each computing node, which can significantly improve the training efficiency of the large language model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and distributed computing technology, and in particular to a large language model distributed training method and system based on dynamic resource scheduling. Background Art

[0002] With the rapid growth in the number of parameters in large language models, traditional distributed training methods have gradually exposed numerous problems and technical bottlenecks in practical applications. Currently, large-scale pre-trained models often contain tens of billions or even trillions of parameters, placing extremely high demands on the underlying computing, storage, and communication systems. To support such massive computational tasks, distributed training has become an inevitable choice, with the widespread adoption of various parallel strategies such as data parallelism, model parallelism, and tensor parallelism. However, in actual training, the design and implementation of traditional distributed training architectures are unable to meet the training requirements of extremely large models.

[0003] First, in heterogeneous computing environments, uneven resource utilization is particularly prominent. Distributed training is typically deployed on clusters composed of compute nodes of varying types and specifications (such as GPUs, TPUs, and high- and low-spec CPU servers). This often leads to some nodes being overloaded while others are idle. This results in training speed being limited by the slowest nodes, significantly reducing overall efficiency.

[0004] Secondly, as model size continues to expand, the communication cost of parameter synchronization increases exponentially. In mainstream training frameworks, multiple computing nodes must frequently perform gradient aggregation and parameter synchronization operations. Especially when using full synchronization mechanisms, large amounts of data are frequently transmitted across the network, resulting in significant communication bottlenecks and severely restricting training performance.

[0005] Furthermore, current distributed training systems generally lack fault tolerance. During long training sessions, node failures, GPU downtime, or network fluctuations are common. However, most training frameworks lack automatic detection and fault isolation mechanisms. Once an anomaly occurs, it can lead to training interruptions or even complete mission failure. Summary of the Invention

[0006] The present invention provides a large language model distributed training method and system based on dynamic resource scheduling, which is used to solve the defect that the existing technology is difficult to support efficient training of ultra-large-scale models.

[0007] The present invention provides a large language model distributed training method based on dynamic resource scheduling, comprising:

[0008] For a distributed training cluster containing heterogeneous computing nodes, resource status data of each computing node is collected at periodic time intervals; the resource status data includes GPU computing power utilization, video memory occupancy, network bandwidth remaining, and gradient data distribution characteristics;

[0009] In the current training batch, the large language model training task is divided into multiple subtasks. Based on the resource status data of each computing node and the task description data of each subtask, a reinforcement learning strategy is used to allocate each subtask to the optimal computing node in the optimal ratio. The task description data includes computing intensity, video memory requirements, communication dependencies, execution latency, and task type labels.

[0010] Compressing the gradient data generated on the computing node using a gradient compression algorithm, and dynamically adjusting the compression rate of the gradient data in combination with the current network bandwidth utilization of the computing node;

[0011] The parameter server performs weighted fusion on the compressed gradient data of different computing nodes based on the parameter synchronization interval, updates the global model parameters based on the fusion results, and broadcasts the global model parameters to each computing node.

[0012] According to the present invention, a large language model distributed training method based on dynamic resource scheduling is provided. Based on the resource status data of each computing node and the task description data of each subtask, a reinforcement learning strategy is used to allocate each subtask to the optimal computing node in an optimal ratio, including:

[0013] Construct a state space consisting of each computing node's GPU utilization, video memory occupancy, remaining network bandwidth, the amount of gradient data in the current batch, and the task description vector output by the graph neural network based on the task description data of various subtasks;

[0014] Constructing an action space including a task allocation weight matrix; wherein the task allocation weight matrix defines the task proportion of each subtask undertaken by each computing node in the corresponding training batch;

[0015] Based on a preset reward function, the state space, and the action space, a pre-trained deep Q network is used to determine the subtask type undertaken by each computing node and the task ratio of the corresponding subtask type.

[0016] According to a large language model distributed training method based on dynamic resource scheduling provided by the present invention, the preset reward function is specifically:

[0017] R = α × resource balancing coefficient + β × single iteration time reduction rate - γ × node overload penalty

[0018] Among them, α, β and γ are preset weight coefficients respectively, and the resource balancing coefficient is determined based on the inverse of the variance of the resource utilization of each computing node.

[0019] According to a large language model distributed training method based on dynamic resource scheduling provided by the present invention, the gradient data generated on the computing node is compressed using a gradient compression algorithm, and the compression rate of the gradient data is dynamically adjusted based on the current network bandwidth utilization of the computing node, including:

[0020] Based on the current compression rate, the gradient data generated on the computing node is sparsely processed and a non-zero gradient index table is generated;

[0021] Dynamically quantize and encode the sparse gradient data, mapping the 32-bit floating-point value to the 8-bit integer space according to the gradient distribution range;

[0022] The network bandwidth utilization of the computing node is monitored in real time. If the current bandwidth utilization exceeds a preset threshold, the compression rate is reduced, and quadratic Huffman coding is enabled to compress the non-zero gradient index table.

[0023] According to a large language model distributed training method based on dynamic resource scheduling provided by the present invention, the quantization range of the dynamic quantization coding is determined by the following formula:

[0024] scale_factor=127max(∣gradmin∣,∣gradmax∣)

[0025] Among them, gradmin and gradmax are the minimum and maximum values ​​of the sparse gradient of the current batch.

[0026] According to a large language model distributed training method based on dynamic resource scheduling provided by the present invention, the method further includes:

[0027] Compute node failure detection is performed based on an elastic fault-tolerance mechanism, and training tasks for the failed node are restored based on incremental checkpoint snapshots and task migration priority queues.

[0028] According to the present invention, a large language model distributed training method based on dynamic resource scheduling is provided. The method detects faults on computing nodes based on an elastic fault-tolerant mechanism and recovers training tasks for the faulty nodes based on incremental checkpoint snapshots and a task migration priority queue, including:

[0029] Save snapshots of full checkpoints and incremental checkpoints at preset time intervals. The incremental checkpoint only stores gradient difference data from the previous snapshot; the gradient difference data is compressed using a differential encoding algorithm.

[0030] Monitor the survival status of the computing node through a heartbeat detection mechanism, and determine whether the computing node is a faulty node based on the number of consecutive non-response times;

[0031] For a failed node, the model parameters are restored based on the snapshot of the latest full checkpoint and the snapshot of the latest incremental checkpoint corresponding to the failed node, and the unfinished tasks of the failed node are added to the task migration priority queue, so that the unfinished tasks are migrated to the backup node based on the priority of the unfinished tasks, and the integrity of the model parameters is verified through the consistency protocol.

[0032] According to a large language model distributed training method based on dynamic resource scheduling provided by the present invention, the method further includes:

[0033] Dynamically adjusting the parameter synchronization interval based on the network bandwidth of each computing node;

[0034] When the remaining network bandwidth of each computing node is higher than a preset threshold, the parameter synchronization interval is shortened to the minimum allowed value;

[0035] When the remaining network bandwidth of any computing node is lower than a preset threshold, the parameter synchronization interval is extended to a maximum allowable value.

[0036] According to a distributed training method for a large language model based on dynamic resource scheduling provided by the present invention, the periodic time interval is 1 second, and the resource status data is collected through a lightweight agent program, which is deployed on each computing node and stores the historical resource status data sequence through a time series database.

[0037] The present invention also provides a large language model distributed training system based on dynamic resource scheduling, comprising:

[0038] A resource monitoring module is used to collect resource status data of each computing node at periodic time intervals for a distributed training cluster containing heterogeneous computing nodes; the resource status data includes GPU computing power utilization, video memory occupancy, network bandwidth remaining, and gradient data distribution characteristics;

[0039] A dynamic scheduling decision module is used to divide the training tasks of the large language model into multiple subtasks in the current training batch. Based on the resource status data of each computing node and the task description data of each subtask, it uses a reinforcement learning strategy to allocate the subtasks to the optimal computing nodes in an optimal ratio. The task description data includes computing intensity, video memory requirements, communication dependencies, execution latency, and task type labels.

[0040] A gradient optimization communication module is used to compress the gradient data generated on the computing node using a gradient compression algorithm, and dynamically adjust the compression rate of the gradient data in combination with the current network bandwidth utilization of the computing node;

[0041] The parameter update module is used to perform weighted fusion of the compressed gradient data of different computing nodes based on the parameter synchronization interval on the parameter server side, update the global model parameters based on the fusion results, and broadcast the global model parameters to each computing node.

[0042] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements any of the above-described distributed training methods for large language models based on dynamic resource scheduling.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the distributed training method for a large language model based on dynamic resource scheduling as described above is implemented.

[0044] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned distributed training methods for large language models based on dynamic resource scheduling.

[0045] The distributed training method and system for large language models based on dynamic resource scheduling provided by the present invention collect resource status data of each computing node at periodic time intervals, thereby dividing the training tasks of the large language model into multiple subtasks in the current training batch, and using a reinforcement learning strategy to distribute the various subtasks to the optimal computing nodes in an optimal proportion based on the resource status data of each computing node and the task description data of each subtask; in addition, a gradient compression algorithm is used to compress the gradient data generated on the computing nodes, and the compression rate of the gradient data is dynamically adjusted in combination with the current network bandwidth utilization of the computing nodes; the parameter server then performs weighted fusion on the compressed gradient data of different computing nodes based on the parameter synchronization interval, updates the global model parameters based on the fusion results, and broadcasts the global model parameters to each computing node, which can significantly improve the training efficiency of the large language model and achieve a significant improvement in resource utilization under the same hardware conditions to support the stable training of models with hundreds of billions of parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 This is a flow chart of a large language model distributed training method based on dynamic resource scheduling provided by the present invention;

[0048] Figure 2 It is a flowchart of the task scheduling method provided by the present invention;

[0049] Figure 3 1 is a flow chart of the gradient compression method provided by the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of the large language model distributed training system based on dynamic resource scheduling provided by the present invention;

[0051] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0053] Figure 1 This is a flow chart of the distributed training method for a large language model based on dynamic resource scheduling provided by the present invention. Figure 1 As shown, the method includes:

[0054] Step 110: For a distributed training cluster comprising heterogeneous computing nodes, resource status data of each computing node is collected at periodic time intervals; the resource status data includes GPU computing power utilization, video memory occupancy, network bandwidth remaining amount, and gradient data distribution characteristics;

[0055] Step 120: In the current training batch, the training task of the large language model is divided into multiple subtasks. Based on the resource status data of each computing node and the task description data of each subtask, a reinforcement learning strategy is used to allocate each subtask to the optimal computing node in an optimal ratio; the task description data includes computing intensity, video memory requirement, communication dependency, execution latency, and task type label;

[0056] Step 130: compressing the gradient data generated on the computing node using a gradient compression algorithm, and dynamically adjusting the compression rate of the gradient data in combination with the current network bandwidth utilization of the computing node;

[0057] In step 140 , the parameter server performs weighted fusion on the compressed gradient data of different computing nodes based on the parameter synchronization interval, updates the global model parameters based on the fusion result, and broadcasts the global model parameters to each computing node.

[0058] Here, a distributed training cluster consisting of heterogeneous compute nodes can be constructed. This distributed training cluster includes at least one parameter server (equipped with a high-bandwidth network interface card) and multiple compute nodes. The compute nodes use a hybrid GPU architecture (for example, four NVIDIA A100 nodes and eight V100 nodes). The parameter server and compute nodes communicate via the RDMA protocol, and the network topology adopts a fat-tree architecture. Furthermore, a distributed training framework (such as PyTorch Distributed or Horovod) is deployed on both the parameter server and the compute nodes. A dynamic resource monitoring agent is installed on each compute node to periodically collect resource status data from each compute node. This resource status data includes GPU computing power utilization, video memory usage, remaining network bandwidth, and gradient data distribution characteristics. The periodic interval can be 1 second. Resource status data is collected by a lightweight agent deployed on each compute node, which can store historical resource status data in a time series database. In some embodiments, the monitoring agent of each computing node uploads the collected resource status data to the status database of the parameter server (for example, using Redis to implement real-time data caching), which stores the resource status data sequence of the last 10 minutes by timestamp for subsequent dynamic resource analysis and scheduling.

[0059] After collecting the resource status data of each computing node, the resource status data of each computing node can be encoded into a vector, for example: '[GPU utilization, video memory usage, bandwidth remaining]', and the vector is normalized to eliminate dimensional differences.

[0060] For the training tasks of large language models, training is usually performed in batches. In the current training batch, the training tasks of the large language model can be divided into multiple subtasks, such as forward propagation tasks (Forward), backpropagation tasks (Backward), etc. Among them, the classification method of the training tasks can be determined based on the parallel training strategy supported by the current environment, and the embodiments of the present invention do not specifically limit this. Since a training batch contains multiple training samples (for example, 10,000), there are also multiple subtasks of the same category. In order to better improve the training efficiency and stability of large-scale language models, based on the resource status data of each computing node and the task description data of each subtask, a reinforcement learning strategy can be used to allocate each subtask to the optimal computing node in the optimal proportion. Here, the task description data includes computing intensity, video memory requirements, communication dependency, execution delay and task type label. It should be noted that when reinforcement learning strategies are used to allocate various subtasks to the optimal computing nodes in the optimal proportion, subtasks of the same or different types can be executed on the same computing node, and multiple subtasks of the same type can be distributed to different computing nodes to improve the resource utilization of the computing nodes, thereby improving the training efficiency and stability of large language models.

[0061] In some embodiments, as Figure 2 As shown, based on the resource status data of each computing node and the task description data of each subtask, the reinforcement learning strategy is used to allocate each subtask to the optimal computing node in the optimal ratio, including:

[0062] Step 210: Construct a state space consisting of the GPU utilization rate, video memory occupancy rate, remaining network bandwidth, amount of gradient data in the current batch, and the task description vector output by the graph neural network based on the task description data of various subtasks.

[0063] Step 220: construct an action space including a task allocation weight matrix; wherein the task allocation weight matrix defines the task ratio of each computing node to each subtask in the corresponding training batch;

[0064] Step 230 : Based on the preset reward function, the state space, and the action space, the pre-trained deep Q network is used to determine the subtask type undertaken by each computing node and the task ratio of the corresponding subtask type.

[0065] A pre-trained deep Q-network can be used to implement scheduling decisions based on reinforcement learning technology, and the model inference frequency is synchronized with periodic time intervals (decision execution occurs once per second). The state space of the deep Q-network encompasses each compute node's GPU utilization, memory occupancy, remaining network bandwidth, the amount of gradient data in the current batch, and the task description vectors output by the graph neural network based on the task description data of each subtask. That is, the input to the deep Q-network can be a multidimensional vector consisting of each compute node's current GPU utilization, memory occupancy, remaining network bandwidth, the amount of gradient data in the current batch, and the task description vectors of each subtask in the current training batch output by the graph neural network. Here, a task node graph can be constructed based on the task description data of each subtask and the temporal dependencies between the subtasks. Each node in the task node graph corresponds to a type of subtask, and the node attributes of each node correspond to the task description data of the corresponding subtask. This task node graph is input into the graph neural network, and the node vector of each node output by the graph neural network can be obtained as the task description vector of the corresponding type of subtask.

[0066] The action space of the deep Q network includes a task allocation weight matrix, wherein the task allocation weight matrix defines the task ratio of each subtask undertaken by each computing node in the corresponding training batch. That is, the output of the deep Q network can be the task ratio of each subtask undertaken by each computing node in the corresponding training batch. In other embodiments, in order to take into account the communication overhead in the distributed training process, the action space may also include a parameter synchronization interval, that is, the output of the deep Q network includes the task ratio of each subtask undertaken by each computing node in the corresponding training batch and the parameter synchronization interval. The parameter synchronization interval determines the parameter synchronization frequency between each computing node and the parameter server. Therefore, by optimizing the parameter synchronization interval in real time through the deep Q network, the task allocation and parameter synchronization strategies can be jointly optimized to further improve the training efficiency and stability of the large language model.

[0067] Based on the preset reward function and the state space and action space designed above, the deep Q network can output the subtask type undertaken by each computing node and the task ratio of the corresponding subtask type, realizing efficient distributed training of large language models.

[0068] In some embodiments, the preset reward function is specifically:

[0069] R = α × resource balancing coefficient + β × single iteration time reduction rate - γ × node overload penalty

[0070] Among them, α, β and γ are preset weight coefficients respectively, and the resource balancing coefficient is determined based on the inverse of the variance of the resource utilization of each computing node.

[0071] In other embodiments, the parameter synchronization interval can also be dynamically adjusted based on the network bandwidth of each computing node. When the remaining network bandwidth of each computing node exceeds a preset threshold, the parameter synchronization interval is shortened to the minimum allowable value; when the remaining network bandwidth of any computing node falls below the preset threshold, the parameter synchronization interval is extended to the maximum allowable value.

[0072] Compute nodes generate model gradient data when executing subtasks. Large amounts of gradient data can create significant communication and storage pressures. To address this, a gradient compression algorithm can be used to compress the gradient data generated on compute nodes. The compression ratio of the gradient data on the corresponding compute node can be dynamically adjusted based on the compute node's current network bandwidth utilization. This prevents compute node overload due to excessive gradient data volume, thereby balancing model training efficiency and accuracy.

[0073] In some embodiments, as Figure 3 As shown, the gradient data generated on the computing node is compressed using a gradient compression algorithm, and the compression rate of the gradient data is dynamically adjusted in combination with the current network bandwidth utilization of the computing node, including:

[0074] Step 310 , performing sparse processing on the gradient data generated on the computing node based on the current compression rate, and generating a non-zero gradient index table;

[0075] Step 320 , dynamically quantize and encode the sparse gradient data, and map the 32-bit floating point value to an 8-bit integer space according to the gradient distribution range;

[0076] Step 330: monitor the network bandwidth utilization of the computing node in real time. If the current bandwidth utilization exceeds a preset threshold, reduce the compression rate and enable secondary Huffman coding to compress the non-zero gradient index table.

[0077] Specifically, the gradient data (i.e., gradient tensors) generated on the compute nodes are flattened into one-dimensional vectors. The absolute values ​​of each gradient are calculated and sorted. Based on the current compression ratio (denoted as K%), the absolute values ​​of the top K% of the gradients are retained and the remaining absolute values ​​are set to zero to achieve sparsification. Simultaneously, the index positions of non-zero gradient absolute values ​​are recorded to generate a non-zero gradient index table. The sparsified gradient data is then dynamically quantized and encoded, mapping the 32-bit floating-point values ​​to an 8-bit integer space based on the gradient distribution range.

[0078] For example, quantization coding can be implemented based on the following:

[0079] quantized_grad=np.round(sparse_grad×scale_factor).astype(np.int8)

[0080] The quantization range scale_factor of dynamic quantization coding can be determined by the following formula:

[0081] scale_factor=127max(∣gradmin∣,∣gradmax∣)

[0082] Here, gradmin and gradmax are the minimum and maximum values ​​of the sparsified gradients for the current batch.

[0083] It should be noted that on the parameter server side, the gradient data after quantization encoding can be dequantized based on the following method:

[0084] dequantized_grad = quantized_grad.astype(np.float32) / scale_factor

[0085] In addition, the network bandwidth utilization of the computing nodes can be monitored in real time. When the current bandwidth utilization exceeds the preset threshold, the aforementioned compression rate is reduced, and the secondary Huffman coding compression non-zero gradient index table is enabled.

[0086] For example, if bandwidth utilization is greater than 80%, the compression rate is reduced from 10% to 5% to further reduce the data volume; if bandwidth utilization is less than 50%, the compression rate is restored to 10%, prioritizing model accuracy.

[0087] The compressed gradient data of the computing nodes will be periodically uploaded to the parameter server at the parameter synchronization interval. The parameter server will then perform weighted fusion on the compressed gradient data of different computing nodes (the weights corresponding to each computing node can be pre-set), thereby updating the global model parameters based on the fusion results and broadcasting the global model parameters to each computing node as the starting point for the next training.

[0088] In some embodiments, in order to ensure high availability of training and reduce the impact of computing node failures, computing node failure detection can be performed based on an elastic fault tolerance mechanism, and the training tasks for which the failed node is responsible can be restored based on incremental checkpoint snapshots and task migration priority queues.

[0089] Specifically, snapshots of full checkpoints and incremental checkpoints can be saved at preset intervals. For example, a full checkpoint (storing the complete model parameters and optimizer state) can be saved every 30 minutes, and incremental checkpoints (recording only the difference gradients from the previous snapshot) can be saved every 5 minutes. Incremental checkpoints store only the gradient difference data from the previous snapshot, compressed using a differential encoding algorithm. In some embodiments, the difference ΔW between the current parameter Wt and the parameter Wt−1 saved in the previous checkpoint can be calculated at the incremental checkpoint. ΔW is then sparsified and compressed using the Zstandard compression algorithm, ensuring that the compressed data size is less than 30% of the full checkpoint. A heartbeat detection mechanism monitors the liveness of compute nodes. For example, a parameter server can send heartbeat packets to compute nodes every 5 seconds and determine whether a compute node is faulty based on the number of consecutive unresponsive times. For example, if a compute node fails to respond three times in a row (with a cumulative timeout of 15 seconds), it is marked as faulty.

[0090] For a faulty node, you can immediately stop assigning new tasks to the faulty node and record its unfinished tasks. Restore the model parameters based on the snapshot of the latest full checkpoint and the snapshot of the latest incremental checkpoint corresponding to the faulty node, and add the unfinished tasks of the faulty node to the task migration priority queue, so that the unfinished tasks can be migrated to the backup node based on the priority of the unfinished tasks. Among them, the unfinished tasks of the faulty node can be added to the task migration priority queue with a higher priority, and according to the current resource status, the backup node with the lowest load is selected to execute the unfinished tasks of the faulty node. In addition, it is necessary to verify the integrity of the model parameters through the consistency protocol, that is, compare the restored model parameters with the latest parameters of the healthy node to ensure consistency. If the difference between the restored model parameters and the latest parameters of the healthy node exceeds the threshold, a full synchronization is triggered, and the model parameters corresponding to the faulty node are updated as the training starting point for the unfinished tasks.

[0091] In summary, the method provided by the embodiment of the present invention collects the resource status data of each computing node at periodic time intervals, thereby dividing the training task of the large language model into multiple subtasks in the current training batch, and based on the resource status data of each computing node and the task description data of each subtask, uses a reinforcement learning strategy to allocate each subtask to the optimal computing node in the optimal proportion; in addition, a gradient compression algorithm is used to compress the gradient data generated on the computing node, and the compression rate of the gradient data is dynamically adjusted in combination with the current network bandwidth utilization of the computing node; the parameter server then performs weighted fusion on the compressed gradient data of different computing nodes based on the parameter synchronization interval, updates the global model parameters based on the fusion results, and broadcasts the global model parameters to each computing node, which can significantly improve the training efficiency of the large language model and achieve a significant improvement in resource utilization under the same hardware conditions to support the stable training of models with hundreds of billions of parameters.

[0092] The following describes the large language model distributed training system based on dynamic resource scheduling provided by the present invention. The large language model distributed training system based on dynamic resource scheduling described below and the large language model distributed training method based on dynamic resource scheduling described above can be referenced to each other.

[0093] Based on any of the above embodiments, Figure 4 This is a schematic diagram of the structure of the large language model distributed training system based on dynamic resource scheduling provided by the present invention. Figure 4 As shown, the system includes:

[0094] Resource monitoring module 410 is used to collect resource status data of each computing node at periodic time intervals for a distributed training cluster containing heterogeneous computing nodes; the resource status data includes GPU computing power utilization, video memory occupancy, network bandwidth remaining, and gradient data distribution characteristics;

[0095] Dynamic scheduling decision module 420 is used to divide the training task of the large language model into multiple subtasks in the current training batch and use a reinforcement learning strategy to allocate the subtasks to the optimal computing nodes in an optimal ratio based on the resource status data of each computing node and the task description data of each subtask; the task description data includes computing intensity, video memory requirements, communication dependencies, execution latency, and task type labels;

[0096] A gradient optimization communication module 430 is configured to compress the gradient data generated on the computing node using a gradient compression algorithm and dynamically adjust the compression rate of the gradient data based on the current network bandwidth utilization of the computing node;

[0097] The parameter update module 440 is used to perform weighted fusion on the compressed gradient data of different computing nodes based on the parameter synchronization interval on the parameter server side, update the global model parameters based on the fusion result, and broadcast the global model parameters to each computing node.

[0098] The system provided by the embodiment of the present invention collects resource status data of each computing node at periodic time intervals, thereby dividing the training task of the large language model into multiple subtasks in the current training batch, and based on the resource status data of each computing node and the task description data of each subtask, uses a reinforcement learning strategy to allocate each subtask to the optimal computing node in the optimal proportion; in addition, a gradient compression algorithm is used to compress the gradient data generated on the computing node, and the compression rate of the gradient data is dynamically adjusted in combination with the current network bandwidth utilization of the computing node; the parameter server then performs weighted fusion on the compressed gradient data of different computing nodes based on the parameter synchronization interval, updates the global model parameters based on the fusion results, and broadcasts the global model parameters to each computing node, which can significantly improve the training efficiency of the large language model and achieve a significant improvement in resource utilization under the same hardware conditions to support the stable training of models with hundreds of billions of parameters.

[0099] Based on any of the above embodiments, the method of allocating various subtasks to optimal computing nodes in an optimal ratio using a reinforcement learning strategy based on the resource status data of each computing node and the task description data of each subtask includes:

[0100] Construct a state space consisting of each computing node's GPU utilization, video memory occupancy, remaining network bandwidth, the amount of gradient data in the current batch, and the task description vector output by the graph neural network based on the task description data of various subtasks;

[0101] Constructing an action space including a task allocation weight matrix; wherein the task allocation weight matrix defines the task proportion of each subtask undertaken by each computing node in the corresponding training batch;

[0102] Based on a preset reward function, the state space, and the action space, a pre-trained deep Q network is used to determine the subtask type undertaken by each computing node and the task ratio of the corresponding subtask type.

[0103] Based on any of the above embodiments, the preset reward function is specifically:

[0104] R = α × resource balancing coefficient + β × single iteration time reduction rate - γ × node overload penalty

[0105] Among them, α, β and γ are preset weight coefficients respectively, and the resource balancing coefficient is determined based on the inverse of the variance of the resource utilization of each computing node.

[0106] Based on any of the above embodiments, compressing the gradient data generated on the computing node using a gradient compression algorithm and dynamically adjusting the compression rate of the gradient data in combination with the current network bandwidth utilization of the computing node includes:

[0107] Based on the current compression rate, the gradient data generated on the computing node is sparsely processed and a non-zero gradient index table is generated;

[0108] Dynamically quantize and encode the sparse gradient data, mapping the 32-bit floating-point value to the 8-bit integer space according to the gradient distribution range;

[0109] The network bandwidth utilization of the computing node is monitored in real time. If the current bandwidth utilization exceeds a preset threshold, the compression rate is reduced, and quadratic Huffman coding is enabled to compress the non-zero gradient index table.

[0110] Based on any of the above embodiments, the quantization range of the dynamic quantization coding is determined by the following formula:

[0111] scale_factor=127max(∣gradmin∣,∣gradmax∣)

[0112] Among them, gradmin and gradmax are the minimum and maximum values ​​of the sparse gradient of the current batch.

[0113] Based on any of the above embodiments, the system further includes a fault-tolerant module, configured to:

[0114] Compute node failure detection is performed based on an elastic fault-tolerance mechanism, and training tasks for the failed node are restored based on incremental checkpoint snapshots and task migration priority queues.

[0115] Based on any of the above embodiments, the method of detecting a fault on a computing node based on an elastic fault tolerance mechanism and recovering the training task for the failed node based on an incremental checkpoint snapshot and a task migration priority queue includes:

[0116] Save snapshots of full checkpoints and incremental checkpoints at preset time intervals. The incremental checkpoint only stores gradient difference data from the previous snapshot; the gradient difference data is compressed using a differential encoding algorithm.

[0117] Monitor the survival status of the computing node through a heartbeat detection mechanism, and determine whether the computing node is a faulty node based on the number of consecutive non-response times;

[0118] For a failed node, the model parameters are restored based on the snapshot of the latest full checkpoint and the snapshot of the latest incremental checkpoint corresponding to the failed node, and the unfinished tasks of the failed node are added to the task migration priority queue, so that the unfinished tasks are migrated to the backup node based on the priority of the unfinished tasks, and the integrity of the model parameters is verified through the consistency protocol.

[0119] Based on any of the above embodiments, the system further includes a parameter synchronization optimization module, which is configured to:

[0120] Dynamically adjusting the parameter synchronization interval based on the network bandwidth of each computing node;

[0121] When the remaining network bandwidth of each computing node is higher than a preset threshold, the parameter synchronization interval is shortened to the minimum allowed value;

[0122] When the remaining network bandwidth of any computing node is lower than a preset threshold, the parameter synchronization interval is extended to a maximum allowable value.

[0123] Based on any of the above embodiments, the periodic time interval is 1 second, and the resource status data is collected through a lightweight agent program, the agent program is deployed on each computing node, and the historical resource status data sequence is stored through a time series database.

[0124] Figure 5 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor (processor) 510, a memory (memory) 520, a communication interface (Communications Interface) 530 and a communication bus 540, wherein the processor 510, the memory 520, and the communication interface 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 520 to execute a large language model distributed training method based on dynamic resource scheduling, the method including: for a distributed training cluster including heterogeneous computing nodes, collecting resource status data of each computing node at periodic time intervals; the resource status data includes GPU computing power utilization, video memory occupancy, network bandwidth remaining amount and gradient data distribution characteristics; in the current training batch, the training task of the large language model is divided into multiple subtasks, and based on the resource status data of each computing node and the task description data of each subtask, the reinforcement learning strategy is used to allocate each subtask to the optimal computing node in an optimal proportion; the task description data includes computing intensity, video memory requirement, communication dependency, execution delay and task type label; a gradient compression algorithm is used to compress the gradient data generated on the computing node, and the compression rate of the gradient data is dynamically adjusted in combination with the current network bandwidth utilization of the computing node; the parameter server performs weighted fusion on the compressed gradient data of different computing nodes based on the parameter synchronization interval, updates the global model parameters based on the fusion result, and broadcasts the global model parameters to each computing node.

[0125] In addition, the logic instructions in the aforementioned memory 520 can be implemented in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0126] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the large language model distributed training method based on dynamic resource scheduling provided by the above methods, the method comprising: for a distributed training cluster including heterogeneous computing nodes, collecting resource status data of each computing node at periodic time intervals; the resource status data includes GPU computing power utilization, video memory occupancy, network bandwidth remaining amount and gradient data distribution characteristics; in the current training batch, the training tasks of the large language model are divided into The system is divided into multiple subtasks, and based on the resource status data of each computing node and the task description data of each subtask, a reinforcement learning strategy is used to allocate each subtask to the optimal computing node in the optimal ratio; the task description data includes computing intensity, video memory requirements, communication dependencies, execution delay and task type label; a gradient compression algorithm is used to compress the gradient data generated on the computing node, and the compression rate of the gradient data is dynamically adjusted in combination with the current network bandwidth utilization of the computing node; the parameter server performs weighted fusion of the compressed gradient data of different computing nodes based on the parameter synchronization interval, updates the global model parameters based on the fusion results, and broadcasts the global model parameters to each computing node.

[0127] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned large language model distributed training method based on dynamic resource scheduling, the method comprising: for a distributed training cluster comprising heterogeneous computing nodes, collecting resource status data of each computing node at periodic time intervals; the resource status data includes GPU computing power utilization, video memory occupancy, network bandwidth remaining amount and gradient data distribution characteristics; in the current training batch, dividing the training task of the large language model into multiple subtasks, and based on the resource status data of each computing node and the task description data of each subtask, using a reinforcement learning strategy to allocate each subtask to the optimal computing node in an optimal proportion; the task description data includes computing intensity, video memory requirement, communication dependency, execution delay and task type label; using a gradient compression algorithm to compress the gradient data generated on the computing node, and dynamically adjusting the compression rate of the gradient data in combination with the current network bandwidth utilization of the computing node; the parameter server performs weighted fusion on the compressed gradient data of different computing nodes based on the parameter synchronization interval, updates the global model parameters based on the fusion result, and broadcasts the global model parameters to each computing node.

[0128] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0129] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A large language model distributed training method based on dynamic resource scheduling, characterized in that: include: For a distributed training cluster containing heterogeneous computing nodes, the resource status data of each computing node is collected at periodic time intervals; The resource status data includes GPU computing power utilization, video memory occupancy, network bandwidth remaining amount and gradient data distribution characteristics; In the current training batch, the large language model training task is divided into multiple subtasks. Based on the resource status data of each computing node and the task description data of each subtask, a reinforcement learning strategy is used to allocate each subtask to the optimal computing node in the optimal ratio. The task description data includes computing intensity, video memory requirements, communication dependencies, execution latency, and task type labels. Compressing the gradient data generated on the computing node using a gradient compression algorithm, and dynamically adjusting the compression rate of the gradient data in combination with the current network bandwidth utilization of the computing node; The parameter server performs weighted fusion on the compressed gradient data of different computing nodes based on the parameter synchronization interval, updates the global model parameters based on the fusion results, and broadcasts the global model parameters to each computing node; The method further comprises: Save snapshots of full checkpoints and incremental checkpoints at preset time intervals. The incremental checkpoint only stores gradient difference data from the previous snapshot; the gradient difference data is compressed using a differential encoding algorithm. Monitor the survival status of the computing node through a heartbeat detection mechanism, and determine whether the computing node is a faulty node based on the number of consecutive non-response times; For a failed node, the model parameters are restored based on the snapshot of the latest full checkpoint and the snapshot of the latest incremental checkpoint corresponding to the failed node, and the unfinished tasks of the failed node are added to the task migration priority queue, so that the unfinished tasks are migrated to the backup node based on the priority of the unfinished tasks, and the integrity of the model parameters is verified through the consistency protocol.

2. The large language model distributed training method based on dynamic resource scheduling according to claim 1 is characterized in that: The method uses a reinforcement learning strategy based on the resource status data of each computing node and the task description data of each subtask to allocate each subtask to the optimal computing node in an optimal ratio, including: Construct a state space consisting of each computing node's GPU utilization, video memory occupancy, remaining network bandwidth, the amount of gradient data in the current batch, and the task description vector output by the graph neural network based on the task description data of various subtasks; Constructing an action space including a task allocation weight matrix; wherein the task allocation weight matrix defines the task proportion of each subtask undertaken by each computing node in the corresponding training batch; Based on a preset reward function, the state space, and the action space, a pre-trained deep Q network is used to determine the subtask type undertaken by each computing node and the task ratio of the corresponding subtask type.

3. The large language model distributed training method based on dynamic resource scheduling according to claim 2 is characterized in that: The preset reward function is specifically: R = α × resource balancing coefficient + β × single iteration time reduction rate - γ × node overload penalty Among them, α, β and γ are preset weight coefficients respectively, and the resource balancing coefficient is determined based on the inverse of the variance of the resource utilization of each computing node.

4. The large language model distributed training method based on dynamic resource scheduling according to claim 1 is characterized in that: The method of compressing the gradient data generated on the computing node using a gradient compression algorithm and dynamically adjusting the compression rate of the gradient data in combination with the current network bandwidth utilization of the computing node includes: Based on the current compression rate, the gradient data generated on the computing node is sparsely processed and a non-zero gradient index table is generated; Dynamically quantize and encode the sparse gradient data, mapping the 32-bit floating-point value to the 8-bit integer space according to the gradient distribution range; The network bandwidth utilization of the computing node is monitored in real time. If the current bandwidth utilization exceeds a preset threshold, the compression rate is reduced, and quadratic Huffman coding is enabled to compress the non-zero gradient index table.

5. The large language model distributed training method based on dynamic resource scheduling according to claim 4 is characterized in that: The quantization range of the dynamic quantization coding is determined by the following formula: scale_factor=127max(∣gradmin∣,∣gradmax∣) Among them, gradmin and gradmax are the minimum and maximum values ​​of the sparse gradient of the current batch.

6. The large language model distributed training method based on dynamic resource scheduling according to claim 1 is characterized in that: The method further comprises: Dynamically adjusting the parameter synchronization interval based on the network bandwidth of each computing node; When the remaining network bandwidth of each computing node is higher than a preset threshold, the parameter synchronization interval is shortened to the minimum allowed value; When the remaining network bandwidth of any computing node is lower than a preset threshold, the parameter synchronization interval is extended to a maximum allowable value.

7. The large language model distributed training method based on dynamic resource scheduling according to claim 1 is characterized in that: The periodic time interval is 1 second, and the resource status data is collected by a lightweight agent program, which is deployed on each computing node and stores the historical resource status data sequence through a time series database.

8. A large language model distributed training system based on dynamic resource scheduling, characterized in that: include: The resource monitoring module is used to collect resource status data of each computing node at periodic time intervals for a distributed training cluster containing heterogeneous computing nodes; The resource status data includes GPU computing power utilization, video memory occupancy, network bandwidth remaining amount and gradient data distribution characteristics; A dynamic scheduling decision module is used to divide the training tasks of the large language model into multiple subtasks in the current training batch. Based on the resource status data of each computing node and the task description data of each subtask, it uses a reinforcement learning strategy to allocate the subtasks to the optimal computing nodes in an optimal ratio. The task description data includes computing intensity, video memory requirements, communication dependencies, execution latency, and task type labels. A gradient optimization communication module is used to compress the gradient data generated on the computing node using a gradient compression algorithm, and dynamically adjust the compression rate of the gradient data in combination with the current network bandwidth utilization of the computing node; The parameter update module is used to perform weighted fusion of the compressed gradient data of different computing nodes based on the parameter synchronization interval on the parameter server side, update the global model parameters based on the fusion results, and broadcast the global model parameters to each computing node; The system further includes a fault-tolerant module for: Save snapshots of full checkpoints and incremental checkpoints at preset time intervals. The incremental checkpoint only stores gradient difference data from the previous snapshot; the gradient difference data is compressed using a differential encoding algorithm. Monitor the survival status of the computing node through a heartbeat detection mechanism, and determine whether the computing node is a faulty node based on the number of consecutive non-response times; For a failed node, the model parameters are restored based on the snapshot of the latest full checkpoint and the snapshot of the latest incremental checkpoint corresponding to the failed node, and the unfinished tasks of the failed node are added to the task migration priority queue, so that the unfinished tasks are migrated to the backup node based on the priority of the unfinished tasks, and the integrity of the model parameters is verified through the consistency protocol.

Images