Inter-server gradient communication and computing collaborative scheduling method for large model training

Abstract

The application provides an inter-server gradient communication and calculation cooperative scheduling method for large model training, GPUs of each calculation server perform forward and backward propagation processes of model training to generate local gradient tensors, and gradient importance is evaluated through a gradient sensing and compression module; according to the classification result of the gradient, a hierarchical dynamic compression strategy is executed; the optimal communication topology and transmission path are dynamically selected, and the compressed gradient package is transmitted to other calculation servers and a global parameter server; the execution sequence of the calculation task and the communication task is coordinated through a double buffering mechanism to realize parallel execution of gradient transmission and next training batch calculation; the global parameter server performs asynchronous gradient aggregation, restores the global gradient and prepares parameter update. The application reduces the gradient communication data volume by 70%-80%, shortens the communication time by more than 60%, greatly alleviates the communication bottleneck problem in large model training, and maximizes the reduction of communication data volume under the premise of ensuring accuracy.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, distributed computing, large model training, and communication scheduling, and in particular to a method for inter-server gradient communication and collaborative scheduling of computation for large model training. Background Technology

[0002] With the rapid development of artificial intelligence technology, the parameter scale of large models continues to break through, expanding from tens of billions of parameters (such as BERT-large) to hundreds of billions (such as GPT-3) and trillions (such as PaLM-2). Single-machine training can no longer meet the computing power and memory requirements of large model training, and distributed parallel training has become the mainstream technical solution for large model training.

[0003] The current mainstream distributed training architectures for large models mainly include three types: data parallelism (DP), model parallelism (MP), and pipeline parallelism (PP). All three architectures rely on gradient synchronization communication between multiple servers. The core communication operations include AllReduce, AllGather, and ReduceScatter. Among them, the AllReduce operation has become the most core communication primitive in data parallelism and model parallelism architectures because it can realize global gradient aggregation.

[0004] In existing large-scale model training systems, gradient communication and local computation tasks are executed independently, using a serial mode where communication occurs only after computation is completed. After completing local gradient computation, the GPU must wait for gradient transmissions from all servers to be completed and aggregated before starting the next batch of computation tasks. This results in the GPU frequently switching between computation and waiting for communication. According to actual test data, using the existing scheduling method, GPU utilization is generally below 60%, and in some scenarios, it is even below 40%, resulting in a significant waste of computing resources. Moreover, the amount of gradient data in large models is directly proportional to the parameter size. The gradient tensor dimension of a model with hundreds of billions of parameters can reach hundreds of billions, and the amount of data in a single gradient synchronization can reach 1-2TB (using FP32 precision). Even using FP16 mixed precision, the data volume can still reach 500GB-1TB. Limited by network bandwidth (existing data centers have a 10Gbps bandwidth for 10Gbps Ethernet, and even using 400Gbps high-speed Ethernet, transmitting 1TB of data would take more than 2000 seconds), gradient communication time accounts for 40%-60% of the total training time, becoming a core bottleneck restricting the training cycle of large models. Meanwhile, in existing distributed training systems, communication topologies (such as ring, tree, and butterfly topologies) and communication primitives (such as AllReduce and ReduceScatter) are configured with fixed parameters, making it impossible to dynamically adjust based on real-time network conditions (bandwidth, latency, packet loss rate). For example, when network congestion occurs between some servers, a fixed ring AllReduce topology can lead to a sharp increase in communication latency; while when the network is in good condition, a tree topology cannot fully utilize bandwidth resources, resulting in bandwidth utilization of less than 50%. Furthermore, existing communication strategies do not consider topological distances and load differences between servers, causing some servers to become communication bottlenecks and further reducing overall communication efficiency. In addition, large-scale distributed training clusters contain heterogeneous resources such as CPUs, GPUs, and RDMA network cards. Existing scheduling schemes schedule these resources independently: GPUs are only responsible for model computation, CPUs for gradient compression, encoding, and communication control, and RDMA network cards are only responsible for data transmission. The lack of a coordination mechanism among these three components leads to resource waste and efficiency bottlenecks. For example, when the GPU is performing computational tasks, the CPU and RDMA network card may be idle; while when the CPU is performing gradient compression, the GPU may be in a waiting state; the high bandwidth advantage of the RDMA network card is not fully utilized, and in some scenarios, gradient data is still forwarded through the CPU, which increases communication latency.

[0005] Therefore, there is an urgent need to provide a novel collaborative scheduling method to overcome the technical bottlenecks in training large models. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a server-side gradient communication and computational collaborative scheduling method for large model training. This invention enables deep collaboration between gradient communication and computational tasks, efficient utilization of heterogeneous resources, and a significant reduction in gradient communication overhead.

[0007] In a first aspect, the technical solution provided by this invention is: a server-to-server gradient communication and computational collaborative scheduling method for large model training, comprising the following steps: S1) The GPUs of each computing server execute the forward and backward propagation process of model training and generate local gradient tensors. The importance of gradients is evaluated through the gradient awareness and compression module. S2) The gradient sensing and compression module executes a hierarchical dynamic compression strategy based on the gradient classification results, and uses different compression and encoding methods for different types of gradients to generate compressed gradient packages and residual compensation data. S3) Real-time acquisition of network status, dynamic selection of optimal communication topology and transmission path, and compression of gradient packets. Transmitted to other computing servers and global parameter server; S4) The execution order of computation and communication tasks is coordinated through a double buffering mechanism to achieve parallel execution of gradient transfer and computation of the next training batch; S5) The global parameter server performs asynchronous gradient aggregation, restores the global gradient, and prepares for parameter updates; S6) Each computing server updates the model parameters based on the received global gradient and dynamically adjusts the scheduling strategy for the next round based on the real-time feedback information. S7) When any termination condition is met, stop the training iteration and output the final model parameters.

[0008] Preferably, in step S1), the GPU of the computation server performs the forward and backward propagation process of model training and generates a local gradient tensor, as follows: S11) Load the current batch of training data to the GPU, whereby the GPU performs forward propagation of model training and calculates the loss function between the model prediction and the actual value; S12) The GPU performs backpropagation for model training and calculates the local gradient tensor G of each layer of the model based on the loss function.

[0009] Preferably, in step S1), the importance of the gradient is evaluated through a gradient sensing and compression module; specifically as follows: S13) Extract gradient magnitude from local gradient tensor G Historical update frequency Importance of the model layer ; S14), based on gradient magnitude Historical update frequency Importance of the model layer Calculate gradient importance score ; S15) Score all gradients by importance. Sort the gradients in descending order and construct a gradient importance ranking table. Based on a preset threshold, the gradients are divided into three categories: critical gradients, important gradients, and redundant gradients. Importance score The first 20% of gradients are considered critical gradients; Importance score Gradients between 20% and 80% are considered important gradients; Importance score The last 20% of the gradients are used as redundant gradients.

[0010] Preferably, in step S2), the gradient sensing and compression module executes a hierarchical dynamic compression strategy based on the gradient classification results, as follows: S21) For critical gradients, FP8 mixed-precision coding is used to convert the original FP32 and FP16 precision gradients to FP8 precision. During the coding process, lossless compression is used to compress the gradients to generate the critical gradient coded packets. ; S22) For important gradients, retain the top K proportion of gradient data and set the remaining gradients in the important gradients to 0; calculate the residuals of the gradients set to 0; compress the residuals using quantization coding levels to generate important gradient coding packets. and residual coding packets ; S23) For redundant gradients, quantize the redundant gradients to 1-bit, reducing the data volume to 1 / 32 of the original FP32 precision; calculate the residuals of the quantized redundant gradients. Through a momentum replenishment mechanism, the residual is accumulated into the corresponding gradient of the next iteration to achieve accuracy compensation; this generates redundant gradient encoding packets. With residual cumulative value ; S24) Encode the key gradient packet Important gradient encoding packets and residual coding packets Redundant gradient coding packets With residual cumulative value Integrate into a unified compressed gradient package Add packet header information and calculate the compression gradient packet. Verification code .

[0011] Preferably, in step S3), the network status is collected in real time, the optimal communication topology and transmission path are dynamically selected, and the compressed gradient packet is compressed. The data is then transferred to other computing servers and the global parameter server; specifically as follows: S31) The network status parameters between each computing server are collected in real time through the SDN controller, including bandwidth B, one-way delay D, packet loss rate H, and network congestion level C; and the network status parameters are normalized. A network status evaluation matrix is ​​constructed with each computing server and the global parameter server in the distributed training cluster as rows and the normalized network status parameters as columns. S32) Based on the network state evaluation matrix, a topology selection algorithm is used to dynamically select the optimal communication topology; S33) Match the corresponding communication primitives according to the selected optimal communication topology; S34) Based on the topological distance and network status of the servers participating in the communication, for each compressed gradient packet... Allocate the optimal transmission path.

[0012] Preferably, in step S4), a double-buffering mechanism is used to coordinate the execution order of computation and communication tasks, so as to achieve parallel execution of gradient transfer and computation of the next training batch; specifically as follows: S41) Initialize two buffer queues, Buffer A and Buffer B, each buffer queue is used to store gradient data and computation task information; Among them, buffer queue A is used to store the gradient data of the current batch; buffer queue B is used to store the computation task data of the next batch. S42) After the GPU completes the gradient calculation for the current batch, it writes the gradient data into the buffer queue Buffer A and sends gradient processing instructions to the gradient sensing and compression module. S43) The gradient sensing and compression module reads gradient data from the buffer queue Buffer A, performs hierarchical compression and encoding, and generates compressed gradient packets. Simultaneously, the computing-communication pipeline scheduling module sends the next batch of computing instructions to the GPU, and the GPU reads the training data from Buffer B to start the next batch of forward and backward propagation computing. S44), compressed gradient package After generation, the communication module reads the compressed gradient packet from the buffer queue Buffer A. Based on the optimized communication topology and transmission path, gradient transmission is initiated through the RDMA network card; S45) After the GPU completes the gradient calculation for the next batch, it writes the gradient data into the buffer queue Buffer B. At the same time, the communication module completes the gradient transmission for the current batch, and the buffer queue is switched. S46), Repeat steps S43 to S45). S47) The compute-communication pipeline scheduling module monitors GPU utilization and communication latency in real time and dynamically adjusts pipeline depth; When the GPU utilization is less than 85%, the pipeline depth is increased. In this embodiment, the pipeline depth is increased from 2 training batches to 3 training batches. When the communication latency is greater than or equal to the computation time, the pipeline depth is reduced. In this embodiment, the pipeline depth is reduced from 3 training batches to 2 training batches.

[0013] Secondly, the technical solution provided by this invention is: an inter-server gradient communication and computational collaborative scheduling system for large model training, comprising: The computation server is used to load the current batch of training data and global gradients, execute the backpropagation process of model training, and generate local gradient tensors. The gradient sensing and compression module is used to evaluate the importance of gradients. Based on the gradient classification results, it executes a hierarchical dynamic compression strategy, using different compression and encoding methods for different types of gradients to generate compressed gradient packets and residual compensation data. The communication topology sensing and optimization module is used to collect network status in real time, dynamically select the optimal communication topology and transmission path, and compress gradient packets. Transmitted to other computing servers and global parameter server; The computation-communication pipeline scheduling module coordinates the execution order of computation and communication tasks through a double buffering mechanism to achieve parallel execution of gradient transmission and computation of the next training batch. The global parameter server is used to perform asynchronous gradient aggregation, recover global gradients and prepare parameter updates, and send the updated global gradients to the computation server.

[0014] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned inter-server gradient communication and computational collaborative scheduling method for training large models.

[0015] The beneficial technical effects of this invention are as follows: 1. This invention constructs a gradient importance evaluation model by analyzing the magnitude of the gradient, historical update frequency, and model layer importance, and identifies key gradients and redundant gradients; 2. Based on gradient importance scores, this invention adopts a hierarchical compression strategy, using different compression methods and compression rates for gradients of different importance. Under the premise of controlling the accuracy loss to ≤2%, the amount of gradient communication data is reduced by 70%-80%, and the communication time is shortened by more than 60%, which greatly alleviates the communication bottleneck problem in the training of large models and maximizes the reduction of communication data while ensuring accuracy. 3. This invention uses SDN technology to collect network status (bandwidth, latency, packet loss rate) in real time, constructs a network status evaluation model, dynamically selects the optimal communication topology and communication primitives, and matches the network status in real time to select the optimal communication topology (ring, tree, butterfly) and transmission path, avoiding single link congestion, controlling gradient transmission latency within 1.5ms, and achieving a transmission success rate of over 99.9%, further improving communication efficiency. 4. This invention employs a double buffering mechanism to execute gradient compression, communication, and the next batch of computation tasks in parallel, effectively eliminating the idle time of the GPU waiting for communication and increasing GPU utilization from below 60% in the prior art to over 90%. At the same time, it dynamically adjusts the pipeline depth and balances the parallelism based on GPU utilization and communication latency, further optimizing the collaborative efficiency of computation and communication. 5. This invention implements a three-stage parallel pipeline of computation-compression-transmission. The GPU focuses on computational tasks such as model forward propagation and backward propagation, the CPU is responsible for non-computational tasks such as gradient compression, encoding, and data preloading, and the RDMA network card directly performs gradient transmission, realizing direct memory-to-memory transmission. This avoids task conflicts and resource idleness between different hardware, and keeps the utilization rate of CPU and RDMA network card within a reasonable range of 50%-70%, achieving efficient collaboration of all hardware resources. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the method of Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the process for generating compressed gradient packets and residual compensation data in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the process for selecting the optimal communication topology in Embodiment 1 of the present invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings: Example 1 like Figure 1 As shown, this embodiment provides a method for inter-server gradient communication and computational collaborative scheduling for large model training, including the following steps: S1) The GPUs of each computing server execute the forward and backward propagation processes of model training, generate local gradient tensors, and evaluate the importance of gradients through the gradient-aware and compression module; specifically, this includes the following steps: S11) Load the current batch of training data to the GPU, the GPU performs forward propagation, and calculates the loss function between the model's predicted values ​​and the actual values; S12) The GPU performs backpropagation and calculates the local gradient tensor G of each layer of the model based on the loss function; ; In the formula, The loss function; For the model number Layer parameters; S13) Extract gradient magnitude from local gradient tensor G Historical update frequency Importance of the model layer ; The gradient magnitude The magnitude is the absolute value of the gradient; the larger the magnitude, the greater the impact on model parameter updates. The historical update frequency This represents the number of times the gradient has been updated in the past N iterations. The higher the update frequency, the more significant the impact of the gradient on model convergence. Importance of the model layer To assign importance factors based on the functions of the model layers; S14), based on gradient magnitude Historical update frequency Importance of the model layer Calculate gradient importance score ,Right now: ; In the formula, , , Let be the weighting coefficient, satisfying In this embodiment, , , Set them to 0.5, 0.3, and 0.2 respectively; S15) Score all gradients by importance. Sort the gradients in descending order and construct a gradient importance ranking table. Then, classify the gradients into three categories based on a preset threshold: Importance score The first 20% of gradients are considered critical gradients; Importance score Gradients between 20% and 80% are considered important gradients; Importance score The last 20% of the gradients are used as redundant gradients.

[0018] S2) The gradient sensing and compression module executes a hierarchical dynamic compression strategy based on the gradient classification results, employing different compression and encoding methods for different types of gradients to generate compressed gradient packets and residual compensation data; for example... Figure 2 As shown, the details are as follows: S21) For critical gradients, FP8 mixed-precision coding is used to convert the original FP32 and FP16 precision gradients to FP8 precision. During the coding process, lossless compression is used to compress the gradients to generate the critical gradient coded packets. ; In this embodiment, the reduction ratio of the data volume when converting the original FP32 precision gradient to FP8 precision is: ; The reduction in data volume for converting gradients from FP16 precision to FP8 precision is as follows: ; Where bit represents the data storage width, FP32 indicates 32-bit floating-point precision, FP16 indicates 16-bit floating-point precision, and FP8 indicates 8-bit floating-point precision. The amount of data is reduced proportionally by compressing the bit width.

[0019] In this embodiment, the compression ratio is set to 1.2:1-1.5:1.

[0020] (S22) For important gradients, retain the top K proportion of gradient data and set the remaining gradients in the important gradients to 0 to achieve sparsity compression; calculate the residual of the gradients in the 0-part portion; compress the residual through quantization coding level to generate important gradient coding packets. and residual coding packets ; In this embodiment, the residual Represented as: ; ; In the formula, This represents the original gradient values ​​of the portion of the important gradient that are set to 0. The fitted values ​​are the retained Top-K gradient values; This represents the gradient values ​​of the top K proportions of the significant gradients; , The fitting coefficients are denoted as .

[0021] In this embodiment, the quantization step size is calculated based on the maximum and minimum values ​​of the residuals of the gradients of the zero-based portions, and each residual is mapped to the corresponding quantization coding level based on the quantization step size. And based on quantization coding level Compress the residuals; that is: In the formula, , These are the maximum residual and the minimum residual, respectively; This is the quantization step size; This is a rounding function.

[0022] S23) For redundant gradients, quantize the redundant gradients to 1-bit, reducing the data volume to 1 / 32 of the original FP32 precision; calculate the residuals of the quantized redundant gradients. Through a momentum replenishment mechanism, the residual is accumulated into the corresponding gradient of the next iteration to achieve accuracy compensation; this generates redundant gradient encoding packets. With residual cumulative value ; In this embodiment, redundant gradients are quantized to 1 bit, as follows: Calculate redundant gradients The symbol, if Then the quantized value ,like Then the quantized value ,Right now: .

[0023] In this embodiment, the residual The calculation expression is as follows: In the formula, Indicates the first The quantization residuals of redundant gradients; For the first The original FP32 precision values ​​of the redundant gradients; For the first The 1-bit quantized value of each redundant gradient; This is the average magnitude of the original values ​​of all redundant gradients in the current batch.

[0024] In this embodiment, the residual accumulation is expressed as: ; In the formula, For the first In the first iteration The cumulative residual of each redundant gradient; This is the momentum compensation coefficient; For the first In the first iteration The cumulative residual of each redundant gradient.

[0025] In this embodiment, the gradient compensation in the next iteration is expressed as: ; In the formula, For the first The 1st iteration after residual compensation One redundant gradient value; For the first In the first iteration The original generated values ​​of redundant gradients. S24) Encode the key gradient packet Important gradient encoding packets and residual coding packets Redundant gradient coding packets With residual cumulative value Integrate into a unified compressed gradient package Add packet header information and calculate the compression gradient packet. Verification code , via verification code Verify whether data is lost or corrupted during transmission; that is: In the formula, For the first Data from a gradient-encoded packet; In this embodiment, the total number of gradient-coded packets is... That is, the key gradient encoding packet Important gradient encoding packets Redundant gradient coding packets ; For the first Relevant data for each residual; In this embodiment, the number of residual data is... That is, residual coded packet and cumulative residuals ; This indicates a modulo operation to constrain the calculated checksum to the range of 0 to 255.

[0026] S3) Real-time acquisition of network status, dynamic selection of optimal communication topology and transmission path, and compression of gradient packets. Transmitted to other computing servers and global parameter servers; such as Figure 3 As shown, the details are as follows: S31) The SDN controller collects network status parameters between computing servers in real time, including bandwidth B, one-way latency D, packet loss rate H, and network congestion level C; and normalizes the network status parameters to construct a network status evaluation matrix. In this embodiment, outliers are removed from the collected raw network status parameters to avoid abnormal data affecting the normalization effect. The outlier judgment criteria are as follows: available bandwidth B≤0 or B>400Gbps, one-way delay D<0.1ms or D>10ms, packet loss rate H<0 or H>5%, and congestion level C<0 or C>1 are all judged as outliers. The outliers are then replaced with the average value of the corresponding parameter collected the previous 5 times. The min-max normalization algorithm is used to map all parameters after outlier removal to the [0,1] interval, thus eliminating the dimensional differences between different parameters.

[0027] A network state evaluation matrix is ​​constructed by using each compute server and the global parameter server in the distributed training cluster as rows and the normalized network state parameters as columns; that is: In the formula, Indicates normalization. This refers to the number of servers.

[0028] S32) Based on the network state evaluation matrix, a topology selection algorithm is used to dynamically select the optimal communication topology; In this embodiment, the network conditions are considered good, i.e., single-trip latency ≤ 1ms, packet loss rate ≤ 0.1%, and bandwidth utilization ≤ 60%; the number of computing servers is less than [number missing]. In the platform scenario, a ring communication topology is adopted; For networks experiencing partial congestion, i.e., some links have latency ≥5ms and packet loss rate ≥1%; the number of servers to calculate is... For scenarios with 100 to 500 units, a tree-structured communication topology is adopted. For other network scenarios, a butterfly communication topology is adopted.

[0029] S33) Match the corresponding communication primitives according to the selected optimal communication topology; In this embodiment, for the ring communication topology, the AllReduce communication primitive is matched to realize global gradient aggregation within the computing server cluster. After aggregation, the aggregated gradient is transmitted to the global parameter server by the designated computing server. For a tree-like communication topology, the ReduceScatter and AllGather communication primitives are matched. The computation servers realize gradient sharding and aggregation through ReduceScatter, and then synchronize the sharding and aggregation results to all computation servers through AllGather. Finally, the computation servers transmit the complete aggregated gradient to the global parameter server. For butterfly communication topology, Broadcast and Reduce communication primitives are matched to realize multi-link parallel transmission between computing servers and multi-path parallel transmission between computing servers and parameter servers.

[0030] S34) Based on the topological distance and network status of the servers participating in the communication, for each compressed gradient packet... Allocate the optimal transmission path to avoid congestion on a single link; In this embodiment, the optimal transmission path allocation aims to achieve optimal global communication efficiency, balanced link load, and minimum transmission latency. A path filtering and priority ranking mechanism is constructed based on the server topology distance, network state evaluation matrix, and compressed gradient packet characteristics. The topology distance is defined as the number of hops between two servers; the fewer the hops, the closer the topology distance and the lower the transmission latency. Specifically: S341), from the current communication topology's set of links In the process, links that meet the basic conditions are selected, and links that do not meet the requirements are removed; if the number of usable links after selection is greater than or equal to... If the result is not found, proceed to the next step of scoring and sorting; otherwise, adjust the filtering threshold until at least two usable links are selected. In this embodiment, the basic conditions are: Topological distance In this embodiment, The topology distance between compute servers under the same switch is 1, and the topology distance between a compute server and a global parameter server connected to the same switch is 1; the topology distance between compute servers across switches is 2, the topology distance between a compute server and a global parameter server across switches is 2, and the topology distance between servers across data centers is 3. Available bandwidth Broadband threshold In this embodiment, the broadband threshold Gbps; One-way delay Delay threshold In this embodiment, the delay threshold 3ms; Packet loss rate Packet loss rate threshold In this embodiment, the packet loss rate threshold .

[0031] S342) For the selected available links, calculate the comprehensive score for each link. ,Right now: ; In the formula, ~ In this embodiment, the corresponding weighting coefficients are used. ~ Set them to 0.4, 0.3, 0.2, and 0.1 respectively; , , , The first The topological distance, available bandwidth, one-way delay, and packet loss rate of each available link; S343) All filtered available links are ranked according to their overall score. Sort the paths in descending order to obtain a list of path priorities; simultaneously combine this with a compressed gradient package. The transmission priority allocation link; Compress gradient pack The highest priority transmission task is assigned to the overall score. The top 20% of available links; Compress gradient pack The second priority transmission task is assigned to the overall score. Available links are between 20% and 70%. Compress gradient pack The third priority transmission task is assigned to the overall score. 70%-100% of the links are available.

[0032] In this embodiment, the link status is synchronously collected every 50ms. When a link experiences the following conditions, a path adjustment is triggered: Link Comprehensive Score ; Link bandwidth utilization ≥70%; Transmission delay greater than Or the packet loss rate is greater than the packet loss rate threshold. ; The third-priority transmission task currently being transmitted on the corresponding link is migrated to the idle link with the highest score; the first-priority transmission task continues to transmit on the corresponding link until the transmission is completed, then the corresponding link is removed from the list of available paths, and the best link is selected again to supplement it.

[0033] In this embodiment, the average transmission delay of the compressed gradient packet C(G) can be controlled within 1.5ms, the transmission success rate can reach more than 99.9%, and a single link can be avoided from being occupied by high-priority packets.

[0034] S4) A double-buffering mechanism is used to coordinate the execution order of computation and communication tasks, so as to achieve parallel execution of gradient transfer and computation of the next training batch; specifically as follows: S41) Initialize two buffer queues, Buffer A and Buffer B, each buffer queue is used to store gradient data and computation task information; Among them, buffer queue A is used to store the gradient data of the current batch; buffer queue B is used to store the computation task data of the next batch. S42) After the GPU completes the gradient calculation for the current batch, it writes the gradient data into the buffer queue Buffer A and sends gradient processing instructions to the gradient sensing and compression module. S43) The gradient sensing and compression module reads gradient data from the buffer queue Buffer A, performs hierarchical compression and encoding, and generates compressed gradient packets. Simultaneously, the computing-communication pipeline scheduling module sends the next batch of computing instructions to the GPU, and the GPU reads the training data from Buffer B to start the next batch of forward and backward propagation computing. S44), compressed gradient package After generation, the communication module reads the compressed gradient packet from the buffer queue Buffer A. Based on the optimized communication topology and transmission path, gradient transmission is initiated through the RDMA network card; S45) After the GPU completes the gradient calculation for the next batch, it writes the gradient data into the buffer queue Buffer B. At the same time, the communication module completes the gradient transmission for the current batch, and the buffer queue is switched. S46), Repeat steps S43 to S45). S47) The compute-communication pipeline scheduling module monitors GPU utilization and communication latency in real time and dynamically adjusts pipeline depth; When the GPU utilization is less than 85%, the pipeline depth is increased. In this embodiment, the pipeline depth is increased from 2 training batches to 3 training batches. When the communication latency is greater than or equal to the computation time, the pipeline depth is reduced. In this embodiment, the pipeline depth is reduced from 3 training batches to 2 training batches.

[0035] In this embodiment, the pipeline depth refers to the number of training batches or tasks that can be processed simultaneously in the compute-communication pipeline.

[0036] S5) The global parameter server performs asynchronous gradient aggregation, restores the global gradient, and prepares for parameter updates; In this embodiment, when the global parameter server receives compressed gradient packets from more than 80% of the servers... Immediately afterwards, gradient decoding and gradient aggregation operations are initiated. After the remaining 20% ​​of the server's gradient transmission is complete, aggregation is then performed. Specifically: The global parameter server compresses gradient packets. Decode the key gradients, important gradients, and redundant gradients separately, and combine them with residual compensation data to recover the complete local gradients. The gradient aggregation operation described uses the AllReduce aggregation algorithm to aggregate the local gradients of each computing server into a global gradient, i.e.: In the formula, For global gradient; Number of servers; For the first The local decoding gradient of the computing server; After aggregation is complete, the global parameter server stores the global gradient to a distributed storage device and sends global gradient update instructions to each computing server.

[0037] S6) Each computing server updates the model parameters based on the received global gradient and dynamically adjusts the scheduling strategy for the next round based on real-time feedback information. In this embodiment, each computing server obtains the global gradient from the global parameter server. The Adam adaptive moment estimation algorithm is used to update the local model parameters, that is: ; ; In the formula, These are the updated model parameters for the current batch; The learning rate; To prevent constants with a denominator of 0; , These are momentum and second-order momentum, respectively. , These are the momentum coefficients; After the parameters are updated, each computing server sends a parameter update completion signal to prepare for the next batch of training.

[0038] In this embodiment, the feedback information includes model training loss, network status, resource load, and gradient compression effect.

[0039] In this embodiment, if the model training loss decreases slowly, the proportion of key gradients is increased and the compression ratio is reduced to retain more gradient information and ensure model convergence. If network congestion occurs, switch to a tree or butterfly topology to improve gradient compression rate, reduce communication data volume, and alleviate network congestion. If GPU utilization U < 85%: Increase pipeline depth to improve computational parallelism and fully utilize GPU resources; If GPU utilization U > 95%: Reduce pipeline depth to avoid GPU overload and prevent increased computational latency; If the communication time accounts for more than 40% of the total time: increase the gradient compression rate, optimize the communication topology, and switch to a higher bandwidth transmission path. If the accuracy error is >2%, reduce the compression ratio, increase the proportion of key gradients, strengthen residual compensation, and ensure the accuracy of model training.

[0040] S7) When any termination condition is met, stop the training iteration and output the final model parameters; The termination condition is as follows: The training iteration count has reached the preset value; The model training loss reaches the preset threshold, and the loss fluctuation over 10 consecutive iterations is ≤0.001; Training time has reached the preset limit; A hardware failure or network interruption was detected, preventing the continuation of training.

[0041] Example 2 This embodiment provides a server-to-server gradient communication and computational collaborative scheduling system for large model training, including: The computation server is used to load the current batch of training data and global gradients, execute the backpropagation process of model training, and generate local gradient tensors. The gradient sensing and compression module is used to evaluate the importance of gradients. Based on the gradient classification results, it executes a hierarchical dynamic compression strategy, using different compression and encoding methods for different types of gradients to generate compressed gradient packets and residual compensation data. The communication topology sensing and optimization module is used to collect network status in real time, dynamically select the optimal communication topology and transmission path, and compress gradient packets. Transmitted to other computing servers and global parameter server; The computation-communication pipeline scheduling module coordinates the execution order of computation and communication tasks through a double buffering mechanism to achieve parallel execution of gradient transmission and computation of the next training batch. The global parameter server is used to perform asynchronous gradient aggregation, recover global gradients and prepare parameter updates, and send the updated global gradients to the computation server.

[0042] Example 3 This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned inter-server gradient communication and computational collaborative scheduling method for training large models.

[0043] In this embodiment, the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. A processor, coupled to the memory, is used to execute computer programs stored in the memory.

[0044] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form.

[0045] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A server-to-server gradient communication and computational collaborative scheduling method for training large models, characterized in that, Includes the following steps: S1) The GPUs of each computing server execute the forward and backward propagation process of model training and generate local gradient tensors. The importance of gradients is evaluated through the gradient awareness and compression module. S2) The gradient sensing and compression module executes a hierarchical dynamic compression strategy based on the gradient classification results, and uses different compression and encoding methods for different types of gradients to generate compressed gradient packages and residual compensation data. S3) Real-time acquisition of network status, dynamic selection of optimal communication topology and transmission path, and compression of gradient packets. Transmitted to other computing servers and global parameter server; S4) The execution order of computation and communication tasks is coordinated through a double buffering mechanism to achieve parallel execution of gradient transfer and computation of the next training batch; S5) The global parameter server performs asynchronous gradient aggregation, restores the global gradient, and prepares for parameter updates; S6) Each computing server updates the model parameters based on the received global gradient and dynamically adjusts the scheduling strategy for the next round based on the real-time feedback information. S7) When any termination condition is met, stop the training iteration and output the final model parameters.

2. The inter-server gradient communication and computational collaborative scheduling method for large model training according to claim 1, characterized in that, The computation server's GPU performs the forward and backward propagation processes for model training and generates local gradient tensors, as follows: S11) Load the current batch of training data to the GPU, whereby the GPU performs forward propagation of model training and calculates the loss function between the model prediction and the actual value; S12) The GPU performs backpropagation for model training and calculates the local gradient tensor G of each layer of the model based on the loss function.

3. The inter-server gradient communication and computational collaborative scheduling method for large model training according to claim 2, characterized in that, In step S1), the importance of gradients is evaluated through the gradient-aware and compression module; specifically as follows: S13) Extract gradient magnitude from local gradient tensor G Historical update frequency Importance of the model layer ; S14), based on gradient magnitude Historical update frequency Importance of the model layer Calculate gradient importance score ; S15) Score all gradients by importance. Sort the gradients in descending order and construct a gradient importance ranking table. Based on a preset threshold, the gradients are divided into three categories: critical gradients, important gradients, and redundant gradients. Importance score The first 20% of gradients are considered critical gradients; Importance score Gradients between 20% and 80% are considered important gradients; Importance score The last 20% of the gradients are used as redundant gradients.

4. The inter-server gradient communication and computational collaborative scheduling method for large model training according to claim 3, characterized in that, In step S2), the gradient sensing and compression module executes a hierarchical dynamic compression strategy based on the gradient classification results, as follows: S21) For critical gradients, FP8 mixed-precision coding is used to convert the original FP32 and FP16 precision gradients to FP8 precision. During the coding process, lossless compression is used to compress the gradients to generate the critical gradient coded packets. ; S22) For important gradients, retain the top K proportion of gradient data and set the remaining gradients in the important gradients to 0; calculate the residuals of the gradients set to 0; compress the residuals using quantization coding levels to generate important gradient coding packets. and residual coding packets ; S23) For redundant gradients, quantize the redundant gradients to 1-bit, reducing the data volume to 1 / 32 of the original FP32 precision; calculate the residuals of the quantized redundant gradients. Through a momentum replenishment mechanism, the residual is accumulated into the corresponding gradient of the next iteration to achieve accuracy compensation; this generates redundant gradient encoding packets. With residual cumulative value ; S24) Encode the key gradient packet Important gradient encoding packets and residual coding packets Redundant gradient coding packets With residual cumulative value Integrate into a unified compressed gradient package Add packet header information and calculate the compression gradient packet. Verification code .

5. The inter-server gradient communication and computational collaborative scheduling method for large model training according to claim 4, characterized in that, In step S3), the network status is collected in real time, the optimal communication topology and transmission path are dynamically selected, and the compressed gradient packet is compressed. The data is then transferred to other computing servers and the global parameter server; specifically as follows: S31) The network status parameters between each computing server are collected in real time through the SDN controller, including bandwidth B, one-way delay D, packet loss rate H, and network congestion level C; and the network status parameters are normalized. A network status evaluation matrix is ​​constructed with each computing server and the global parameter server in the distributed training cluster as rows and the normalized network status parameters as columns. S32) Based on the network state evaluation matrix, a topology selection algorithm is used to dynamically select the optimal communication topology; S33) Match the corresponding communication primitives according to the selected optimal communication topology; S34) Based on the topological distance and network status of the servers participating in the communication, for each compressed gradient packet... Allocate the optimal transmission path.

6. The inter-server gradient communication and computational collaborative scheduling method for large model training according to claim 5, characterized in that, In step S34), the specific details are as follows: S341), from the current communication topology's set of links In the process, links that meet the basic conditions are selected, and links that do not meet the requirements are removed; if the number of usable links after selection is greater than or equal to... If the result is not found, proceed to the next step of scoring and sorting; otherwise, adjust the filtering threshold until at least two usable links are selected. S342) For the selected available links, calculate the comprehensive score for each link. ; S343) All filtered available links are ranked according to their overall score. Sort the paths in descending order to obtain a list of path priorities; simultaneously combine this with a compressed gradient package. The transmission priority allocation link; Compress gradient pack The highest priority transmission task is assigned to the overall score. The top 20% of available links; Compress gradient pack The second priority transmission task is assigned to the overall score. Available links are between 20% and 70%. Compress gradient pack The third priority transmission task is assigned to the overall score. 70%-100% of the links are available.

7. The inter-server gradient communication and computational collaborative scheduling method for large model training according to claim 6, characterized in that, In step S4), a double-buffering mechanism is used to coordinate the execution order of computation and communication tasks, so as to achieve parallel execution of gradient transfer and computation of the next training batch; specifically as follows: S41) Initialize two buffer queues, Buffer A and Buffer B, each buffer queue is used to store gradient data and computation task information; Among them, buffer queue A is used to store the gradient data of the current batch; buffer queue B is used to store the computation task data of the next batch. S42) After the GPU completes the gradient calculation for the current batch, it writes the gradient data into the buffer queue Buffer A and sends gradient processing instructions to the gradient sensing and compression module. S43) The gradient sensing and compression module reads gradient data from the buffer queue Buffer A, performs hierarchical compression and encoding, and generates compressed gradient packets. Simultaneously, the computing-communication pipeline scheduling module sends the next batch of computing instructions to the GPU, and the GPU reads the training data from Buffer B to start the next batch of forward and backward propagation computing. S44), compressed gradient package After generation, the communication module reads the compressed gradient packet from the buffer queue Buffer A. Based on the optimized communication topology and transmission path, gradient transmission is initiated through the RDMA network card; S45) After the GPU completes the gradient calculation for the next batch, it writes the gradient data into the buffer queue Buffer B. At the same time, the communication module completes the gradient transmission for the current batch, and the buffer queue is switched. S46), Repeat steps S43 to S45). S47) The compute-communication pipeline scheduling module monitors GPU utilization and communication latency in real time and dynamically adjusts pipeline depth; Increase pipeline depth when GPU utilization is less than 85%. Reduce pipeline depth when communication latency is greater than or equal to computation time.

8. The inter-server gradient communication and computational collaborative scheduling method for large model training according to claim 7, characterized in that, In step S5), when the global parameter server receives compressed gradient packets from more than 80% of the servers... Immediately afterwards, gradient decoding and gradient aggregation operations are initiated. After the remaining 20% ​​of the server's gradient transmission is complete, aggregation is then performed. Specifically: The global parameter server compresses gradient packets. Decode the key gradients, important gradients, and redundant gradients separately, and combine them with residual compensation data to recover the complete local gradients. The gradient aggregation operation described above uses the AllReduce aggregation algorithm to aggregate the local gradients of each computing server into a global gradient. After aggregation is complete, the global parameter server stores the global gradient to a distributed storage device and sends global gradient update instructions to each computing server.

9. A server-to-server gradient communication and computational collaborative scheduling system for large model training, characterized in that: include: The computation server is used to load the current batch of training data and global gradients, execute the backpropagation process of model training, and generate local gradient tensors. The gradient sensing and compression module is used to evaluate the importance of gradients. Based on the gradient classification results, it executes a hierarchical dynamic compression strategy, using different compression and encoding methods for different types of gradients to generate compressed gradient packets and residual compensation data. The communication topology sensing and optimization module is used to collect network status in real time, dynamically select the optimal communication topology and transmission path, and compress gradient packets. Transmitted to other computing servers and global parameter server; The computation-communication pipeline scheduling module coordinates the execution order of computation and communication tasks through a double buffering mechanism to achieve parallel execution of gradient transmission and computation of the next training batch. The global parameter server is used to perform asynchronous gradient aggregation, recover global gradients and prepare parameter updates, and send the updated global gradients to the computation server.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the inter-server gradient communication and computational collaborative scheduling method for large model training as described in any one of claims 1-8.

Images