A data communication method and communication system

Abstract

This invention provides a data communication method and system, relating to the field of artificial intelligence. In this method, computing nodes first compress the original gradient matrix generated during model training to obtain compressed gradient data packets. Then, based on the topology of the cluster network and the real-time link utilization, a transmission path is determined, and the compressed gradient data packets are sent to an aggregation node. The aggregation node then decompresses and aggregates the compressed gradient data packets to obtain an initial global gradient matrix. This allows the compressed gradient data packets to be transmitted to the aggregation node via a relatively efficient path, ensuring transmission efficiency. Subsequently, the aggregation node can filter gradient elements with significant compression loss from the compressed gradient data packets sent by the computing nodes and request the original values ​​of these gradient elements from the computing nodes as needed. These original values ​​are used to correct the initial global gradient matrix, thus improving communication efficiency between nodes while ensuring aggregation effectiveness.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a data communication method and communication system. Background Technology

[0002] In distributed model training, each computing node needs to upload its local raw gradient matrix to the aggregation node so that the aggregation node can generate the global gradient matrix. However, uploading the entire raw gradient matrix can cause link congestion, leading to a decrease in communication efficiency. Summary of the Invention

[0003] This invention provides a data communication method and system that can solve the technical problem of low communication efficiency between computing nodes and aggregation nodes.

[0004] This invention provides a data communication method applied to a computing node, comprising: The original gradient matrix generated during model training is compressed to obtain a compressed gradient data package. The transmission path is determined based on the topology of the cluster network and the real-time link utilization. The compressed gradient data packets are then sent to the aggregation node according to the transmission path, so that the aggregation node can decompress and aggregate the compressed gradient data packets to obtain the initial global gradient matrix. Receive incremental data requests sent by the aggregation node; the incremental data request contains the element coordinates obtained by the aggregation node based on the residual between the decompressed gradient matrix and the predicted gradient matrix corresponding to the compressed gradient data packet, and the predicted gradient matrix is ​​predicted by the aggregation node based on the global gradient matrix of the previous round; The gradient elements corresponding to the coordinates of the elements in the original gradient matrix are sent to the aggregation node so that the aggregation node can use the gradient elements to correct the initial global gradient matrix and obtain the global gradient matrix.

[0005] The present invention also provides a data communication method applied to an aggregation node, comprising: The system receives compressed gradient data packets sent by the compute nodes and decompresses and aggregates the compressed gradient data packets to obtain the initial global gradient matrix. The compressed gradient data packets are obtained by compressing the original gradient matrix generated by the compute nodes during model training and are sent to the aggregation node through a transmission path determined according to the topology of the cluster network and the real-time link utilization. The residual between the decompressed gradient matrix and the predicted gradient matrix corresponding to the compressed gradient data packet is determined, and the element coordinates in the decompressed gradient matrix are selected based on the residual; the predicted gradient matrix is ​​obtained by the aggregation node based on the global gradient matrix of the previous round. Send an incremental data request containing element coordinates to the compute node; The system receives gradient elements sent by the computing node and uses these gradient elements to correct the initial global gradient matrix, thus obtaining the global gradient matrix. The gradient elements are the data corresponding to the element coordinates in the original gradient matrix.

[0006] The present invention also provides a communication system, comprising: Compute nodes are used to execute the data communication methods described above for use on compute nodes. The aggregation node is used to execute the data communication methods described above for the aggregation node.

[0007] This invention provides a data communication method applied to a computing node, comprising: compressing the original gradient matrix generated during model training to obtain a compressed gradient data packet; determining a transmission path based on the topology of the cluster network and the real-time link utilization, and sending the compressed gradient data packet to an aggregation node according to the transmission path, so that the aggregation node decompresses and aggregates the compressed gradient data packet to obtain an initial global gradient matrix; receiving an incremental data request sent by the aggregation node; the incremental data request includes element coordinates obtained by the aggregation node based on the residual between the decompressed gradient matrix corresponding to the compressed gradient data packet and the predicted gradient matrix, the predicted gradient matrix being predicted by the aggregation node based on the previous round of global gradient matrix; and sending the gradient elements corresponding to the element coordinates in the original gradient matrix to the aggregation node, so that the aggregation node uses the gradient elements to correct the initial global gradient matrix to obtain a global gradient matrix.

[0008] The beneficial effects of this invention are as follows: First, the computing node can compress the original gradient matrix generated during model training to obtain compressed gradient data packets, thereby reducing data transmission volume. Then, the computing node can determine the transmission path based on the cluster network topology and real-time link utilization, and send the compressed gradient data packets to the aggregation node according to the transmission path. The aggregation node then decompresses and aggregates the compressed gradient data packets to obtain the initial global gradient matrix. In this way, the compressed gradient data packets can be transmitted to the aggregation node through a more efficient path, ensuring transmission efficiency. Subsequently, when the computing node receives an incremental data request from the aggregation node, it can send the gradient elements corresponding to the element coordinates in the original gradient matrix to the aggregation node. This allows the aggregation node to use the gradient elements to correct the initial global gradient matrix, obtaining the global gradient matrix. The incremental data request includes element coordinates obtained by the aggregation node based on the residual between the decompressed gradient matrix corresponding to the compressed gradient data packets and the predicted gradient matrix. The predicted gradient matrix is ​​predicted by the aggregation node based on the global gradient matrix from the previous round. As can be seen, the aggregation node can filter gradient elements with large compression loss from the compressed gradient data packets sent by the computing node, and request the original values ​​of these gradient elements from the computing node as needed. These original values ​​can be used to correct the initial global gradient matrix. On the one hand, this can make up for the accuracy loss caused by compression, and on the other hand, it can reduce the amount of data transmission compared to transmitting the full original gradient matrix. Therefore, it can improve the communication efficiency between the computing node and the aggregation node while ensuring the aggregation effect.

[0009] The present invention also provides a communication system that has the above-mentioned beneficial effects. Attached Figure Description

[0010] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart of a data communication method provided in an embodiment of the present invention; Figure 2 A flowchart of another data communication method provided in an embodiment of the present invention; Figure 3 A complete flowchart of a data communication method provided in an embodiment of the present invention; Figure 4 A flowchart of tensor fragmentation and sparsification processing is provided for an embodiment of the present invention; Figure 5 A flowchart of dynamic quantization and encoding processing of a sparse gradient matrix is ​​provided in an embodiment of the present invention; Figure 6 A flowchart of a many-to-one reduction communication operation is provided as an embodiment of the present invention; Figure 7 A flowchart for performing index-aligned decompression and aggregation operations is provided as an embodiment of the present invention; Figure 8 A schematic diagram of a communication optimization system provided in an embodiment of the present invention; Figure 9 This is a structural block diagram of a data communication device provided in an embodiment of the present invention; Figure 10 This is a structural block diagram of another data communication device provided in an embodiment of the present invention; Figure 11 This is a structural block diagram of a communication system provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0013] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] In the field of artificial intelligence, the training of machine learning models (such as language processing models and image processing models) generally adopts a distributed training approach. This involves setting up a cluster network containing compute nodes and aggregation nodes. There can be multiple compute nodes and only one aggregation node. The aggregation node can be a dedicated device for parameter aggregation (such as a parameter server) or a special compute node. Compute nodes and aggregation nodes communicate via a network, which can be constructed using various network devices (such as switches and routers). In each round of training, each compute node uses different training data to perform forward propagation on its local model and generates gradient data based on the model's loss. This gradient data is then sent to the aggregation node. The aggregation node aggregates the gradient data sent by each compute node to obtain global gradient data and broadcasts this global gradient data to all compute nodes. Finally, the compute nodes use this global gradient data to perform backpropagation on their local models to update the model parameters, thus completing one round of training.

[0016] In related technologies, distributed training processes suffer from low communication efficiency, specifically: 1. Computational nodes typically send the original gradient matrix to the aggregation node. When the model parameters are large, the original gradient matrix contains a large amount of data, leading to rapid exhaustion of the cluster network bandwidth and hindering high communication efficiency. 2. The lack of link scheduling in data transmission between computational nodes and aggregation nodes makes it easy for data to be transmitted through inefficient links, further reducing the communication efficiency between nodes.

[0017] In view of this, to address the technical problem of improving the communication efficiency between computing nodes and aggregating nodes, this invention provides a data communication method. The computing node can compress the original gradient matrix to obtain compressed gradient data packets and send these packets to the aggregating node using a transmission path with better communication performance. This allows the aggregating node to generate a global gradient matrix using the compressed gradient data packets. Furthermore, the aggregating node can filter out gradient elements with significant compression loss from the compressed gradient data packets as needed and request the original values ​​of these gradient elements from the computing node to correct the global gradient matrix. Thus, this invention can reduce the amount of data transmitted between computing nodes and aggregating nodes while maintaining the accuracy of the global gradient matrix aggregation. Simultaneously, it can control the transmission path used between computing nodes and aggregating nodes to ensure better data transmission performance, thereby improving the data transmission efficiency between nodes.

[0018] The data communication method provided in this embodiment will be described below, starting with its implementation on the computing node side. For easier understanding, please refer to [link to documentation]. Figure 1 , Figure 1A flowchart of a data communication method provided in an embodiment of the present invention, the method being applied to a computing node, may include: S11. Compress the original gradient matrix generated during model training to obtain a compressed gradient data package.

[0019] During the backpropagation phase of distributed model training, each computing node generates its local original gradient matrix. The original gradient matrix is ​​a data matrix containing model parameter update information, where each value represents the gradient element of the corresponding model parameter. To reduce communication overhead, the computing nodes in this embodiment can compress the original gradient matrix to generate compressed gradient data packets. Compressed gradient data packets are data packets whose size is significantly reduced after compression processes such as sparsification, quantization, and encoding.

[0020] It is evident that compression can significantly reduce data volume, thereby substantially reducing network transmission load and improving gradient synchronization efficiency.

[0021] It should be noted that this embodiment does not limit the specific method of compressing the original gradient matrix. For example, the original gradient matrix can be sparsified, retaining only the effective gradient elements that show significant changes, resulting in a sparse gradient matrix, which can then be compressed. In this case, the number of effective data in the sparse gradient matrix can be effectively reduced. Furthermore, the floating-point values ​​in the original gradient matrix can be quantized into integer values, reducing the data size required to store a single gradient element. Additionally, the original gradient matrix can be encoded into a smaller form to reduce the data size.

[0022] The following describes a specific compression method. In one implementation, compressing the original gradient matrix to obtain a compressed gradient data packet may include: S111. Determine and retain the effective gradient elements in the original gradient matrix to obtain a sparse gradient matrix, and divide the sparse gradient matrix into sub-tensor blocks according to the preset number of partitions.

[0023] When compressing the original gradient matrix, the computing node first performs sparsification, which involves identifying and retaining the valid gradient elements in the original gradient matrix while setting the invalid gradient elements to zero, resulting in a sparse gradient matrix. Valid gradient elements are non-zero gradient elements that have a significant impact on model updates.

[0024] Subsequently, the computing node can divide the sparse gradient matrix into sub-tensor blocks according to a preset number of partitions. A sub-tensor block refers to a local data block formed after the sparse gradient matrix is ​​partitioned. In this way, the computing node can perform independent quantization processing on each sub-tensor block, improving the compression effect.

[0025] It should be noted that this embodiment does not limit the method for determining effective gradient elements. For example, a threshold can be preset, and the absolute value of each effective gradient element can be determined to be greater than the threshold. If it is greater, it is determined to be a valid gradient element; if not, it is determined to be an ineffective gradient element. Considering that model training is a dynamic process, for example, the gradient distribution is relatively dispersed in the early stage of training, while the gradient often concentrates on a few neurons during the convergence period. If a fixed threshold is used, it is easy to lose important gradient information, or the compression ratio is insufficient when the gradient sparsity is not high, making it impossible to achieve the optimal balance between traffic and model accuracy. Therefore, this embodiment can also dynamically determine the threshold according to the actual situation of each sparse gradient matrix, and select effective gradient elements based on the dynamically determined threshold.

[0026] The process of selecting effective gradient elements is described below. In one implementation, determining and retaining effective gradient elements in the original gradient matrix to obtain a sparse gradient matrix may include: S1111 Determine the norm of the original gradient matrix and the mean absolute value of the gradient elements in the original gradient matrix, and determine the screening threshold based on the norm and the mean absolute value.

[0027] When determining valid gradient elements, the computation node can determine the norm of the original gradient matrix and the mean absolute value of the gradient elements in the original gradient matrix. The norm characterizes the overall numerical magnitude of the gradient matrix. A larger norm indicates that the absolute values ​​of the elements in the gradient matrix are generally high, meaning the overall magnitude of the gradient update is large. A smaller norm indicates that the gradient elements tend to be flat, meaning the overall magnitude of the gradient update is small. Subsequently, the computation node can determine a selection threshold based on the norm and the mean absolute value to identify valid gradient elements using a dynamically determined threshold.

[0028] The original gradient tensor of the i-th node L2 norm and the mean of the absolute values ​​of the gradient elements : ; ; Where M is the gradient tensor The total number of elements in the middle, for The value of the j-th gradient element; Determine the screening threshold based on L2 norm and mean absolute value. : ; in, and The preset weighting coefficients satisfy... .

[0029] S1112. Gradient elements in the original gradient matrix whose absolute values ​​are greater than or equal to the screening threshold are retained as valid gradient elements, and gradient elements in the original gradient matrix whose absolute values ​​are less than the screening threshold are set to zero, thus obtaining a sparse gradient matrix.

[0030] In this embodiment, the computing node can retain gradient elements in the original gradient matrix whose absolute values ​​are greater than or equal to a filtering threshold as valid gradient elements, and set gradient elements whose absolute values ​​are less than the filtering threshold to zero, thus obtaining a sparse gradient matrix. This can be represented as: Traversal Each gradient element in ,Will With dynamic threshold Compare; (1) If Then retain the gradient element and compare its value with the value in the gradient. 3D coordinate position Record to the intermediate buffer; (2) If If so, the gradient element is set to zero, and the corresponding mask bits are generated. .

[0031] It is evident that by using dynamic thresholding based on statistical features, the sparsity intensity can be adaptively adjusted at different training stages, avoiding the loss of important information or insufficient compression rate caused by fixed thresholds, thus achieving a better balance between communication volume and model accuracy.

[0032] S112. Set header information for the sub-tensor block and quantize and compress the sub-tensor block to obtain the compressed payload; the header information includes the sub-tensor block identifier, the computation node identifier, and the effective element count.

[0033] In this embodiment, to facilitate the individual parsing of each sub-tensor block, the computing node can set header information for each sub-tensor block. The header information includes the sub-tensor block identifier, the computing node identifier, and the count of valid elements, so that the aggregation node can identify the data source and block structure.

[0034] Subsequently, the computing nodes quantize and compress each sub-tensor block to obtain the compressed payload. The compressed payload refers to the binary data volume of the sub-tensor block after quantization and encoding.

[0035] It should be noted that this embodiment does not limit the specific method of quantization compression. For example, a quantization scaling factor can be preset and used to quantize and compress the sub-tensor blocks. Considering that model training is a dynamic process, a fixed quantization scaling factor is difficult to adapt well to the dynamic changes of the model. Therefore, this embodiment can also dynamically determine the quantization scaling factor according to the actual situation of each sub-tensor block, and perform quantization compression according to the dynamically determined quantization scaling factor, thereby better adapting to the numerical distribution of gradient elements at each local position in the sparse gradient matrix.

[0036] The following describes a quantization compression method. In one embodiment, quantizing and compressing a sub-tensor block to obtain a compressed load may include: S1121. Determine the maximum and minimum values ​​of the effective gradient elements in the sub-tensor block, and determine the quantization scaling factor based on the maximum and minimum values.

[0037] When quantizing and compressing a sub-tensor block, the computation node first determines the maximum and minimum values ​​of the effective gradient elements in the sub-tensor block, and then determines the quantization scaling factor based on the difference between the maximum and minimum values. The quantization scaling factor is a proportional coefficient used to map floating-point values ​​to an integer range, controlling the quantization precision.

[0038] For each sub-tensor block, calculate the maximum value of its internal effective gradient elements. and minimum value Then, the quantization scaling factor is calculated. : ; Where b is the number of quantization bits, which can be 4 or 8.

[0039] S1122. Linearly quantize the effective gradient elements according to the quantization scaling factor and the minimum value to obtain integer values.

[0040] The computation nodes linearly quantize the effective gradient elements according to the quantization scaling factor and the minimum value, obtaining integer values. Each effective gradient element can be quantized using the following formula. Perform linear quantization to obtain an integer value. : ; in, This is the rounding function. These are the original floating-point gradient values. This is the quantized integer value.

[0041] S1123. Extract the exponent and mantissa data of the sub-tensor block.

[0042] To ensure data accuracy during subsequent decompression and maintain compression efficiency, the computing node can extract the exponent data E and mantissa data M from the sub-tensor block and store these two types of data separately. The exponent data and mantissa data refer to the exponent and mantissa portions separated according to the floating-point storage format; separate storage further improves compression efficiency.

[0043] S1124. Form an integer sequence from the integer values ​​and encode the integer sequence to obtain the encoded value.

[0044] The computing node can sequentially form an integer value sequence from the given integer values ​​and then entropy encode this sequence to obtain the encoded value. This encoded value can be a variable-length binary codeword generated through Huffman coding or arithmetic coding, allowing for shorter code lengths to be assigned to frequently occurring values, further reducing the data volume.

[0045] S1125. Set the exponent data, mantissa data, and encoded value as the compressed payload.

[0046] The compute node sets the exponent data, mantissa data, and encoded value as a compressed payload to provide the aggregation node with complete dequantized and decompressed data.

[0047] S113. Set the index mapping table corresponding to the sparse gradient matrix; the index mapping table contains the offset of the effective gradient element in the original gradient matrix and the storage address of the effective gradient element after compression.

[0048] In this embodiment, to facilitate data reading and aggregation by the aggregation node, the computing node can also set up an index mapping table corresponding to the sparse gradient matrix. , where i represents the i-th sparse gradient matrix. The index mapping table is a table that records the offsets of effective gradient elements in the original gradient matrix and the storage addresses of effective gradient elements in the compressed payload, used to accurately locate and restore each gradient element during decompression.

[0049] S114. Encapsulate the header information, quantization compression information of the sub-tensor block, compression payload, and index mapping table into a compressed gradient data packet.

[0050] In this embodiment, the computing node encapsulates the header information, quantization compression information of the sub-tensor block, compressed payload, and index mapping table into a compressed gradient data packet. The quantization compression information includes parameters required for dequantization, such as the quantization scaling factor and the minimum effective gradient element value. By encapsulating the index information along with the compressed data, the aggregation node can quickly reconstruct the original position of each gradient element based on the index mapping table, avoiding element-by-element traversal and improving decompression and aggregation efficiency.

[0051] S12. Determine the transmission path based on the topology of the cluster network and the real-time link utilization, and send the compressed gradient data packets to the aggregation node according to the transmission path so that the aggregation node can decompress and aggregate the compressed gradient data packets to obtain the initial global gradient matrix.

[0052] In this embodiment, to ensure high transmission efficiency, in addition to compressing the original gradient matrix, the computing node can also determine a transmission path with better communication performance with the aggregation node, so as to send compressed gradient data packets to the aggregation node through the transmission path.

[0053] To this end, compute nodes can obtain the topology information of the cluster network and the real-time bandwidth utilization of each link. The topology information includes the overall topology of the cluster network, from which multiple transmission paths between compute nodes and aggregation nodes can be determined. A single transmission path can consist of one or more physical links, which refer to the physical links between any two adjacent devices in the cluster network (such as between compute nodes, between a compute node and a network device (such as a switch), or between an aggregation node and a network device). Real-time bandwidth utilization refers to the ratio of the actual traffic on the link to the physical bandwidth at the current moment. Topology information can be preset in each compute node and aggregation node. Real-time link utilization can be obtained by querying the cluster management device.

[0054] As can be seen, by combining network topology and real-time link status to select transmission paths, computing nodes can avoid congested links as much as possible, reduce queuing delays and transmission jitter, and ensure that compressed ladder data packets are transmitted to the aggregation node along the optimal path.

[0055] The following describes one method for determining the transmission path. In one implementation, determining the transmission path based on the topology of the cluster network and the real-time link utilization may include: S121. Obtain the adjacency matrix of the cluster network and the real-time bandwidth utilization of each link.

[0056] The compute nodes obtain the adjacency matrix of the cluster network. An adjacency matrix is ​​a data structure that describes the connection relationships between nodes in a network in the form of a two-dimensional array, where the element values ​​represent the existence and basic attributes of the corresponding link. The compute nodes can simultaneously obtain the real-time bandwidth utilization of each link.

[0057] S122. Determine the candidate paths between the computing node and the aggregation node based on the adjacency matrix, and determine the path weights using the physical bandwidth and real-time bandwidth utilization of each link in the candidate paths.

[0058] Compute nodes can determine multiple candidate paths between themselves and aggregate nodes based on the adjacency matrix. A candidate path is a set of paths consisting of several physical links that may traverse from a compute node to an aggregate node. For each candidate path, a path weight is calculated using the physical bandwidth and real-time bandwidth utilization of each physical link in that path. The path weight is a weight value determined by comprehensively considering the link bandwidth capacity and the current level of congestion, and its value is related to the link's congestion status. For example, the higher the real-time bandwidth utilization or the smaller the physical bandwidth, the larger the path weight, indicating that the path is less suitable for transmission.

[0059] In one scenario, the path weight is calculated for computation node i and aggregation node j. : ; Where e is the path Physical links between them This represents the physical bandwidth of link e. This represents the current real-time bandwidth utilization of the link, with a value ranging from 0 to 1.

[0060] S123. Determine the transmission path from the candidate paths based on the path weight.

[0061] When a larger path weight indicates a less suitable path for transmission, the computing node can determine the candidate path with the smallest path weight as the transmission path. Specifically, the computing node can select the path weight based on Dijkstra's algorithm or the minimum spanning tree algorithm. The shortest communication path is used as the data transmission channel. In this way, by dynamically selecting low-weight paths, high-volume gradient data can avoid congested links, reducing queuing delays and transmission jitter, and improving the real-time performance and stability of gradient synchronization.

[0062] Finally, the transmission path can be set in the compressed gradient data packet, for example, in the header of the compressed gradient data packet, so that each device in the cluster network can transmit the compressed gradient data packet according to the specified transmission path.

[0063] S13. Receive incremental data request sent by the aggregation node; the incremental data request contains the element coordinates obtained by the aggregation node based on the residual between the decompressed gradient matrix and the predicted gradient matrix corresponding to the compressed gradient data packet, and the predicted gradient matrix is ​​predicted by the aggregation node based on the global gradient matrix of the previous round.

[0064] In this embodiment, since compressing the original gradient matrix results in data loss, excessive loss can cause the model to have difficulty converging. Therefore, during the generation of the global gradient matrix, the aggregation node can determine whether there is excessive data loss in the compressed gradient data packet. If so, it can identify the gradient elements with significant loss in the compressed gradient data packet and request the computing node to send the original values ​​of these gradient elements to compensate for the global gradient matrix.

[0065] To identify gradient elements with significant losses, the aggregation node compares the decompressed gradient matrix with the predicted gradient matrix during the generation of the initial global gradient matrix. The predicted gradient matrix refers to the gradient prediction value for the current iteration determined by the aggregation node based on the global gradient matrix obtained in the previous iteration. The decompressed gradient matrix is ​​the gradient matrix restored after decoding and dequantization. The residual is the set of differences between corresponding elements in the decompressed and predicted gradient matrices, used to quantize the information bias caused by compression. By calculating the residual, the aggregation node can assess the degree of information loss caused by this round of compression. If the residual indicates that the accuracy loss exceeds expectations, the aggregation node can send an incremental data request to the compute node. This incremental data request contains the coordinates of elements determined by the residual filtering; these coordinates are the location identifiers of the gradient elements in the original gradient matrix that need to be supplemented and transmitted. The compute node can then feed back the original value of the corresponding gradient element to the aggregation node based on these coordinates.

[0066] S14. Send the gradient elements corresponding to the element coordinates in the original gradient matrix to the aggregation node so that the aggregation node can use the gradient elements to correct the initial global gradient matrix and obtain the global gradient matrix.

[0067] In this embodiment, after receiving an incremental data request, the compute node extracts the gradient elements corresponding to the element coordinates from the locally stored original gradient matrix. The compute node can then send these uncompressed, high-precision gradient elements to the aggregator node. The aggregator node uses the received gradient elements to replace or correct the corresponding values ​​in the initial global gradient matrix to obtain the final global gradient matrix. Therefore, by transmitting only a small number of gradient elements with large residuals instead of the entire original gradient matrix, this embodiment can compensate for the loss of compression accuracy while keeping the additional communication overhead at a low level, thereby improving communication efficiency while ensuring aggregation accuracy.

[0068] It is worth noting that gradient elements can also be transmitted through a better transmission path between the computation node and the aggregation node, which is determined by the computation node, to ensure transmission effectiveness.

[0069] Based on the above embodiments, the compute nodes can first compress the original gradient matrix generated during model training to obtain compressed gradient data packets, thereby reducing data transmission volume. Subsequently, the compute nodes can determine the transmission path based on the cluster network topology and real-time link utilization, and send the compressed gradient data packets to the aggregation node according to the transmission path. The aggregation node then decompresses and aggregates the compressed gradient data packets to obtain the initial global gradient matrix. In this way, the compressed gradient data packets can be transmitted to the aggregation node through a more efficient path, ensuring transmission efficiency. Subsequently, when receiving incremental data requests from the aggregation node, the compute nodes can send the gradient elements corresponding to the element coordinates in the original gradient matrix to the aggregation node. This allows the aggregation node to use the gradient elements to correct the initial global gradient matrix, obtaining the global gradient matrix. The incremental data request includes element coordinates obtained by the aggregation node based on the residual between the decompressed gradient matrix corresponding to the compressed gradient data packets and the predicted gradient matrix. The predicted gradient matrix is ​​predicted by the aggregation node based on the global gradient matrix from the previous round. As can be seen, the aggregation node can filter gradient elements with large compression loss from the compressed gradient data packets sent by the computing node, and request the original values ​​of these gradient elements from the computing node as needed. These original values ​​can be used to correct the initial global gradient matrix. On the one hand, this can make up for the accuracy loss caused by compression, and on the other hand, it can reduce the amount of data transmission compared to transmitting the full original gradient matrix. Therefore, it can improve the communication efficiency between the computing node and the aggregation node while ensuring the aggregation effect.

[0070] Based on the above embodiments, even if the computing node has optimized the transmission path, network devices (such as switches) along that path may still experience congestion, affecting transmission performance. Therefore, the computing node can also monitor the congestion status of network devices and, upon determining that a network device is congested, proactively segment the compressed gradient data packets into smaller micro-batch packets for easier processing and transmission by the network devices. The following description uses a switch as an example. In one embodiment, before sending the compressed gradient data packets to the aggregation node, the following may also be included: S151. Obtain the real-time output queue length of the switch device in the transmission path.

[0071] Before sending the compressed gradient data packets, the compute node obtains the real-time output queue length of each switch device in the transmission path. The real-time output queue length refers to the number of data packets waiting to be forwarded at a switch port, directly reflecting the congestion level of that device.

[0072] S152. If the real-time output queue length is greater than the preset queue length threshold, the compressed gradient data packet will be split into micro-batch packets with a size less than or equal to the preset transmission unit threshold. If the real-time output queue length exceeds a preset queue length threshold, the compute node can determine that the path is at risk of congestion. It can then split the compressed gradient data packet into multiple micro-batch packets, each smaller than or equal to a preset transmission unit threshold. The preset queue length threshold is a pre-defined threshold used to determine if a switching device is congested. The preset transmission unit threshold is the preset maximum transmission unit (MTU), which refers to the maximum number of bytes allowed in a single data packet transmitted on a network link. By splitting large packets into micro-batch packets, head-of-line congestion caused by large packet queuing in congested links can be avoided, reducing tail-of-line latency.

[0073] Accordingly, sending the compressed gradient data packet to the aggregation node may include: S141. Set a sequence number for the micro-batch packet and send the micro-batch packet to reassemble it into a compressed gradient data packet based on the sequence number at the aggregation node.

[0074] The compute nodes assign sequence numbers to each micro-batch packet and send them sequentially according to their timestamps. The sequence number is a number used to identify the order of data packets. The aggregation node can reassemble the micro-batch packets into complete compressed gradient data packets based on the sequence numbers. This sequence number reassembly mechanism ensures data integrity while splitting the transmission, preventing aggregation errors caused by out-of-order arrival.

[0075] Specifically, the compute node can monitor the output queue length of the switching node. If the conditions are met If the condition is as described above, it is considered a congested state. This is the preset queue length threshold.

[0076] If the system is determined to be in a congested state, then large-sized compressed gradient data packets will be sent. Split into micro-batch packages The size of the micro-batch package meets the requirements. < ,in This is the preset maximum transmission unit threshold.

[0077] Based on timestamp The micro-batch packets (i.e., sequence numbers) are sent sequentially and reassembled into the original compressed gradient data packets at the aggregation node based on the sequence numbers.

[0078] Based on the above embodiments, a retransmission mechanism can be set between the computing node and the aggregation node to prevent data loss. In one embodiment, before sending the compressed gradient data packet to the aggregation node, the following steps are also included: S161. Write the compressed gradient data packets into the local retransmission buffer and assign a sequence number to each compressed gradient data packet.

[0079] Before sending, the compute node writes compressed gradient data packets into the local retransmission buffer and assigns a unique sequence number to each compressed gradient data packet. The local retransmission buffer is a storage area used to temporarily store copies of sent data so that they can be quickly retrieved and retransmitted in case of data loss.

[0080] Accordingly, after sending the compressed gradient data packet to the aggregation node, the following is also included: S171. Receive a retransmission request sent by the aggregation node. The retransmission request contains a list of missing sequence numbers. After the compute node completes its transmission, it receives a retransmission request from the aggregator node. A retransmission request is a supplementary transmission instruction initiated by the aggregator node when it detects missing data packets. It contains a list of missing sequence numbers, i.e., the set of data packet numbers that were not successfully received.

[0081] S172. Extract the corresponding compressed gradient data packet from the local retransmission buffer according to the sequence number list and retransmit it.

[0082] The compute nodes retrieve the corresponding compressed gradient data packets from their local retransmission buffer based on the sequence number list and retransmit them, with the number of retransmissions not exceeding the preset maximum. Through this selective retransmission mechanism, only the lost specific data packets need to be retransmitted, avoiding the bandwidth waste caused by full retransmissions and improving transmission reliability.

[0083] Based on the above embodiments, considering the different levels of importance of different network layers, gradient data of important network layers should be transmitted with priority. To this end, the computing node can also add an importance marker of the network layer to the compressed gradient data packet, so that the network device can schedule the transmission of the compressed gradient data packet according to the marker. In one embodiment, before sending the compressed gradient data packet to the aggregation node, the method further includes: S181. Extract the neural network layer corresponding to the compressed gradient data packet and determine the preset sensitivity weights corresponding to the neural network layer.

[0084] The computing node extracts the neural network layer identifier corresponding to the compressed gradient data packet and determines the preset sensitivity weights for that neural network layer. The preset sensitivity weights are priority coefficients pre-assigned to different neural network layers, used to characterize the importance of the gradient of that layer to model convergence.

[0085] S182. Add the preset sensitivity weight as a priority field to the header of the compressed gradient data packet so that the switching device can perform queue scheduling of the compressed gradient data packet according to the priority field. The compute nodes add a preset sensitivity weight as a priority field to the header of the compressed gradient data packets, such as in the Differential Service Code Point field of the Internet Protocol header or the priority field of a custom protocol header. The switching device uses this priority field to queue the compressed gradient data packets, inserting high-priority packets into the low-latency queue for priority forwarding. When network bandwidth resources are insufficient, the switching device can discard low-priority packets and trigger the compute nodes' fast retransmission mechanism.

[0086] S183. If the compressed gradient data packet is discarded, the retransmission waiting time is determined based on the current round-trip delay and the preset sensitivity weight. After the retransmission waiting time, the compressed gradient data packet is retransmitted.

[0087] If compressed gradient data packets are dropped by the switch during congestion, the compute node can determine the retransmission waiting time based on the current round-trip time and a preset sensitivity weight. Round-trip time refers to the time required for a data packet to travel from the sender to the receiver and back for acknowledgment. The retransmission waiting time is negatively correlated with the sensitivity weight; the higher the sensitivity weight, the shorter the retransmission waiting time. This ensures that important gradient data is recovered more quickly, reducing the impact of lost critical information on model training.

[0088] The retransmission wait time can be expressed as: Retransmission waiting time The calculation formula is as follows: ; in, This is the current estimated round-trip time delay.

[0089] As can be seen, this embodiment can prioritize the efficient transmission of high-priority gradient data, while ensuring the transmission of low-priority gradient data through a retransmission mechanism, thereby improving the transmission effect.

[0090] Based on the above embodiments, after generating the global gradient matrix, the aggregation node can broadcast it to all computing nodes. To improve broadcasting efficiency, the computing nodes are divided into at least two node groups, each of which may contain a leader node. The aggregation node can send the global gradient matrix only to the leader node, and the leader node will then send it a second time within the node group, thereby ensuring efficient distribution. In one embodiment, the method may further include: S191. Receive the global gradient matrix sent by the aggregation node to update the local model based on the global gradient matrix.

[0091] The compute node receives the global gradient matrix from the aggregation node and updates the local model weight parameters based on the global gradient matrix, thus completing the parameter synchronization for this round.

[0092] S192. When the computing node is the leader node of the node group, the global gradient matrix is ​​sent to other computing nodes in the node group through remote direct data access. During the broadcast phase, each compute node is divided into at least two node groups, each containing a leader node. The leader node is a special compute node responsible for data forwarding within its group. When a compute node is designated as the leader node of a node group, it distributes the global gradient matrix to other compute nodes within the group via Remote Direct Data Access (RDA). RDA is a network technology that bypasses the operating system kernel to directly read and write remote memory, enabling high-throughput, low-latency data distribution. This embodiment does not limit the underlying implementation of the RDA connection; for example, it can be implemented using an underlying network interface card (NIC) in the form of RDMA (Remote Direct Memory Access) or NTB (Non-Transparent Bridge).

[0093] S193. Monitor the confirmation response time of other computing nodes; S194. If the confirmed response time is greater than the preset response time threshold, reduce the rate at which data is sent to other computing nodes.

[0094] The leader node can monitor the acknowledgment response time of other computing nodes in real time. If the acknowledgment response time exceeds a preset threshold, it indicates that the corresponding receiver may have a processing bottleneck or link degradation, and the leader node reduces the rate at which it sends data to that computing node. Through adaptive rate adjustment, network congestion and buffer overflows caused by excessive data transmission to slow nodes can be avoided, improving the overall efficiency and stability of the broadcast phase.

[0095] For example, the leader node can monitor the ACK response time of each node in real time. ,like Exceeding the average of the sliding time window twice as much, that is, satisfying This will reduce the sending rate.

[0096] Based on the above embodiments, the implementation of this method on the aggregation node side will be described below. Please refer to... Figure 2 , Figure 2 A flowchart illustrating another data communication method provided in an embodiment of the present invention. This method is applied to an aggregation node and includes: S21. Receive the compressed gradient data packet sent by the computing node, and decompress and aggregate the compressed gradient data packet to obtain the initial global gradient matrix. The compressed gradient data packet is obtained by compressing the original gradient matrix generated by the computing node during model training, and is sent to the aggregation node through the transmission path determined according to the topology of the cluster network and the real-time link utilization.

[0097] As described above, compressed gradient data packets can be obtained by compressing the original gradient matrix generated during model training by the computing nodes, and then sent to the aggregation node through a transmission path determined according to the topology of the cluster network and the real-time link utilization, thereby ensuring communication efficiency. Subsequently, the aggregation node can receive the compressed gradient data packets sent by the computing nodes, and perform decompression and aggregation operations on the received compressed gradient data packets to generate the initial global gradient matrix for this round of iteration.

[0098] The following describes a method for decompression and aggregation processing. In one embodiment, decompressing and aggregating compressed gradient data packets may include: S211. Parse the header information of the compressed gradient data packet, extract the index mapping table, quantization compression information, and compressed payload; the index mapping table records the offset of the effective gradient element in the original gradient matrix and the storage address of the effective gradient element after compression. The aggregation node can parse the header information of the compressed gradient data packet, extract the index mapping table, quantization compression information, and compressed payload. The index mapping table records the offset of the valid gradient element in the original gradient matrix and the storage address of the valid gradient element in the compressed payload, enabling the aggregation node to accurately locate the position of each gradient element in the original data structure.

[0099] S212. Establish a global accumulator and initialize it as a zero matrix. The dimension of the global accumulator is the same as that of the original gradient matrix.

[0100] The aggregation node establishes a global accumulator, which is initialized as a zero matrix. The global accumulator is a matrix structure used to accumulate the gradient data from each node; its dimensions are consistent with the original gradient matrix, ensuring that the aggregation result matches the dimensions of the model parameters.

[0101] S213. Decode the compressed load to obtain an integer numerical sequence, and dequantize the integer numerical sequence according to the quantization compression information to obtain an approximate gradient value.

[0102] The aggregation node decodes the compressed payload to obtain an integer numerical sequence, and then dequantizes the integer numerical sequence based on the quantization scaling factor and minimum offset in the quantization compression information to obtain gradient approximations. Dequantization refers to the process of restoring discrete integer values ​​to floating-point approximations.

[0103] S214. Based on the offset recorded in the index mapping table, the gradient approximation value is accumulated to the corresponding position of the global accumulator to obtain the initial global gradient matrix.

[0104] The aggregation node accumulates the approximate gradient values ​​of each element to the corresponding position in the global accumulator based on the offsets recorded in the index mapping table. After processing the compressed gradient data packets from all computation nodes, the accumulated result in the global accumulator is the initial global gradient matrix. By using index-aligned decompression aggregation, invalid operations on zero elements can be avoided, thus improving aggregation efficiency.

[0105] Steps S211~S214 can be represented as: The aggregation node receives compressed gradient data packets from N nodes. .

[0106] Parse the header information of each data packet and extract the index mapping table. With quantization parameters .

[0107] Create a global accumulator, whose dimensions are consistent with the original model parameter dimensions, and initialize it as an all-zero matrix.

[0108] Compression load of the i-th node Entropy decoding is performed to obtain an integer sequence. .

[0109] and Perform dequantization operation ,approximation ,in This is the minimum offset during quantization.

[0110] According to the index mapping table Coordinate information in ,Will Accumulate to the corresponding position in the global accumulator.

[0111] Repeat the above process until all compressed gradient data packets for all nodes have been processed, resulting in an intermediate tensor containing the accumulated results. .

[0112] S22. Determine the residual between the decompressed gradient matrix and the predicted gradient matrix corresponding to the compressed gradient data packet, and filter the element coordinates in the decompressed gradient matrix according to the residual; the predicted gradient matrix is ​​obtained by the aggregation node based on the global gradient matrix of the previous round.

[0113] In this embodiment, considering that compression can lead to gradient data loss, affecting model training performance and even preventing model convergence, the aggregation node can filter out gradient elements with significant loss from the compressed gradient data packet and request the computation node to send the original values ​​of these gradient elements to compensate for the loss.

[0114] Aggregator nodes can decompress compressed gradient data packets to obtain decompressed gradient matrices. They can also predict gradient matrices based on the previous round's global gradient matrix. For example, the aggregation node maintains a gradient momentum term, which records the weighted cumulative value of historical global gradient matrices. The aggregation node linearly combines the previous round's global gradient matrix with the historical momentum term according to preset momentum coefficients to generate the current round's predicted gradient matrix. This method provides smoother prediction results when gradient directions change asymptotically, improving the accuracy of residual detection. Subsequently, the aggregation node calculates the residual between the decompressed gradient matrix and the predicted gradient matrix, and filters element coordinates in the decompressed gradient matrix based on the residual. The residual refers to the set of differences between gradient elements at corresponding positions, used to quantify the information bias caused by compression. In other words, when the gradient elements in the decompressed gradient matrix deviate significantly from those in the predicted gradient matrix, it can be considered that the gradient elements in the decompressed gradient matrix have not changed according to the expected trend, and therefore may have a bias. Therefore, gradient elements with significant biases can be effectively filtered out.

[0115] The following describes one method for filtering element coordinates. In one implementation, determining the residual between the decompressed gradient matrix and the predicted gradient matrix corresponding to the compressed gradient data packet, and filtering element coordinates in the decompressed gradient matrix based on the residual, may include: S221. Determine the residuals between the gradient elements at each position in the decompression gradient matrix and the corresponding positions in the prediction gradient matrix, and use the residuals to determine the norm of the decompression gradient matrix. The aggregation node determines the residual between each position in the decompression gradient matrix and the corresponding position in the predicted gradient matrix, and uses the residual to calculate the norm of the decompression gradient matrix, which can be expressed as: ; in, Predict the norm of the residuals between gradient matrices. This is the L2 norm, which is used here to comprehensively assess the overall size of the residuals.

[0116] S222. Determine whether the norm is greater than the preset residual threshold.

[0117] S223. If the value is greater than the value, then select the coordinates of the elements with the preset ratio from the decompression gradient matrix in descending order of residuals.

[0118] If the norm exceeds a preset residual threshold, the aggregation node triggers a correction. It then selects the coordinates of elements from the decompressed gradient matrix, prior to a preset proportion, in descending order of residual size. These coordinates are then used as the coordinates for the nodes requiring correction. By requesting supplementary data only for the coordinates of the largest residuals, additional communication overhead can be kept within a limited range, while prioritizing the correction of deviations that have the greatest impact on the global gradient.

[0119] Specifically, if the residual If the norm exceeds a preset convergence threshold, the aggregation node generates a correction instruction. Based on the correction instruction, the coordinates of the top K% elements with the largest residuals are selected, and the selection threshold is calculated. The formula is: ; in, For percentile function, K is a preset percentage constant.

[0120] S23. Send an incremental data request containing element coordinates to the computing node; The aggregation node sends an incremental data request containing the coordinates of the filtered elements to the compute node, requesting the transmission of the original gradient data at these specific locations.

[0121] S24. Receive the gradient elements sent by the computing node, and use the gradient elements to correct the initial global gradient matrix to obtain the global gradient matrix; the gradient elements are the data corresponding to the element coordinates in the original gradient matrix.

[0122] The aggregation node receives gradient elements from the computation nodes. These gradient elements are the high-precision values ​​corresponding to the coordinates of elements in the original gradient matrix. The aggregation node uses these gradient elements to correct the corresponding values ​​in the initial global gradient matrix, obtaining the global gradient matrix. In other words, the residual compensation data is superimposed onto the intermediate tensor. At the corresponding coordinates, the final global aggregate gradient tensor is generated. By correcting only the coordinates with larger residuals, compression loss can be effectively compensated without significantly increasing communication overhead, ensuring global gradient accuracy and model convergence stability.

[0123] Based on the above embodiments, to improve the efficiency of distributing the global gradient matrix, the aggregation node can also compress the global gradient matrix. In one implementation, after obtaining the global gradient matrix, the following may be included: S251. Determine the entropy value of the global gradient matrix.

[0124] After obtaining the global gradient matrix, the aggregation node calculates the entropy value of the global gradient matrix. Entropy is an indicator that characterizes the randomness and compressibility of data; the lower the entropy value, the more redundant or repetitive patterns exist in the data.

[0125] S252. If the entropy value is less than the preset entropy threshold, a dictionary-based compression algorithm is used to compress the global gradient matrix, and the compressed global gradient matrix is ​​broadcast.

[0126] If the entropy value is less than a preset entropy threshold, it indicates that the global gradient matrix has high compressibility. The aggregation node uses a dictionary-based compression algorithm to compress the global gradient matrix to generate smaller broadcast data. Through secondary compression, network bandwidth can be further saved during the broadcast phase, accelerating global parameter synchronization.

[0127] The step of broadcasting the global aggregated gradient tensor to each computing node includes a flow shaping mechanism: Calculate the global aggregate gradient tensor The entropy value H(G) is used to characterize the randomness and compressibility of the data: ; in, Let G be the probability distribution of the gradient value x in the tensor G; If the conditions are met Then a dictionary-based compression algorithm is used to compress the data. Perform secondary compression, in which This is a preset entropy threshold value; Based on the above embodiments, a retransmission mechanism can be set between computing nodes and aggregation nodes to prevent data loss.

[0128] S261. The received compressed gradient data packets are temporarily stored in the sliding window receiving buffer, and the sequence numbers of each compressed gradient data packet are checked to see if they are consecutive. The aggregation node temporarily stores the received compressed gradient data packets into the sliding window receive buffer and checks whether the sequence numbers of each compressed gradient data packet are consecutive. The sliding window receive buffer is a temporary storage area with a fixed capacity, managed according to the first-in, first-out principle, used to temporarily store out-of-order data packets.

[0129] S262. If a non-continuous sequence number is detected, start the out-of-order reordering timer.

[0130] If a sequence number discrepancy is detected, indicating missing or out-of-order data packets, the aggregation node starts an out-of-order reassembly timer. The out-of-order reassembly timer is a countdown mechanism used to wait for missing data packets to arrive.

[0131] S263. If the compressed gradient data packet corresponding to the missing sequence number arrives before the out-of-order reassembly timer expires, the step of decompressing and aggregating the compressed gradient data packet to obtain the initial global gradient matrix is ​​initiated. If the compressed gradient data packet corresponding to the missing sequence number arrives before the out-of-order reassembly timer expires, the aggregation node reassembles it in order and enters the decompression and aggregation process. By completing the reassembly within the timer's validity period, a certain degree of network out-of-order processing can be tolerated without interrupting the aggregation process.

[0132] S264. If the out-of-order reassembly timer times out and the compressed gradient data packet corresponding to the missing sequence number has not arrived, a retransmission request containing a list of missing sequence numbers is sent to the computing node.

[0133] If the out-of-order reordering timer times out and the compressed gradient data packet corresponding to the missing sequence number has not yet arrived, the aggregation node sends a retransmission request containing a list of missing sequence numbers to the compute node. By selectively requesting retransmissions instead of waiting for the full data, invalid waiting time can be reduced while ensuring data integrity, thus improving the robustness of gradient synchronization.

[0134] The above data communication method will be fully described below based on specific examples.

[0135] Please refer to Figure 3 , Figure 3 A complete flowchart of a data communication method provided in an embodiment of the present invention. This method includes: 1. In the backpropagation phase of distributed training, the original gradient tensor set of each computing node is represented as: ,in The total number of nodes. For the first Gradient data for each node; 2. The computation nodes perform tensor partitioning and sparsification based on statistical characteristics on the original gradient tensor set, generating a sparse gradient matrix containing effective gradient elements. and the corresponding index mapping table ; 3. Perform dynamic quantization and encoding on the sparse gradient matrix to generate a compressed gradient data packet set. ; 4. Based on the index mapping table and the preset aggregation tree algorithm, perform many-to-one reduction communication operation of compressed gradient data packets in the cluster communication network; 5. Receive compressed gradient data packets on the parameter server or master node, perform index-aligned decompression and aggregation operations, and generate a global aggregated gradient tensor. ; 6. Aggregate the global gradient tensor Broadcast to each computing node to update the model weight parameters of each node.

[0136] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating tensor partitioning and sparsification processing as provided in an embodiment of the present invention. The process involves performing tensor partitioning and sparsification processing on the original gradient tensor set based on statistical features, including the following steps: 1. Calculate the first... The original gradient tensor of each node L2 norm and the mean of the absolute values ​​of the gradient elements .

[0137] 2. Calculate the dynamic threshold based on L2 norm and mean absolute value. .

[0138] 3. Traversal Each gradient element in ,Will With dynamic threshold Comparison: like Then retain the gradient element and compare its value with the value in the gradient. 3D coordinate position Record to the intermediate buffer; like If so, the gradient element is set to zero, and the corresponding mask bits are generated. ; 4. Based on the preset granularity K, the valid gradient elements in the intermediate buffer are divided into K sub-tensor blocks, and independent header information is generated for each sub-tensor block. The header information includes the sub-tensor block ID, the source node ID, and the count of valid elements. 5. Generate an index mapping table based on the correspondence between mask bits and sub-tensor blocks. ,in It records the mapping relationship between the offset of non-zero elements in the original tensor and the compressed storage address.

[0139] Please refer to Figure 5 , Figure 5 This invention provides a flowchart for dynamic quantization and encoding of a sparse gradient matrix according to an embodiment of the present invention. The steps for dynamic quantization and encoding of a sparse gradient matrix include: 1. For each sub-tensor block, calculate the maximum value of its internal effective gradient elements. and minimum value ; 2. Calculate the quantization scaling factor ; 3. Apply the formula to each effective gradient element. Perform linear quantization to obtain an integer value. ; 4. Extract the exponent data E and mantissa data M from the sub-tensor block, and store E and M separately; 5. Use Huffman coding or arithmetic coding to process the quantized integer numerical sequence. Perform lossless compression to generate a compressive load; 6. Add the header information and quantization scaling factor of the sub-tensor block. The exponential bit data E and the compressed payload are encapsulated into a compressed gradient data packet. .

[0140] Please refer to Figure 6 , Figure 6 A flowchart of a many-to-one reduction communication operation is provided as an embodiment of the present invention. The many-to-one reduction communication operation of compressed gradient data packets is performed in a cluster communication network, and the steps include: 1. Obtain the adjacency matrix A of the cluster network topology and the real-time bandwidth utilization of each link. ; 2. For the source node With the target aggregation node Calculate path weights ; 3. Select path weights based on Dijkstra's algorithm or minimum spanning tree algorithm. The shortest communication path serves as the data transmission channel; 4. Determine if current network traffic is congested, specifically by monitoring the output queue length of the switching nodes. If the conditions are met If the condition is as described above, it is considered a congested state. This is a preset queue length threshold; 5. If congestion is detected, large-size compressed gradient data packets will be sent. Split into micro-batch packages The size of the micro-batch package meets the requirements. < ,in The preset maximum transmission unit threshold; 6. Based on timestamp Micro-batch packets are sent sequentially and reassembled into original compressed gradient data packets at the receiving end based on the sequence number.

[0141] Please refer to Figure 7 , Figure 7 This invention provides a flowchart for performing index-aligned decompression and aggregation operations. The steps for performing index-aligned decompression and aggregation operations include: 1. The parameter server receives data from... Compressed gradient data packets for each node ; 2. Parse the header information of each data packet and extract the index mapping table. With quantization parameters ; 3. Create a global accumulator, whose dimensions are consistent with the original model parameter dimensions, and initialize it as an all-zero matrix; 4. Traverse the first Compression load of each node Entropy decoding is performed to obtain an integer sequence. ; 5. Basis and Perform dequantization operation Restored to a floating-point gradient approximation ,in This is the minimum value offset during quantization; 6. Based on the index mapping table Coordinate information in ,Will Accumulate to the corresponding position in the global accumulator; 7. Repeat the above process until the compressed gradient data packets of all nodes have been processed, and an intermediate tensor containing the accumulated results is obtained. .

[0142] Please refer to Figure 8 , Figure 8 This is a schematic diagram of a communication optimization system provided in an embodiment of the present invention. This embodiment also provides a communication optimization system, including: The acquisition module is used to acquire the original gradient tensor set of each computing node in the current iteration cycle during the backpropagation phase of distributed training. The processing module is used to perform tensor fragmentation and sparsification based on statistical features on the original gradient tensor set, generating a sparse gradient matrix containing effective gradient elements and a corresponding index mapping table. The first generation module is used to dynamically quantize and encode the sparse gradient matrix based on the preset communication topology and the current network link bandwidth status, and generate a set of compressed gradient data packets. The operation module is used to perform many-to-one reduction communication operations of compressed gradient data packets in the cluster communication network based on the index mapping table and the preset aggregation tree algorithm. The second generation module is used to receive compressed gradient data packets from the parameter server or master node, perform index-aligned decompression and aggregation operations, and generate a global aggregated gradient tensor. The update module is used to broadcast the global aggregated gradient tensor to each computing node to update the model weight parameters of each node.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0144] Please refer to Figure 9 , Figure 9This is a structural block diagram of a data communication device provided in an embodiment of the present invention. The device is applied to a computing node and includes: Compression module 301 is used to compress the original gradient matrix generated during model training to obtain compressed gradient data packets; The transmission module 302 is used to determine the transmission path according to the topology of the cluster network and the real-time link utilization, and send the compressed gradient data packet to the aggregation node according to the transmission path, so that the aggregation node can decompress and aggregate the compressed gradient data packet to obtain the initial global gradient matrix. The request receiving module 303 is used to receive incremental data requests sent by the aggregation node; the incremental data request includes the element coordinates obtained by the aggregation node based on the residual between the decompressed gradient matrix and the predicted gradient matrix corresponding to the compressed gradient data packet, and the predicted gradient matrix is ​​predicted by the aggregation node based on the global gradient matrix of the previous round; The compensation sending module 304 is used to send the gradient elements corresponding to the element coordinates in the original gradient matrix to the aggregation node, so that the aggregation node can use the gradient elements to correct the initial global gradient matrix and obtain the global gradient matrix.

[0145] Optionally, the compression module includes: The sparsification unit is used to determine and retain the effective gradient elements in the original gradient matrix to obtain a sparse gradient matrix, and to divide the sparse gradient matrix into sub-tensor blocks according to a preset number of partitions. The sub-tensor block processing unit is used to set header information for the sub-tensor block and to quantize and compress the sub-tensor block to obtain the compressed payload; the header information includes the sub-tensor block identifier, the computation node identifier, and the effective element count. The index setting unit is used to set the index mapping table corresponding to the sparse gradient matrix; the index mapping table contains the offset of the effective gradient element in the original gradient matrix and the storage address of the compressed effective gradient element. The encapsulation unit is used to encapsulate the header information, the quantization and compression information of the sub-tensor block, the compression payload, and the index mapping table into a compressed gradient data packet.

[0146] Optionally, the sparsification unit includes: The norm calculation unit is used to determine the norm of the original gradient matrix, the mean absolute value of the gradient elements in the original gradient matrix, and to determine the screening threshold based on the norm and the mean absolute value. The filtering unit is used to retain gradient elements in the original gradient matrix whose absolute value is greater than or equal to the filtering threshold as valid gradient elements, and to set the gradient elements in the original gradient matrix whose absolute value is less than the filtering threshold to zero, thus obtaining a sparse gradient matrix.

[0147] Optionally, the sub-tensor block processing unit includes: The extreme value determination unit is used to determine the maximum and minimum values ​​of the effective gradient elements in the sub-tensor block, and to determine the quantization scaling factor based on the maximum and minimum values; The quantization unit is used to linearly quantize the effective gradient elements according to the quantization scaling factor and the minimum value to obtain an integer value. Extraction unit, used to extract the exponent and mantissa data of the sub-tensor block; The encoding unit is used to form an integer sequence from integer values ​​and to encode the integer sequence to obtain the encoded value. The load setting unit is used to set the exponent data, mantissa data, and encoded value as a compressed load.

[0148] Optionally, the transmission module includes: The acquisition unit is used to acquire the adjacency matrix of the cluster network and the real-time bandwidth utilization of each link; The path determination unit is used to determine candidate paths between computing nodes and aggregation nodes based on the adjacency matrix, and to determine path weights using the physical bandwidth and real-time bandwidth utilization of each link in the candidate paths. The selection unit is used to determine the transmission path from the candidate paths based on the path weight.

[0149] Optionally, the device further includes: The congestion handling module is used to obtain the real-time output queue length of the switching devices in the transmission path; if the real-time output queue length is greater than the preset queue length threshold, the compressed gradient data packet is split into micro-batch packets with a size less than or equal to the preset transmission unit threshold. The transmission module is also used for: Set a sequence number for the micro-batch packet and send the micro-batch packet to reassemble it into a compressed gradient data packet based on the sequence number at the aggregation node.

[0150] Optionally, the device further includes: The retransmission module is used to write compressed gradient data packets into the local retransmission buffer and assign sequence numbers to each compressed gradient data packet; receive retransmission requests sent by the aggregation node, which contain a list of missing sequence numbers; and extract the corresponding compressed gradient data packets from the local retransmission buffer according to the sequence number list for retransmission.

[0151] Optionally, the device further includes: The priority module is used to extract the neural network layer corresponding to the compressed gradient data packet and determine the preset sensitivity weight corresponding to the neural network layer; the preset sensitivity weight is added as a priority field to the data packet header of the compressed gradient data packet so that the switch device can perform queue scheduling of the compressed gradient data packet according to the priority field; if the compressed gradient data packet is dropped, the retransmission waiting time is determined according to the current round-trip delay and the preset sensitivity weight, and the compressed gradient data packet is retransmitted after the retransmission waiting time.

[0152] Optionally, the computing nodes are divided into at least two groups of nodes; The device also includes: The broadcast receiving module is used to receive the global gradient matrix sent by the aggregation node, so as to update the local model according to the global gradient matrix; when the computing node is the leader node of the node group, it sends the global gradient matrix to other computing nodes in the node group through remote direct data access; it monitors the confirmation response time of other computing nodes; if the confirmation response time is greater than the preset response time threshold, it reduces the rate of sending data to other computing nodes.

[0153] Please refer to Figure 10 , Figure 10 This is a structural block diagram of another data communication device provided in an embodiment of the present invention. The device is applied to an aggregation node and includes: The decompression and aggregation module 401 is used to receive compressed gradient data packets sent by the computing nodes and decompress and aggregate the compressed gradient data packets to obtain the initial global gradient matrix. The compressed gradient data packets are obtained by compressing the original gradient matrix generated by the computing nodes during model training and are sent to the aggregation node through a transmission path determined according to the topology of the cluster network and the real-time link utilization. The filtering module 402 is used to determine the residual between the decompressed gradient matrix and the predicted gradient matrix corresponding to the compressed gradient data packet, and to filter the element coordinates in the decompressed gradient matrix according to the residual; the predicted gradient matrix is ​​obtained by the aggregation node based on the global gradient matrix of the previous round. Request module 403 is used to send an incremental data request containing element coordinates to the compute node; The compensation module 404 is used to receive the gradient elements sent by the computing node and use the gradient elements to correct the initial global gradient matrix to obtain the global gradient matrix; the gradient elements are the data corresponding to the element coordinates in the original gradient matrix.

[0154] Optionally, the decompression aggregation module includes: The parsing unit is used to parse the header information of the compressed gradient data packet, extract the index mapping table, quantization compression information, and compression payload; the index mapping table records the offset of the effective gradient element in the original gradient matrix and the storage address of the effective gradient element after compression. The accumulator establishment unit is used to establish the global accumulator, initialize the global accumulator as a zero matrix, and the dimension of the global accumulator is consistent with the original gradient matrix. The decoding unit is used to decode the compressed payload to obtain an integer numerical sequence, and to dequantize the integer numerical sequence according to the quantization compression information to obtain an approximate gradient value. The accumulator unit is used to accumulate the gradient approximation to the corresponding position of the global accumulator according to the offset recorded in the index mapping table, so as to obtain the initial global gradient matrix.

[0155] Optionally, the filtering module includes: The residual determination unit is used to determine the residual between each position in the decompression gradient matrix and the corresponding position in the prediction gradient matrix, and to use the residual to determine the norm corresponding to the decompression gradient matrix. The judgment unit is used to determine whether the norm is greater than a preset residual threshold. The coordinate filtering unit is used to filter the coordinates of elements with a preset ratio from the decompression gradient matrix in descending order of residuals if the value is greater than the specified value.

[0156] Optionally, the device further includes: The secondary compression module is used to determine the entropy value of the global gradient matrix. If the entropy value is less than the preset entropy threshold, a dictionary-based compression algorithm is used to compress the global gradient matrix and broadcast the compressed global gradient matrix.

[0157] Optionally, the device further includes: The receive buffer module temporarily stores the received compressed gradient data packets in the sliding window receive buffer and checks whether the sequence numbers of each compressed gradient data packet are consecutive. If the sequence numbers are not consecutive, an out-of-order reassembly timer is started. If the compressed gradient data packet corresponding to the missing sequence number arrives before the out-of-order reassembly timer expires, the compressed gradient data packets are decompressed and aggregated to obtain the initial global gradient matrix. If the out-of-order reassembly timer expires and the compressed gradient data packet corresponding to the missing sequence number has not arrived, a retransmission request containing a list of missing sequence numbers is sent to the computing node.

[0158] For a description of the features in the embodiments corresponding to the data communication device, please refer to the relevant descriptions in the embodiments corresponding to the data communication method, which will not be repeated here.

[0159] Please refer to Figure 11 , Figure 11 This is a structural block diagram of a communication system provided in an embodiment of the present invention. An embodiment of the present invention also provides a communication system, including: Computing node 10 is used to execute the data communication method described above for computing nodes; Aggregator node 20 is used to execute the data communication method applied to the aggregation node described above.

[0160] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data communication method embodiments.

[0161] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data communication method embodiments when it is run.

[0162] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0163] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data communication method embodiments.

[0164] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data communication method embodiments.

[0165] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0166] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0167] The data communication method and communication system provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A data communication method applied to a computing node, characterized in that, include: The original gradient matrix generated during model training is compressed to obtain a compressed gradient data package. The transmission path is determined based on the topology of the cluster network and the real-time link utilization. The compressed gradient data packet is then sent to the aggregation node according to the transmission path, so that the aggregation node can decompress and aggregate the compressed gradient data packet to obtain the initial global gradient matrix. Receive incremental data requests sent by the aggregation node; the incremental data requests include element coordinates obtained by the aggregation node based on the residual between the decompressed gradient matrix and the predicted gradient matrix corresponding to the compressed gradient data packet, and the predicted gradient matrix is ​​predicted by the aggregation node based on the global gradient matrix of the previous round; The gradient elements corresponding to the coordinates of the elements in the original gradient matrix are sent to the aggregation node, so that the aggregation node can use the gradient elements to correct the initial global gradient matrix and obtain the global gradient matrix.

2. The data communication method according to claim 1, characterized in that, The original gradient matrix is ​​compressed to obtain a compressed gradient data packet, which includes: The sparse gradient matrix is ​​obtained by determining and retaining the effective gradient elements in the original gradient matrix, and the sparse gradient matrix is ​​divided into sub-tensor blocks according to the preset number of partitions. A header information is set for the sub-tensor block, and the sub-tensor block is quantized and compressed to obtain a compressed payload; the header information includes the sub-tensor block identifier, the computation node identifier, and the effective element count; Set an index mapping table corresponding to the sparse gradient matrix; the index mapping table includes the offset of the effective gradient element in the original gradient matrix and the storage address of the compressed effective gradient element; The header information, the quantization and compression information of the sub-tensor block, the compression payload, and the index mapping table are encapsulated into the compressed gradient data packet.

3. The data communication method according to claim 2, characterized in that, The sparse gradient matrix is ​​obtained by determining and retaining the effective gradient elements in the original gradient matrix, including: Determine the norm of the original gradient matrix and the mean absolute value of the gradient elements in the original gradient matrix, and determine the screening threshold based on the norm and the mean absolute value. The gradient elements in the original gradient matrix whose absolute values ​​are greater than or equal to the filtering threshold are retained as valid gradient elements, and the gradient elements in the original gradient matrix whose absolute values ​​are less than the filtering threshold are set to zero, thus obtaining the sparse gradient matrix.

4. The data communication method according to claim 2, characterized in that, The sub-tensor block is quantized and compressed to obtain a compression load, including: Determine the maximum and minimum values ​​of the effective gradient elements in the sub-tensor block, and determine the quantization scaling factor based on the maximum and minimum values; The effective gradient elements are linearly quantized according to the quantization scaling factor and the minimum value to obtain an integer value; Extract the exponent and mantissa data of the sub-tensor block; The integer values ​​are formed into an integer value sequence, and the integer value sequence is encoded to obtain the encoded value; The exponent data, the mantissa data, and the encoded value are set as the compressed payload.

5. The data communication method according to claim 1, characterized in that, The transmission path is determined based on the topology of the cluster network and the real-time link utilization, including: Obtain the adjacency matrix of the cluster network and the real-time bandwidth utilization of each link; Candidate paths between the computing node and the aggregation node are determined based on the adjacency matrix, and path weights are determined using the physical bandwidth and real-time bandwidth utilization of each link in the candidate paths. The transmission path is determined from the candidate paths based on the path weight.

6. The data communication method according to claim 1, characterized in that, Before sending the compressed gradient data packet to the aggregation node, the method further includes: Obtain the real-time output queue length of the switch devices in the transmission path; If the real-time output queue length is greater than the preset queue length threshold, the compressed gradient data packet is split into micro-batch packets with a size less than or equal to the preset transmission unit threshold. Sending the compressed gradient data packet to the aggregation node includes: A sequence number is set for the micro-batch packet, and the micro-batch packet is sent to be reassembled into the compressed gradient data packet based on the sequence number at the aggregation node.

7. The data communication method according to claim 1, characterized in that, Before sending the compressed gradient data packet to the aggregation node, the method further includes: Write the compressed gradient data packets into the local retransmission buffer and assign a sequence number to each of the compressed gradient data packets; After sending the compressed gradient data packet to the aggregation node, the process further includes: Receive a retransmission request sent by the aggregation node, the retransmission request containing a list of missing sequence numbers; The corresponding compressed gradient data packet is extracted from the local retransmission buffer according to the sequence number list and retransmitted.

8. The data communication method according to claim 1, characterized in that, Before sending the compressed gradient data packet to the aggregation node, the method further includes: Extract the neural network layer corresponding to the compressed gradient data packet, and determine the preset sensitivity weights corresponding to the neural network layer; The preset sensitivity weight is added as a priority field to the header of the compressed gradient data packet so that the switching device can perform queue scheduling of the compressed gradient data packet according to the priority field. If the compressed gradient data packet is dropped, a retransmission waiting time is determined based on the current round-trip delay and the preset sensitivity weight, and the compressed gradient data packet is retransmitted after the retransmission waiting time has elapsed.

9. The data communication method according to claim 1, characterized in that, The computing nodes are divided into at least two groups of nodes; The method further includes: Receive the global gradient matrix sent by the aggregation node, and update the local model according to the global gradient matrix; When the computing node is the leader node of the node group, the global gradient matrix is ​​sent to other computing nodes in the node group through remote direct data access. Monitor the confirmation response time of other computing nodes; If the confirmation response time is greater than a preset response time threshold, the rate at which data is sent to the other computing nodes is reduced.

10. A data communication method applied to an aggregation node, characterized in that, include: Receive compressed gradient data packets sent by computing nodes, and decompress and aggregate the compressed gradient data packets to obtain an initial global gradient matrix; The compressed gradient data packet is obtained by compressing the original gradient matrix generated by the computing node during model training, and is sent to the aggregation node through a transmission path determined according to the topology of the cluster network and the real-time link utilization. The residual between the decompressed gradient matrix and the predicted gradient matrix corresponding to the compressed gradient data packet is determined, and the element coordinates in the decompressed gradient matrix are selected according to the residual; the predicted gradient matrix is ​​obtained by the aggregation node based on the global gradient matrix of the previous round. Send an incremental data request containing the coordinates of the element to the computing node; The system receives gradient elements sent by the computing node and uses these gradient elements to correct the initial global gradient matrix to obtain a global gradient matrix; the gradient elements are the data corresponding to the element coordinates in the original gradient matrix.

11. The data communication method according to claim 10, characterized in that, The decompression and aggregation process for the compressed gradient data packets includes: The header information of the compressed gradient data packet is parsed to extract the index mapping table, quantization compression information, and compression payload; the index mapping table records the offset of the effective gradient element in the original gradient matrix and the storage address of the compressed effective gradient element; Establish a global accumulator and initialize it as a zero matrix. The dimension of the global accumulator is the same as that of the original gradient matrix. The compressed load is decoded to obtain an integer numerical sequence, and the integer numerical sequence is dequantized according to the quantization compression information to obtain an approximate gradient value; Based on the offset recorded in the index mapping table, the gradient approximation is accumulated to the corresponding position of the global accumulator to obtain the initial global gradient matrix.

12. The data communication method according to claim 10, characterized in that, Determine the residual between the decompressed gradient matrix and the predicted gradient matrix corresponding to the compressed gradient data packet, and filter element coordinates in the decompressed gradient matrix based on the residual, including: Determine the residual between each position in the decompression gradient matrix and the corresponding position in the prediction gradient matrix, and use the residual to determine the norm corresponding to the decompression gradient matrix; Determine whether the norm is greater than a preset residual threshold; If the value is greater than the value, then the coordinates of the elements with a preset ratio are selected from the decompression gradient matrix in descending order of residual value.

13. The data communication method according to claim 10, characterized in that, After obtaining the global gradient matrix, the process also includes: Determine the entropy value of the global gradient matrix; If the entropy value is less than a preset entropy threshold, a dictionary-based compression algorithm is used to compress the global gradient matrix, and the compressed global gradient matrix is ​​then broadcast.

14. The data communication method according to claim 10, characterized in that, Also includes: The received compressed gradient data packets are temporarily stored in the sliding window receive buffer, and the sequence numbers of each compressed gradient data packet are checked to see if they are consecutive. If a non-contiguous sequence number is detected, the out-of-order reordering timer is started; If the compressed gradient data packet corresponding to the missing sequence number arrives before the out-of-order reassembly timer expires, the step of decompressing and aggregating the compressed gradient data packet to obtain the initial global gradient matrix is ​​entered. If the out-of-order reassembly timer times out and the compressed gradient data packet corresponding to the missing sequence number does not arrive, a retransmission request containing a list of missing sequence numbers is sent to the computing node.

15. A communication system, characterized in that, include: A computing node for executing the data communication method as described in any one of claims 1 to 9; An aggregation node is used to perform the data communication method as described in any one of claims 10 to 14.

Images