Gradient communication compression method and system for shared memory heterogeneous processors

Abstract

The application discloses a gradient communication compression method and system for a shared memory heterogeneous processor, and comprises the following steps: each node respectively performs Top-k accurate sorting on the gradient obtained by performing forward propagation and reverse propagation on a deep learning model based on a general CPU in a specified round to compress and obtain the latest threshold value, and performs gradient compression based on the cooperation of a heterogeneous acceleration core and the general CPU and the threshold value of the previous round; each node respectively performs gradient communication, sparse decoding and parameter updating based on the general CPU; the training round step is increased by one; and it is judged whether the training round is ended according to the training round step to decide whether to continue iteration or exit. The application aims to optimize the communication algorithm for distributed deep learning training of the shared memory heterogeneous processor, solves the problem of low efficiency of the traditional gradient communication compression method, reduces the communication time, reduces the compression cost and improves the communication efficiency.

Description

Technical Field

[0001] This invention belongs to the field of computer system architecture, specifically relating to a gradient communication compression method and system for heterogeneous processors with shared memory. Background Technology

[0002] In recent years, Deep Neural Networks (DNNs) have achieved remarkable success in various fields. The widespread application of deep learning in natural language processing, image processing, audio analysis, financial forecasting, recommender systems, and medical diagnosis has gradually made it a focus of scientific research and industrial practice. Simultaneously, as distributed DNN model training has become commonplace, parallel intelligent training methods are becoming increasingly complex. To address this, hybrid parallel training frameworks based on PyTorch and TensorFlow, such as DeepSpeed, Merak, and Whale, have been proposed, aiming to improve the efficiency of parallel training at the training framework level. The rapid development of deep learning has largely benefited from powerful computing capabilities. However, with the increase in data volume and DNN model complexity, the communication time between nodes in distributed model training becomes unbearable. The number of parameters in large-scale deep learning models has rapidly expanded from millions to billions; therefore, communication overhead will become a major performance bottleneck in distributed training.

[0003] Currently, there are two main parallel schemes for distributed training: model parallelism and data parallelism. Each exhibits unique advantages in different scenarios, but the most important and simplest form is data parallelism. The core idea of ​​data parallelism is to replicate model parameters to all computing nodes. In a single iteration, each computing node processes different mini-batches of data to compute local gradient updates and synchronizes gradients with other nodes before updating model parameters. This generates significant communication overhead, and bandwidth costs increase even faster as the model grows. Furthermore, the communication speed between nodes is much lower than the communication speed within a node, further limiting the system's scalability. For example, in the new generation Sunway supercomputer, the communication bandwidth between supernodes is 3 / 16 times that of the communication bandwidth within a supernode. Currently, distributed SGD training algorithms can be categorized into four dimensions: communication synchronization, system architecture, parallelism of communication and computation, and compression techniques. Based on the frequency of communication between nodes and the number of nodes, synchronous communication can be divided into four schemes: synchronous, stale-synchronous, asynchronous, and Local Stochastic Gradient Descent (Local SGD), with the communication frequency decreasing in that order. In this dimension, a balance between convergence speed and communication frequency needs to be considered. The system architecture dimension can be divided into centralized and decentralized structures. Centralized structures can maintain global model parameters and support various synchronization mechanisms. Decentralized structures are more flexible, avoiding the problem of centralized structures relying too heavily on a central node. Regarding the parallelism of communication and computation, since deep learning models are hierarchical, communication and computation tasks can be parallelized; that is, by optimizing the order of computation and communication, the communication cost can be minimized. Compression methods compress the model parameters or parameter gradients that require communication to reduce the amount of communication. Most compression methods are lossy, so a trade-off must be found between convergence and communication overhead. Recently, researchers have paid extensive attention to gradient sparsity. Gradient sparsity means that during the training of a deep learning model, only a small number of parameter gradients contribute significantly to the model update. For example, in distributed training, gradient communication can be significantly sparsified, introducing up to 99.9% zero values ​​without significant loss of accuracy. Compression techniques are usually supplemented by strategies such as communication synchronization, system architecture, and parallelism of communication and computation: (1) PQASGD, in each iteration, the stochastic gradient is first calculated based on the local mini-batch data and the local model parameters are updated. When the predetermined number of cycles is reached, each node quantizes all parameter changes since the last synchronization and sends the quantized values ​​to all nodes. Finally, each node updates the model with the average quantized value; (2) gTop-k, after aggregating all gradients, gTop-k sparsifies the global gradient vector again and selects the k largest global gradients.Although the choice of architecture does not directly affect the synchronization time, the system architecture will work together with the compression technology to affect the performance of distributed model training; (3) ByteScheduler partitions and rearranges tensor transmissions, applying tensor partitioning to communication scheduling (even feedforward computation can be parallelized with communication). In addition, since gradient quantization and gradient sparsification are orthogonal, these two compression techniques can also be combined: (1) SparCML can sparsify gradient exchange with a precision of 4 bits per coordinate, and simultaneously overlap computation and communication; (2) STC is a new sparse ternary compression method. Specifically, STC uses Top-k sparsity and Ternary quantization.

[0004] In modern computing systems, heterogeneous computing architectures are gradually becoming mainstream, making shared-memory heterogeneous processors an important development direction. Shared-memory heterogeneous processors integrate multiple processors, allocating different tasks to the most suitable processor based on their respective characteristics. This not only improves task processing speed through parallel execution but also significantly optimizes energy consumption. Simultaneously, shared memory ensures efficient data exchange and accelerates communication between cores. When facing large-scale neural network training, shared-memory heterogeneous processors can leverage their advantages. Their heterogeneous architecture can fully utilize the computational strengths of different processors to execute complex and diverse tasks in parallel, while shared memory also helps workloads efficiently obtain allocated tasks. These cutting-edge achievements fully highlight the potential of combining gradient communication compression techniques with various synchronization strategies. However, current gradient synchronization methods in training methods have high communication overhead, and there is a lack of gradient compression methods optimized for shared-memory heterogeneous processors. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a gradient communication compression method and system for heterogeneous processors with shared memory, addressing the aforementioned problems of the prior art. This invention aims to optimize the communication algorithm for distributed deep learning training on heterogeneous processors with shared memory, solve the problem of low efficiency of traditional gradient communication compression methods, reduce communication time, reduce compression costs, and improve communication efficiency.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A gradient communication compression method for heterogeneous processors with shared memory includes the following steps:

[0008] S1, initialize the training round step and compression ratio;

[0009] S2, each node performs Top-k precise sorting based on the gradients obtained from forward and backward propagation of the deep learning model in a specified round using a general-purpose CPU, combined with the compression ratio, to obtain the latest threshold for compression. Gradient compression is then performed based on the heterogeneous acceleration cores of shared memory in the heterogeneous processor and the general-purpose CPU in collaboration with the threshold from the previous round.

[0010] S3, each node communicates gradients with other nodes based on the general-purpose CPU to exchange gradients. After the gradient exchange is completed, the collected gradients are sorted and decoded based on the general-purpose CPU, and then the parameters of the deep learning model are updated.

[0011] S4 increments the training round step;

[0012] S5: Determine whether the training round has ended based on the training round step. If it has ended, exit; otherwise, jump to step S2.

[0013] Optionally, the forward and backward propagation in step S2 are performed via heterogeneous accelerated cores in a shared-memory heterogeneous processor.

[0014] Optionally, step S2 includes:

[0015] S101: Traverse the list T of gradient tensors obtained by performing forward and backward propagation on the deep learning model to obtain the current gradient tensor t. If the traversal is complete, end and exit; otherwise, jump to step S102.

[0016] S102, based on the general-purpose CPU, multiply the compression ratio by the size of the current gradient tensor t to obtain the number of gradients k that should remain after compression for the current gradient tensor t;

[0017] S103, determine if the training epoch step is divisible by the specified epoch τ. If it is, then based on the general-purpose CPU, transform the gradient tensor t to obtain the one-dimensional result and the result after taking the absolute value. Sort the result by Topk and select the k largest gradient values ​​and positions with the highest absolute values, storing them in the gradient value variable value and the position variable position, respectively. Also, based on the general-purpose CPU, use the variable topk_values ​​to store the kth largest gradient value as the updated threshold. Simultaneously, based on the heterogeneous acceleration kernel, compare the kth largest gradient value of this epoch with the kth largest gradient threshold topk_values ​​of the previous epoch to obtain the gradient position p to be retained. ositions; using the gradient positions to be retained as indices, extract the gradient values ​​values ​​from the current gradient tensor t; using a general-purpose CPU, pad or prune the gradient values ​​values ​​and gradient positions to the gradient size required for compression to achieve alignment; use a list tensor_idx to record the number of compressed gradient values ​​for each parameter; using the list tensor_idx as the starting position, put the gradient values ​​values ​​of each parameter into the list flat_values ​​and the gradient positions into the list flat_positions; jump to step S101.

[0018] Optionally, in step S102, the function expression for multiplying the compression ratio by the size of the current gradient tensor t based on the general-purpose CPU to obtain the number of gradients k that should remain after compression for the current gradient tensor t is:

[0019] ,

[0020] In the above formula, This indicates taking the maximum value. For compression ratio, The tensor size of the current gradient tensor t.

[0021] Optionally, the function expression for transforming the gradient tensor t using a general-purpose CPU to obtain a one-dimensional result and taking its absolute value, and then sorting it using Topk and selecting the k largest gradient values ​​and positions by absolute value, is as follows:

[0022] ,

[0023] in, These represent the top k largest gradient values ​​and positions, respectively. For Top-k sorting functions, This represents the result of transforming the gradient tensor t to obtain a one-dimensional form and taking its absolute value. To make it one-dimensional, To take the absolute value, This indicates that the results are sorted by value.

[0024] Optionally, in step S3, each node performs gradient communication based on a general-purpose CPU to exchange gradients with other nodes, including: asynchronously sending the gradient value of the node to perform gradient compression using a communication function, waiting for other nodes to return the corresponding gradient compression results, and ending the waiting after receiving the corresponding gradient compression results from all or all other nodes to update the model parameters after performing gradient decoding on the gradient values ​​of other nodes.

[0025] Optionally, in step S3, organizing and decoding the collected gradients based on the general-purpose CPU means accumulating the gradient values ​​of other nodes and the gradient value of this node based on position using the general-purpose CPU.

[0026] Furthermore, the present invention also provides a gradient communication compression system for heterogeneous processors with shared memory, comprising a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the gradient communication compression method for heterogeneous processors with shared memory.

[0027] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the gradient communication compression method for heterogeneous processors with shared memory via a processor.

[0028] Furthermore, the present invention also provides a computer program product, including a computer program or instructions that are programmed or configured to execute the gradient communication compression method for heterogeneous processors with shared memory via a processor.

[0029] Compared with existing technologies, this invention has the following main advantages: In the neural network model training stage, this invention primarily uses heterogeneous accelerated cores, while in the gradient compression and gradient communication stages, it primarily uses general-purpose CPUs. By combining heterogeneous accelerated cores and general-purpose CPUs, it optimizes the communication algorithm for distributed deep learning training on shared-memory heterogeneous processors, achieving the following beneficial effects: 1) This invention can significantly reduce communication time while maintaining a convergence speed close to that of dense communication. 2) This invention reduces the cost of gradient compression itself, achieving a simpler and more effective gradient selection mechanism. 3) This invention has good scalability and can maintain high communication efficiency even in situations with numerous and complex nodes. 4) This invention is suitable for various supercomputer environments and has the advantage of good versatility. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0031] Figure 2 This is a communication diagram illustrating the combination of local stochastic gradient descent and synchronization in an embodiment of the present invention.

[0032] Figure 3 This is a flowchart of compression, communication, and decoding in an embodiment of the present invention.

[0033] Figure 4 This is a flowchart of Top-k gradient compression in an embodiment of the present invention.

[0034] Figure 5 This is a schematic diagram illustrating the operation allocation of heterogeneous processors sharing memory during gradient compression in an embodiment of the present invention.

[0035] Figure 6 This is a schematic diagram illustrating the principle of the threshold-based approximate Top-k in an embodiment of the present invention. Detailed Implementation

[0036] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0037] like Figure 1 As shown, the gradient communication compression method for heterogeneous processors with shared memory in this embodiment includes the following steps:

[0038] S1, initialize the training round step and compression ratio;

[0039] S2, each node performs Top-k precise sorting based on the gradients obtained from forward and backward propagation of the deep learning model in a specified round using a general-purpose CPU, combined with the compression ratio, to obtain the latest threshold for compression. Gradient compression is then performed based on the heterogeneous acceleration cores of shared memory in the heterogeneous processor and the general-purpose CPU in collaboration with the threshold from the previous round.

[0040] S3, each node communicates gradients with other nodes based on the general-purpose CPU to exchange gradients. After the gradient exchange is completed, the collected gradients are sorted and decoded based on the general-purpose CPU, and then the parameters of the deep learning model are updated.

[0041] S4 increments the training round step;

[0042] S5: Determine whether the training round has ended based on the training round step. If it has ended, exit; otherwise, jump to step S2.

[0043] This embodiment uses the distributed deep learning framework PyTorch as an example, and implements gradient synchronization communication operations during the software execution process for distributed deep learning system software such as PyTorch and TensorFlow. After each node completes forward and backward propagation, it obtains its own gradient value. See also... Figure 1 This embodiment of the gradient communication compression method for shared-memory heterogeneous processors begins by initializing the training epoch step and a custom compression ratio (compression_ratio), which is also the starting point of the entire deep learning model training. It checks if the training epoch step is divisible by 20 (step%20 = 0). If it is, the gradients in this epoch are precisely sorted using torch.topk and updated accordingly. Gradient filtering is performed based on the threshold comparison from the previous epoch, and the gradient length is modified to match the compression ratio. The gradients of all parameters are merged into a list for gradient communication. After communication is complete, the gradient outputs of each parameter are traversed to decode the sparse parameter gradients and update the parameters. Finally, the training epoch step is incremented, and the training is checked against the training epoch step to determine if training has ended. If not, the next epoch begins.

[0044] Figure 2 This diagram illustrates the communication between local stochastic gradient descent and synchronization in this embodiment. Taking the parallel data processing of four nodes (A, B, C, and D) as an example, each node performs model training independently. In the diagram, a through d represent different training processes: a represents the forward propagation process; b represents the backpropagation process; c represents the gradient aggregation process, which can be divided into gradient compression, gradient communication, gradient summation, and averaging; and d represents the parameter update process. The four nodes repeat the cycle of forward propagation (a), backpropagation (b), gradient aggregation (c), and parameter update (d) twice. The gradient aggregation (c) process includes gradient compression and communication operations, which helps each node obtain more gradient information for parameter updates. This embodiment iterates through the above process.

[0045] Figure 3This is a flowchart of compression, communication, and decoding in an embodiment of the present invention. The basic Top-k compression algorithm can be divided into four steps. First, calculate the value of k for all gradients trained in one session. The value of k represents the number of gradients retained after compression. Simultaneously, use start_id and stop_id to record the interval of each gradient. Second, flattening and tensor compression are involved. Flattening refers to reducing the tensor dimension to 1 for comparison. The square labeled 'a' in the diagram represents the gradient data with the largest absolute value, which will be retained, while the data represented by the other squares will not be used for parameter updates. Third, the compressed gradients are communicated via Allgather between nodes. Gradients from squares other than those labeled 'a' will not be transmitted. In the diagram, this represents the relative position of the retained gradients; that is, both the magnitude and position information of the retained gradients are transmitted between nodes. Finally, each node decodes the collected gradient data. The square labeled 'b' represents the gradient at that position that has been accumulated.

[0046] Figure 5 This diagram illustrates the operation allocation of shared memory heterogeneous processors during gradient compression in this embodiment. See also... Figure 5 In step S2, forward and backward propagation are executed via heterogeneous accelerated cores in a shared-memory heterogeneous processor. Figure 5 As shown, due to the stronger parallel processing capability of heterogeneous acceleration cores, the forward propagation, backward propagation, threshold comparison, and parameter update stages of model training are all implemented on the heterogeneous acceleration cores. Among them, threshold comparison involves matrix operations, which are more suitable for implementation on heterogeneous acceleration cores. Most operations in the gradient compression and communication decoding stages are offloaded to the general-purpose CPU, including calculating the k value, gradient deformation, aligning the number of elements with k, gradient communication, gradient aggregation, Topk sorting, and updating the threshold. In this embodiment, by leveraging the advantages of shared memory heterogeneous processors, the general-purpose CPU directly accesses gradient data through shared memory to achieve compression operations, while the heterogeneous acceleration core continues to train the model. This not only utilizes the advantages of the general-purpose CPU in efficiently processing small batches of data operations, but also fully leverages the parallel advantages of heterogeneity, eliminating the communication overhead between the general-purpose CPU and the heterogeneous acceleration core.

[0047] Figure 4 The flowchart for Top-k gradient compression in this embodiment is shown below. Figure 4 In this embodiment, step S2 includes:

[0048] S101, traverse the gradient tensor list T obtained by performing forward and backward propagation on the deep learning model to obtain the current gradient tensor t. If the traversal is complete, end and exit; otherwise, jump to step S102; the gradient tensor list T is:

[0049] ,

[0050] in, ~ These are the gradient tensors in the gradient tensor list T.

[0051] S102, based on the general-purpose CPU, multiply the compression ratio by the size of the current gradient tensor t to obtain the number of gradients k that should remain after compression for the current gradient tensor t;

[0052] S103, determine whether the training round step is divisible by the specified round τ (in this embodiment, τ is 20). If it is divisible by the specified round τ, then the gradient tensor t is transformed by the general-purpose CPU to obtain the one-dimensional result and the result after taking the absolute value. The result is sorted by Topk and the k largest gradient values ​​and positions are selected and stored in the gradient value variable value and the position variable position, respectively. The k largest gradient value is stored in the variable topk_values ​​as the updated threshold by the general-purpose CPU. At the same time, the k largest gradient value of the current round is compared with the k largest gradient threshold topk_values ​​of the previous round by the heterogeneous acceleration kernel to obtain the gradient positions to be retained. Using the gradient positions to be retained as indices, the gradient values ​​are extracted from the current gradient tensor t. Based on a general-purpose CPU, the gradient values ​​and gradient positions are padded (with zeros) or pruned (removing excess gradient values) to the required gradient size for compression, thus aligning the tensors to maintain the same shape for easy communication. A list `tensor_idx` is used to record the number of compressed gradient values ​​for each parameter. Using `tensor_idx` as the starting position, the gradient values ​​for each parameter are placed into the list `flat_values`, and the gradient positions are placed into the list `flat_positions`. The process then jumps to step S101. In this embodiment, both `flat_values` and `flat_positions` are one-dimensional tensors, and the initialization method for `flat_values` is as follows:

[0053] ,

[0054] in, This is the preset total number of gradient elements after compression. In PyTorch, the torch.empty function is used to create an uninitialized tensor of a specified size and data type to initialize parameters.

[0055] The list flat_positions is initialized as follows:

[0056] ,

[0057] in, `torch.int` is a parameter that specifies the data type of the tensor. `torch.int` indicates that the elements in the tensor will be integers. PyTorch supports various data types, including integers, floating-point numbers, complex numbers, etc. The list `tensor_idx` is initialized to 0, i.e., `tensor_idx = [0]`; the variable `topk_values` is initialized to empty, i.e., `topk_values ​​= [].`

[0058] In step S102 of this embodiment, the function expression for multiplying the compression ratio by the size of the current gradient tensor t based on the general-purpose CPU to obtain the number of gradients k that should remain after compression for the current gradient tensor t is as follows:

[0059] ,

[0060] In the above formula, This indicates taking the maximum value. For compression ratio, The tensor size of the current gradient tensor t.

[0061] In step S103 of this embodiment, the gradient tensor t is transformed using a general-purpose CPU to obtain a one-dimensional result and the result after taking the absolute value. The function expression for sorting by Topk and selecting the k largest gradient values ​​and positions by absolute value is as follows:

[0062] ,

[0063] in, These represent the top k largest gradient values ​​and positions, respectively. For Top-k sorting functions, This represents the result of transforming the gradient tensor t to obtain a one-dimensional form and taking its absolute value. To make it one-dimensional, To take the absolute value, This indicates that the results are sorted by value.

[0064] One advantage of the gradient communication compression method for shared-memory heterogeneous processors in this embodiment is its ease of parallel implementation. During gradient compression, this method uses `torch.topk` and simple size comparison to obtain the position of the larger gradient within the original gradient tensor. The compressed gradient needs to be stored as a one-dimensional array. However, since both the output and input data structures in this embodiment are gradient tensor arrays, it is necessary to ensure that each gradient corresponds to the correct position of its associated parameter and tensor. Therefore, gradient positions (`positions`) are used to store the position of each valid gradient value in the one-dimensional gradient tensor, and a list `tensor_idx` is used to record the interval of the valid gradient for each parameter within the array. This gradient communication compression method for shared-memory heterogeneous processors is implemented based on the deep learning computing frameworks PyTorch and Merak, and the gradient is stored as a multi-dimensional tensor at both the algorithm input and output.

[0065] Figure 6 This diagram illustrates the principle of the threshold-based approximate Top-k algorithm in this embodiment. The circle below the top horizontal axis represents the training epochs, showing epochs 1 to 26. Every 20 epochs, a new epoch is performed, in which only operation ① is executed; the remaining epochs execute operation ②. Operation ① demonstrates the use of Top-k sorting to accurately determine the k largest gradients. Operation ② compares the threshold on the left with the gradient tensor on the right, obtaining the magnitude and position information of gradients greater than or equal to the threshold (marked with dashed circles in the diagram). Assuming k is 3, the three gradients with relatively large absolute values ​​to the left of the dashed line in operation ① will be retained, and the k-th largest gradient will be determined as the latest threshold.

[0066] In step S3 of this embodiment, each node performs gradient communication based on a general-purpose CPU to exchange gradients with other nodes. This includes: asynchronously sending the gradient value of the node to perform gradient compression using a communication function, waiting for other nodes to return their corresponding gradient compression results, and ending the waiting after receiving all or all required gradient compression results from other nodes to decode the gradient values ​​of other nodes and update the model parameters. Specifically, the all_gather function used for communication in this embodiment is actually implemented based on torch.distributed.all_gather. all_gather adds a feature that allows for non-communication when dealing with a single node. Messages will be sent asynchronously using all_gather, which will fully utilize the parallelism of the inter-node network and the nodes themselves. During gradient decoding, the gradient needs to be accumulated by combining the position information. Enabling torch.distributed.all_gather enables communication between nodes, asynchronously waiting for all communication to end, and using lists workers_values ​​and workers_positions to collect the complete gradient values ​​and gradient positions of all nodes; each node iteratively decodes the gradient value into the gradient list grad_out according to its position, and the gradient value of the unretained gradient is 0 by default. The gradient list grad_out is the decoded output, which can be used to update the parameters of the deep learning model.

[0067] In step S3 of this embodiment, organizing and decoding the collected gradients based on the general-purpose CPU means accumulating the gradient values ​​of other nodes and the gradient value of this node based on position using the general-purpose CPU to restore the original gradient tensor shape.

[0068] To represent the compressed gradients with less data, this embodiment involves multiple modifications to the gradient storage format. The data structures involved in the algorithm include: grad_in: a tensor array storing the original gradient tensors of each parameter; Values_list: a 2D array, shape=(gradients_number, gradient_k), containing the effective gradient values ​​of each parameter after compression; Position_list: a 2D array, shape=(gradients_number, gradient_k), containing the effective gradient positions of each parameter after compression; flat_values: a 1D array containing the top-k gradient values ​​of all parameters; flat_positions: a 1D array containing the top-k gradient positions of all parameters; worker_values: a 2D array, shape=(workers_number, sum_k), containing the top-k gradient values ​​of all nodes; worker_positions: a 2D array, shape=(workers_number, sum_k), containing the top-k gradient positions of all nodes; grad_out: a tensor array storing the processed gradient tensors of each parameter. These data structures demonstrate the storage format of gradient data in each step of the algorithm.

[0069] In summary, addressing the issue that existing gradient communication compression methods rarely consider the characteristics of domestically produced supercomputers and the synergistic effects with other communication optimization algorithms, this embodiment proposes a gradient communication compression method for shared-memory heterogeneous processors. This method includes the following steps: First, using a general-purpose CPU, perform Top-k precise sorting of gradient values ​​in a specified round to obtain the latest threshold. Second, perform approximate Top-k gradient compression based on the threshold, with some operations offloaded to the general-purpose CPU to fully utilize the advantages of shared heterogeneous processors. Third, nodes communicate in parallel to exchange gradients. After communication, each node locally organizes and decodes the collected gradients and performs normal parameter updates. Then, the system starts the next round of model training until the new round of backpropagation is completed. This embodiment can significantly reduce communication time while maintaining a convergence speed close to that of dense communication. In terms of algorithm overhead, it reduces the cost of the gradient compression process itself. This embodiment has good scalability and maintains high communication efficiency even in scenarios with numerous and complex nodes. This embodiment is suitable for various supercomputer environments and has the advantage of good versatility.

[0070] Furthermore, this embodiment also provides a gradient communication compression system for heterogeneous processors with shared memory, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the gradient communication compression method for heterogeneous processors with shared memory.

[0071] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the gradient communication compression method for heterogeneous processors with shared memory via a processor.

[0072] Furthermore, this embodiment also provides a computer program product, including a computer program or instructions that are programmed or configured to execute the gradient communication compression method for heterogeneous processors with shared memory via a processor.

[0073] Those skilled in the art will understand that the technical solutions provided by the embodiments of this application may be in the form of a method, system, or computer program product. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create an implementation for the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0074] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A gradient communication compression method for heterogeneous processors with shared memory, characterized in that, Includes the following steps: S1, initialize the training round step and compression ratio; S2, each node performs Top-k precise sorting based on the gradients obtained from forward and backward propagation of the deep learning model in a specified round using a general-purpose CPU, combined with the compression ratio, to obtain the latest threshold for compression. Gradient compression is then performed based on the heterogeneous acceleration cores of shared memory in the heterogeneous processor and the general-purpose CPU in collaboration with the threshold from the previous round. S3, each node communicates gradients with other nodes based on the general-purpose CPU to exchange gradients. After the gradient exchange is completed, the collected gradients are sorted and decoded based on the general-purpose CPU, and then the parameters of the deep learning model are updated. S4 increments the training round step; S5: Determine whether the training round has ended based on the training round step; if it has ended, exit. Otherwise, proceed to step S2; In step S2, forward propagation and backward propagation are performed through heterogeneous accelerated cores in a shared-memory heterogeneous processor; Step S2 includes: S101: Traverse the list T of gradient tensors obtained by performing forward and backward propagation on the deep learning model to obtain the current gradient tensor t. If the traversal is complete, end and exit. Otherwise, proceed to step S102; S102, based on the general-purpose CPU, multiply the compression ratio by the size of the current gradient tensor t to obtain the number of gradients k that should remain after compression for the current gradient tensor t; S103, determine if the training epoch step is divisible by the specified epoch τ. If it is, then based on the general-purpose CPU, transform the gradient tensor t to obtain the one-dimensional result and the result after taking the absolute value. Sort the result by Topk and select the k largest gradient values ​​and positions with the highest absolute values, storing them in the gradient value variable value and the position variable position, respectively. Also, based on the general-purpose CPU, use the variable topk_values ​​to store the kth largest gradient value as the updated threshold. Simultaneously, based on the heterogeneous acceleration kernel, compare the kth largest gradient value of this epoch with the kth largest gradient threshold topk_values ​​of the previous epoch to obtain the gradient position p to be retained. ositions; using the gradient positions to be retained as indices, extract the gradient values ​​values ​​from the current gradient tensor t; using a general-purpose CPU, pad or prune the gradient values ​​values ​​and gradient positions to the gradient size required for compression to achieve alignment; use a list tensor_idx to record the number of compressed gradient values ​​for each parameter; using the list tensor_idx as the starting position, put the gradient values ​​values ​​of each parameter into the list flat_values ​​and the gradient positions into the list flat_positions; jump to step S101.

2. The gradient communication compression method for heterogeneous processors with shared memory according to claim 1, characterized in that, In step S102, the function expression for multiplying the compression ratio by the size of the current gradient tensor t based on the general-purpose CPU to obtain the number of gradients k that should remain after compression for the current gradient tensor t is as follows: ， In the above formula, This indicates taking the maximum value. For compression ratio, The tensor size of the current gradient tensor t.

3. The gradient communication compression method for heterogeneous processors with shared memory according to claim 1, characterized in that, In step S103, the gradient tensor t is transformed using a general-purpose CPU to obtain a one-dimensional result and the result after taking the absolute value. The function expression for sorting by Topk and selecting the k largest gradient values ​​and positions by absolute value is as follows: ， in, These represent the top k largest gradient values ​​and positions, respectively. For Top-k sorting functions, This represents the result of transforming the gradient tensor t to obtain a one-dimensional form and taking its absolute value. To make it one-dimensional, To take the absolute value, This indicates that the results are sorted by value.

4. The gradient communication compression method for heterogeneous processors with shared memory according to claim 3, characterized in that, In step S3, each node performs gradient communication based on the general-purpose CPU to exchange gradients with other nodes, including: asynchronously sending the gradient value of the node to perform gradient compression using the communication function, waiting for other nodes to return the corresponding gradient compression results, and ending the waiting after receiving the corresponding gradient compression results from all or all other nodes that meet the requirements, so as to sparsify the gradient values ​​of other nodes, decode the gradients, and update the model parameters.

5. The gradient communication compression method for heterogeneous processors with shared memory according to claim 4, characterized in that, In step S3, organizing and decoding the collected gradients based on the general-purpose CPU means accumulating the gradient values ​​of other nodes and the gradient values ​​of this node based on their positions using the general-purpose CPU.

6. A gradient communication compression system for heterogeneous processors with shared memory, comprising interconnected microprocessors and memory, characterized in that, The microprocessor is programmed or configured to execute the gradient communication compression method for shared memory heterogeneous processors as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute the gradient communication compression method for shared memory heterogeneous processors as described in any one of claims 1 to 5 via a processor.

8. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute the gradient communication compression method for shared memory heterogeneous processors as described in any one of claims 1 to 5 via a processor.

Images