High-performance data parallel DNN training-oriented sparse compression method and device
By combining the local and global Top-k sparse compression scheme of multi-dimensional gradient tensors, DNN training is sparsely compressed, which solves the problems of large communication overhead and reduced accuracy in traditional methods, and realizes efficient gradient information retention and training throughput Increase in quantity.
Patent Information
- Application Number
- CN202411182560.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-22
AI Technical Summary
The traditional Top-k sparse compression method leads to large communication overhead and reduces model accuracy in parallel DNN training for high-performance data, and is prone to loss of gradient dimension information, affecting convergence accuracy and training throughput.
Combining the first sparse compression scheme and the second sparse compression scheme, the multi-dimensional gradient tensor is alternately compressed. The first scheme selects the largest k/M elements in each dimension, and the second scheme selects k among all elements. The largest element is adjusted according to the minimum value during the iteration process to determine the compression scheme, ensuring the retention of gradient dimension information and communication efficiency.
The model accuracy and training throughput are improved, and the dimension loss and low GPU utilization of traditional Top-k sparse compression are avoided. At the same time, the sparse compression method is selected more carefully in later iterations to optimize training efficiency.
Smart Images

Figure CN119005265A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer storage, and more specifically, to a sparse compression method and device for high-performance data parallel DNN training. Background Art
[0002] In the prior art, many gradient compression algorithms have been proposed to alleviate the communication bottleneck in high-performance data-parallel DNN training systems by reducing the amount of data transmitted during gradient synchronization. These algorithms include sparsification compression. For different gradient sparsification compression techniques, Top-k sparsification compression has been proven to be one of the most effective compression methods. The traditional Top-k sparsification process selects the k gradient elements with the largest absolute values from all elements of the gradient, and sets the remaining elements to zero.
[0003] However, Top-k sparsification is a lossy compression that will damage the convergence accuracy of the training model. Although the error feedback method can alleviate the accuracy loss, the traditional Top-k sparsification compression may cause the gradient to lose some important information, such as the gradient dimension, making it impossible to recover the convergence accuracy; and the traditional Top-k sparsification usually uses radix sorting to select the largest k elements. In this way, the traditional Top-k sparsification compression requires repeated and global radix sorting of the gradient dimension, so multiple buckets need to be synchronized between GPU cores multiple times, resulting in a large sparsification compression overhead. At the same time, global sorting requires synchronization of sorting results between GPU cores, which reduces the degree of parallelism and makes the compression process inefficient, thereby reducing the training throughput. Summary of the invention
[0004] In view of the defects of the prior art, the purpose of this application is to provide a sparse compression method and device for high-performance data parallel DNN training, aiming to solve the problem that the traditional Top-k sparse compression method has high communication overhead and reduces the performance of the model accuracy.
[0005] To achieve the above objectives, in a first aspect, the present application provides a sparse compression method for high-performance data parallel DNN training, comprising:
[0006] Get the multi-dimensional gradient tensor generated by backpropagation in a data-parallel DNN training system;
[0007] Constructing a matrix based on the input dimension information and the output dimension information of the multi-dimensional gradient tensor to obtain a multi-dimensional gradient tensor matrix;
[0008] Compressing the multi-dimensional gradient tensor matrix using a first sparsification compression scheme and a second sparsification compression scheme;
[0009] Wherein, the first sparse compression scheme includes:
[0010] Locally sorting the elements in each dimension of the multi-dimensional gradient tensor matrix;
[0011] In each dimension of the sorted multidimensional gradient tensor, select the k / M largest elements and retain them, and set the remaining elements to zero, where k is the number of gradient elements obtained based on the preset compression rate and the number of all elements of the multidimensional gradient tensor, and M is the number of dimensions;
[0012] The second sparse compression scheme includes:
[0013] Among all the elements of the multi-dimensional gradient tensor matrix, k elements with the largest absolute values are selected and retained, and the remaining elements are set to zero.
[0014] This application combines traditional Top-k sparse compression with another full-dimensional Top-k sparse compression to ensure that the gradient tensor obtains elements from each dimension, avoiding the accuracy loss caused by dimensionality loss when only using traditional Top-k sparse compression in the prior art, and at the same time avoiding the weakening of data updates and slow convergence speed caused by the reduction of gradient magnitude when only using full-dimensional Top-k sparse compression, thereby ensuring the convergence speed while improving the model accuracy.
[0015] According to the sparse compression method for high-performance data parallel DNN training provided by the present invention, the multi-dimensional gradient tensor matrix is compressed using the first sparse compression scheme and the second sparse compression scheme, including:
[0016] For each iteration of model training, the first sparse compression scheme and the second sparse compression scheme are used alternately to compress the multi-dimensional gradient tensor matrix, and each time the second sparse compression scheme is used, the minimum value of the k elements is obtained, and the minimum value is compared with a preset threshold. If the minimum value is less than the preset threshold, the first sparse compression scheme is used in each subsequent iteration to compress the multi-dimensional gradient tensor matrix.
[0017] Based on the preset threshold, this application adopts more full-dimensional Top-k sparse compression in later iterations, realizes the selection of more detailed and practical sparse compression methods, avoids the limitation of training throughput caused by the use of traditional Top-k sparse compression in half of the iterations in the alternating compression scheme, and improves the training throughput.
[0018] According to the sparse compression method for high-performance data parallel DNN training provided by the present invention, the method also includes:
[0019] After each N iterations, the preset threshold is re-determined.
[0020] According to the sparse compression method for high-performance data parallel DNN training provided by the present invention, the method also includes:
[0021] If the absolute value of the difference between the re-determined preset threshold and the preset threshold before re-determination is less than the preset update limit, increase the N by 1; or,
[0022] If the absolute value of the difference between the re-determined preset threshold and the preset threshold before re-determination is greater than N / 5, N is reduced by 1.
[0023] According to the sparse compression method for high-performance data parallel DNN training provided by the present invention, after compressing the multi-dimensional gradient tensor matrix using the first sparse compression scheme and the second sparse compression scheme, the method further includes:
[0024] Get the gradient elements and indices of a compressed multi-dimensional gradient tensor.
[0025] According to the sparse compression method for high-performance data parallel DNN training provided by the present invention, after obtaining the gradient elements and indexes after sparse compression, the method further includes:
[0026] Based on the gradient elements and indices of the compressed multi-dimensional gradient tensor, global gradient synchronization and decompression are performed, and parameter updating is performed.
[0027] In a second aspect, the present application provides a sparse compression device for high-performance data parallel DNN training, comprising:
[0028] The acquisition module is used to obtain the multi-dimensional gradient tensor generated by back propagation in the data parallel DNN training system;
[0029] A sparse module, used to construct a matrix based on the input dimension information and the output dimension information of the multi-dimensional gradient tensor to obtain a multi-dimensional gradient tensor matrix;
[0030] A compression module, configured to compress the multi-dimensional gradient tensor matrix using a first sparse compression scheme and a second sparse compression scheme;
[0031] Wherein, the first sparse compression scheme includes:
[0032] Locally sorting the elements in each dimension of the multi-dimensional gradient tensor matrix;
[0033] In each dimension of the sorted multidimensional gradient tensor, select the k / M largest elements and retain them, and set the remaining elements to zero, where k is the number of gradient elements obtained based on the preset compression rate and the number of all elements of the multidimensional gradient tensor, and M is the number of dimensions;
[0034] The second sparse compression scheme includes:
[0035] Among all the elements of the multi-dimensional gradient tensor matrix, k elements with the largest absolute values are selected and retained, and the remaining elements are set to zero.
[0036] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the sparse compression method for high-performance data-parallel DNN training described in the first aspect or any possible implementation of the first aspect.
[0037] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the sparse compression method for high-performance data-parallel DNN training described in the first aspect or any possible implementation of the first aspect.
[0038] In a fifth aspect, the present application provides a computer program product, which, when running on a processor, enables the processor to execute the sparse compression method for high-performance data-parallel DNN training described in the first aspect or any possible implementation of the first aspect.
[0039] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.
[0040] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:
[0041] (1) Based on the problem that traditional Top-k sparse compression easily loses gradient dimension information and has low GPU utilization, full-dimensional Top-k compression and traditional Top-k sparse compression are selected alternately during gradient sparse compression, thereby retaining all gradient dimension information to ensure higher model accuracy and solving the problem that traditional Top-k sparse compression easily loses gradient dimension information and has low GPU utilization.
[0042] (2) In later iterations, full-dimensional Top-k sparse compression is more often used, which enables a more detailed and practical selection of sparse compression methods. This avoids the limitation on training throughput caused by the use of traditional Top-k sparse compression in half of the iterations in the alternating compression scheme, improves training throughput, and thus reduces communication overhead. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0044] Figure 1 It is a flowchart of a sparse compression method for high-performance data parallel DNN training provided in an embodiment of the present application;
[0045] Figure 2 is a schematic diagram of a second sparse compression scheme provided in an embodiment of the present application;
[0046] Figure 3 is a schematic diagram of a first sparse compression scheme provided in an embodiment of the present application;
[0047] Figure 4 is a schematic diagram of an alternating compression scheme provided in an embodiment of the present application;
[0048] Figure 5 It is a system architecture diagram of a method for sparse compression of a data parallel deep neural network training system deployed across nodes provided by the present application;
[0049] Figure 6 It is a structural schematic diagram of a sparse compression device for high-performance data parallel DNN training provided in an embodiment of the present application;
[0050] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0052] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.
[0053] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.
[0054] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0055] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.
[0056] First, the following contents are introduced:
[0057] A DNN model basically contains multiple neural network layers, each with many model parameters. In order to achieve model convergence (i.e. the model training process reaches a stable state), DNN training will iterate multiple times (epochs) on the dataset, and each epoch contains multiple iterations. In each iteration, the model runs forward and backward propagation to generate the gradient of each layer, thereby updating the model parameters.
[0058] In order to speed up the training of DNN models, data parallel DNN training is usually applied to multiple worker nodes, and each node has a complete copy of the model. In each iteration, the dataset is divided into mini-batches and assigned to different worker nodes. Each worker node uses the assigned mini-batch to calculate the gradient of each layer, and then synchronizes with other worker nodes to jointly update the global model parameters.
[0059] The gradient of each layer of a DNN is usually represented as a multi-dimensional tensor. Basically, different dimensions of the gradient capture different features during training. For example, in computer vision, different dimensions determined by different channels have their own importance. Therefore, more dimensions of the gradient enable the model to capture more features and thus achieve higher convergence accuracy; for example, in translation in natural language processing, more dimensions of the attention head determine lower perplexity (ppl).
[0060] Modern deep learning models rapidly scale from millions of parameters to billions of parameters, and each worker node in a data-parallel DNN training system frequently communicates with other nodes to exchange gradients to update model parameters. Therefore, the increasing communication overhead becomes a major performance bottleneck, and a common way to reduce communication overhead is to apply gradient compression technology. However, simple gradient compression usually leads to very poor convergence performance, so many existing studies focus on high-performance data-parallel DNN training systems, aiming to ensure higher convergence accuracy for distributed DNN training while significantly improving training throughput.
[0061] Many gradient compression algorithms have been proposed to alleviate the communication bottleneck in high-performance data-parallel DNN training systems by reducing the amount of data transmitted during gradient synchronization. Gradient compression techniques are generally divided into three categories: sparse compression, quantization compression, and low-rank compression.
[0062] Sparsification is a key approach to reduce the amount of communication, which exploits the sparsity of the gradient tensor and selects a subset of all its elements (in terms of absolute value) for communication.
[0063] Among different gradient sparsification compression techniques, Top-k sparsification compression (or simply Top-k) has been shown to be one of the most effective compression methods. The traditional Top-k sparsification process selects the k gradient elements with the largest absolute values from all the elements of the gradient, while setting the rest to zero.
[0064] However, Top-k sparsification is a lossy compression that will damage the convergence accuracy of the training model. Although the error feedback method can alleviate the accuracy loss, the traditional Top-k sparsification compression may cause the gradient to lose some important information (such as the dimension of the gradient), making it impossible to recover the convergence accuracy.
[0065] Traditional Top-k sparsification in Pytorch usually uses radix sort to select the largest k elements. Radix sort is a linear sorting algorithm that achieves sorting by processing elements bit by bit and assigning them to different buckets according to the value of each bit; by repeatedly sorting by their significant bits (from the lowest bit to the highest bit), radix sorting achieves the final sorting order. In this way, traditional Top-k sparsification compression requires repeated and global radix sorting of the gradient dimensions, so multiple buckets need to be synchronized multiple times between GPU cores, resulting in a large sparsification compression overhead. At the same time, global sorting requires synchronizing the sorting results between GPU cores, which reduces the degree of parallelism and makes the compression process inefficient, thereby reducing the training throughput.
[0066] In view of the defects of the prior art, the present application provides a sparse compression method and device for high-performance data parallel DNN training, aiming to solve the problem that the traditional Top-k sparse compression method has high communication overhead and reduces the performance of the model accuracy.
[0067] Next, combine Figure 1-Figure 5 The sparse compression method for high-performance data parallel DNN training provided in the embodiments of the present application is introduced.
[0068] Figure 1 is a flow chart of a sparse compression method for high-performance data parallel DNN training provided in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:
[0069] Step 100, obtaining a multi-dimensional gradient tensor generated by back propagation in a data parallel DNN training system;
[0070] Data parallelism allows each node to have a complete copy of the model. In each iteration, the dataset is divided into small batches and assigned to different worker nodes. Each worker node uses the assigned small batches for forward propagation and backpropagation to obtain the multi-dimensional gradient tensor generated during the backpropagation process.
[0071] Step 110, constructing a matrix based on the input dimension information and the output dimension information of the multi-dimensional gradient tensor to obtain a multi-dimensional gradient tensor matrix;
[0072] According to the input dimension M and output dimension N information of the original multidimensional gradient tensor of each network layer, the original gradient tensor is constructed (also called reshaped) into a matrix (M×N), and then matrix-based sparse compression is performed, that is, local Top-k sorting is performed on each input dimension (M) of the original multidimensional gradient tensor.
[0073] Step 120, compressing the multi-dimensional gradient tensor matrix using a first sparse compression scheme and a second sparse compression scheme;
[0074] Among them, the first sparse compression scheme includes:
[0075] Locally sort the elements within each dimension of a multi-dimensional gradient tensor matrix;
[0076] In each dimension of the sorted multidimensional gradient tensor, select the k / M largest elements and retain them, and set the remaining elements to zero, where k is the number of gradient elements obtained based on the preset compression rate and the number of all elements of the multidimensional gradient tensor, and M is the number of dimensions;
[0077] The second sparse compression scheme includes:
[0078] Select the k elements with the largest absolute values from all the elements of the multidimensional gradient tensor matrix and retain them, while setting the rest to zero.
[0079] Among them, the first sparse compression scheme is also called full-dimensional Top-k sparse compression.
[0080] Figure 2 is a schematic diagram of a second sparse compression scheme provided in an embodiment of the present application, such as Figure 2 As shown in the second sparse compression scheme, the gradient tensor is flattened, the elements are globally sorted, and the largest k elements are selected. Figure 2 In one embodiment shown, the gradient tensor has four dimensions, namely D1, D2, D3, and D4, and needs to be compressed to five elements. Ultimately, three elements are selected in D1, two elements are selected in D2, and no elements are selected in D3 and D4, i.e., dimension loss occurs.
[0081] In order to overcome the problem that the second sparse compression scheme easily loses gradient dimension information, the present application uses the first sparse compression scheme and the second sparse compression scheme to compress the multi-dimensional gradient tensor at the same time.
[0082] Figure 3 is a schematic diagram of a first sparse compression scheme provided in an embodiment of the present application, such as Figure 3 As shown in the figure, the traditional Top-k sparse compression is the second sparse compression scheme, and the full-dimensional Top-k sparse compression is the first sparse compression scheme. By comparing the traditional Top-k sparse compression and the full-dimensional Top-k sparse compression, we can find that:
[0083] 1. Traditional Top-k sparse compression causes dimensionality loss, while full-dimensional Top-k sparse compression does not cause dimensionality loss.
[0084] 2. Compared with traditional Top-k sparse compression, the size of the elements selected by full-dimensional Top-k sparse compression is smaller overall, which leads to slower convergence speed.
[0085] Optionally, the k value is calculated by multiplying a preset compression rate by the number of all elements of the multidimensional gradient tensor. For example, if the compression rate is a and the number of all elements of the multidimensional gradient tensor is Numel, then k=a*Numel.
[0086] Therefore, the first sparsification compression scheme and the second sparsification compression scheme are combined to ensure that the gradient tensor obtains elements from each dimension, avoiding the precision loss caused by dimension loss when only using traditional Top-k sparsification compression, and at the same time avoiding the weakening of data update and slow convergence speed caused by the reduction of gradient magnitude when only using full-dimensional Top-k sparsification compression, thus improving the model accuracy while ensuring the convergence speed.
[0087] The sparse compression method for high-performance data parallel DNN training provided by the present invention selects full-dimensional Top-k sparse compression and traditional Top-k sparse compression to be compressed alternately during gradient sparse compression, thereby retaining all gradient dimensional information to ensure higher model accuracy, and solving the problems of traditional Top-k sparse compression that easily loses gradient dimensional information and has low GPU utilization.
[0088] In some embodiments, step 120 specifically includes:
[0089] Step 1201, for each iteration of model training, alternately use the first sparse compression scheme and the second sparse compression scheme to compress the multi-dimensional gradient tensor matrix, and each time the second sparse compression scheme is used, obtain the minimum value of the k elements, and compare the minimum value with a preset threshold. If the minimum value is less than the preset threshold, the first sparse compression scheme is used to compress the multi-dimensional gradient tensor matrix in each subsequent iteration.
[0090] Figure 4 is a schematic diagram of an alternating compression scheme provided in an embodiment of the present application, such as Figure 4 As shown, for each iteration of model training, the first sparse compression scheme and the second sparse compression scheme are used alternately to compress the multi-dimensional gradient tensor matrix, that is, if the current iteration executes the first sparse compression scheme, the next iteration executes the second sparse compression scheme.
[0091] Optionally, each time the second sparsification compression scheme is executed, it is necessary to obtain the minimum value of the selected k elements. Since the model gradually tends to converge in the later iterations and the parameter update becomes slower, the minimum value of the k elements will also become smaller. When the minimum value is less than the preset threshold, it indicates that the traditional Top-k sparsification compression and the full-dimensional Top-k sparsification compression have similar model update speeds. In the subsequent iterations, the first sparsification compression scheme is used to compress the multi-dimensional gradient tensor.
[0092] Optionally, the preset threshold is generally set to 1 / 10 of the minimum value obtained for the first time.
[0093] In some embodiments, the method further comprises:
[0094] After every N iterations, the preset threshold is re-determined.
[0095] Optionally, N is set as the re-estimation cycle. After each N iterations, the preset threshold can be reset to 1 / 10 of the minimum value of the k elements when the second sparse compression scheme was executed last time, and the preset threshold is used in the next N-1 iterations.
[0096] In one embodiment of the present application, the value of N is 15.
[0097] In some embodiments, the method further comprises:
[0098] If the absolute value of the difference between the re-determined preset threshold and the preset threshold before re-determination is less than the preset update limit, N is increased by 1; or,
[0099] If the absolute value of the difference between the re-determined preset threshold and the preset threshold before re-determination is greater than N / 5, N is reduced by 1.
[0100] The present application monitors the changes in the preset threshold during the iteration process and adjusts the re-estimation cycle in real time. Specifically, if the absolute value of the difference between the re-determined preset threshold and the preset threshold before re-determination is less than the preset update limit, N is increased by 1; if the absolute value of the difference between the re-determined preset threshold and the preset threshold before re-determination is greater than N / 5, N is reduced by 1.
[0101] In one embodiment of the present application, the preset update limit is set to 1 / 10 of the preset threshold before re-determination.
[0102] In some embodiments, after step 120, the method further includes:
[0103] Step 130, obtaining the gradient elements and indices of the compressed multi-dimensional gradient tensor.
[0104] In some embodiments, after step 130, the method further includes:
[0105] Based on the gradient elements and indices of the compressed multi-dimensional gradient tensor, global gradient synchronization and decompression are performed, and parameter updates are performed.
[0106] Optionally, after the present application performs sparse compression on the multi-dimensional gradient tensor, it is necessary to synchronize the global gradient through communication, and restore the compressed gradient to the size of the original gradient tensor through the decompression function, fill the lost elements with 0 through the scattering function, change the shape of the tensor to the original shape, and update the parameters through the following formula:
[0107]
[0108] Among them, x i,t and G i,tare the neural network parameters and gradients of the i-th working node in the t-th iteration, η is the learning rate, and K is the number of working nodes.
[0109] Figure 5 : is a schematic diagram of the system architecture of a method for sparse compression of a data parallel deep neural network training system deployed across nodes provided by the present application, such as Figure 5 As shown, in one embodiment of the present application, the method includes:
[0110] 1a. Alternating compression: The first sparse compression scheme and the second sparse compression scheme are used, namely, full-dimensional Top-k sparse compression and traditional Top-k sparse compression. In the later iterations, when the minimum value of the k elements in the traditional Top-k sparse compression is less than 10 times the re-estimation limit N, the subsequent iterations all use full-dimensional Top-k sparse compression;
[0111] 2a. Threshold estimation: Every N cycles, 1 / 10 of the value of the kth largest element is obtained using traditional Top-k sparsification compression as the updated threshold estimate, where N is the threshold update cycle, and N is dynamically adjusted according to the size of the threshold.
[0112] 3a. Gradient aggregation: Three collective communication primitive APIs are provided, namely Allreduce, Allgather, and Allgather_Faster to achieve efficient communication and aggregation of compressed gradients;
[0113] 4a. Residual Gradient Error Feedback: Implemented a grace-like error feedback API, including a memory.compensate function that compensates the locally generated gradient with the error feedback of the previous iteration at each iteration, and a memory.update function that computes the difference between the compensated gradient and the compressed gradient to update the residual and store it in memory.
[0114] Combine the following Figure 6 The sparse compression device for high-performance data parallel DNN training provided by the present invention is described. The sparse compression device for high-performance data parallel DNN training described below and the sparse compression method for high-performance data parallel DNN training described above can refer to each other.
[0115] Figure 6 is a schematic diagram of the structure of a sparse compression device for high-performance data parallel DNN training provided in an embodiment of the present application, such as Figure 6 As shown, the device includes an acquisition module 610, a sparse module 620 and a compression module 630, wherein:
[0116] An acquisition module 610 is used to acquire a multi-dimensional gradient tensor generated by back propagation in a data parallel DNN training system;
[0117] A sparse module 620 is used to construct a matrix based on the input dimension information and the output dimension information of the multi-dimensional gradient tensor to obtain a multi-dimensional gradient tensor matrix;
[0118] A compression module 630, configured to compress the multi-dimensional gradient tensor matrix using a first sparsification compression scheme and a second sparsification compression scheme;
[0119] Among them, the first sparse compression scheme includes:
[0120] Locally sort the elements within each dimension of a multi-dimensional gradient tensor matrix;
[0121] In each dimension of the sorted multidimensional gradient tensor, select the k / M largest elements and retain them, and set the remaining elements to zero, where k is the number of gradient elements obtained based on the preset compression rate and the number of all elements of the multidimensional gradient tensor, and M is the number of dimensions;
[0122] The second sparse compression scheme includes:
[0123] Select the k elements with the largest absolute values from all the elements of the multidimensional gradient tensor matrix and retain them, while setting the rest to zero.
[0124] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.
[0125] Based on the method in the above embodiment, Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, an embodiment of the present application provides an electronic device, which may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the sparse compression method for high-performance data parallel DNN training in the above embodiment.
[0126] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the sparse compression method for high-performance data parallel DNN training described in each embodiment of the present application.
[0127] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the sparse compression method for high-performance data-parallel DNN training in the above embodiment.
[0128] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the sparse compression method for high-performance data parallel DNN training in the above embodiment.
[0129] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.
[0130] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.
[0131] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.
[0132] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.
[0133] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. A sparse compression method for high-performance data parallel DNN training, characterized in that: include: Get the multi-dimensional gradient tensor generated by backpropagation in a data-parallel DNN training system; Constructing a matrix based on the input dimension information and the output dimension information of the multi-dimensional gradient tensor to obtain a multi-dimensional gradient tensor matrix; Compressing the multi-dimensional gradient tensor matrix using a first sparsification compression scheme and a second sparsification compression scheme; Wherein, the first sparse compression scheme includes: Locally sorting the elements in each dimension of the multi-dimensional gradient tensor matrix; In each dimension of the sorted multidimensional gradient tensor, select the k / M largest elements and retain them, and set the remaining elements to zero, where k is the number of gradient elements obtained based on the preset compression rate and the number of all elements of the multidimensional gradient tensor, and M is the number of dimensions; The second sparse compression scheme includes: Among all the elements of the multi-dimensional gradient tensor matrix, k elements with the largest absolute values are selected and retained, and the remaining elements are set to zero.
2. The sparse compression method for high-performance data parallel DNN training according to claim 1, characterized in that: The method of compressing the multi-dimensional gradient tensor matrix using a first sparse compression scheme and a second sparse compression scheme includes: For each iteration of model training, the first sparse compression scheme and the second sparse compression scheme are used alternately to compress the multi-dimensional gradient tensor matrix, and each time the second sparse compression scheme is used, the minimum value of the k elements is obtained, and the minimum value is compared with a preset threshold. If the minimum value is less than the preset threshold, the first sparse compression scheme is used in each subsequent iteration to compress the multi-dimensional gradient tensor matrix.
3. The sparse compression method for high-performance data parallel DNN training according to claim 2, characterized in that: The method further comprises: After each N iterations, the preset threshold is re-determined.
4. The sparse compression method for high-performance data parallel DNN training according to claim 3, characterized in that: The method further comprises: If the absolute value of the difference between the re-determined preset threshold and the preset threshold before re-determination is less than the preset update limit, increase the N by 1; or, If the absolute value of the difference between the re-determined preset threshold and the preset threshold before re-determination is greater than N / 5, N is reduced by 1.
5. The sparse compression method for high-performance data parallel DNN training according to claim 1, characterized in that: After compressing the multi-dimensional gradient tensor matrix using the first sparse compression scheme and the second sparse compression scheme, the method further includes: Get the gradient elements and indices of a compressed multi-dimensional gradient tensor.
6. The sparse compression method for high-performance data parallel DNN training according to claim 5, characterized in that: After obtaining the gradient elements and indexes after sparse compression, the method further includes: Based on the gradient elements and indices of the compressed multi-dimensional gradient tensor, global gradient synchronization and decompression are performed, and parameter updating is performed.
7. A sparse compression device for high-performance data parallel DNN training, characterized in that: include: The acquisition module is used to obtain the multi-dimensional gradient tensor generated by back propagation in the data parallel DNN training system; A sparse module, used to construct a matrix based on the input dimension information and the output dimension information of the multi-dimensional gradient tensor to obtain a multi-dimensional gradient tensor matrix; A compression module, configured to compress the multi-dimensional gradient tensor matrix using a first sparse compression scheme and a second sparse compression scheme; Wherein, the first sparse compression scheme includes: Locally sorting the elements in each dimension of the multi-dimensional gradient tensor matrix; In each dimension of the sorted multidimensional gradient tensor, select the k / M largest elements and retain them, and set the remaining elements to zero, where k is the number of gradient elements obtained based on the preset compression rate and the number of all elements of the multidimensional gradient tensor, and M is the number of dimensions; The second sparse compression scheme includes: Among all the elements of the multi-dimensional gradient tensor matrix, k elements with the largest absolute values are selected and retained, and the remaining elements are set to zero.
8. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the sparse compression method for high-performance data parallel DNN training as described in any one of claims 1-6.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor executes the sparse compression method for high-performance data parallel DNN training as described in any one of claims 1-6.
10. A computer program product, characterized in that When the computer program product runs on a processor, the processor executes the sparse compression method for high-performance data parallel DNN training as described in any one of claims 1-6.
Citation Information
Patent Citations
Gradient compression method for distributed DNN training in edge computing environment
CN113467949A
System for efficient large-scale data distribution in distributed and parallel processing environment
US20210200610A1
Cited By
Cloud-assisted transparent compression distributed model distribution method and distribution network system
CN119652892A
Communication compression method and device for model parallel training and electronic equipment
CN119865533A
Gradient communication compression method and system for shared memory heterogeneous processor
CN121858320A
