Data transmission method, device, electronic device and medium

Abstract

The embodiments of the present application disclose a data transmission method, apparatus, electronic device and computer-readable storage medium, which divide the gradient matrix of the data to be processed into multiple data sets of fixed length; according to the set screening granularity, a preset number of gradient values ​​are screened out from each data set. The screened gradient values ​​are combined into a target data set; based on the position of the gradient value in the gradient matrix, the gradient mask corresponding to each target data set is determined. The target data set and its corresponding gradient mask are transmitted to the opposite device, and the opposite device can restore the target data set to the corresponding position according to the gradient mask, so as to reconstruct the gradient matrix. By setting the screening granularity to achieve compression of the gradient matrix and restoring the target data set based on the gradient mask, the sparsity of the compressed gradient matrix is ​​reduced, and it can be ensured that the compressed gradient matrix is ​​an effective approximation of the original gradient matrix.

Description

Technical Field

[0001] The present application relates to the field of device communication technology, and in particular to a data transmission method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Distributed deep learning model training typically relies on GPU (Graphics Processing Unit) cluster acceleration. Specific parallel algorithms can be categorized as data parallel and model parallel, with data parallel being the most common. In data parallelism, a complete model is hosted on each GPU. Training data is distributed to different GPUs, each of which independently performs forward processing to calculate the loss function, then performs backpropagation to calculate the gradient matrix. Finally, multiple GPUs perform an AllReduce collective communication to synchronize gradients, and the average gradient is used to update the weight matrix. This process repeats until model training is complete.

[0003] The most common implementation of AllReduce collective communication across multiple GPUs is the Ring-AllReduce algorithm. This algorithm uses a ring topology to connect multiple GPUs, breaking data into small blocks and transmitting them in a loop within the ring. Each GPU receives a block of data from the previous GPU in the ring topology and also sends a block of data of the same size to the next GPU in the ring topology. This approach balances the transmit and receive bandwidth of each link. Assuming the number of GPUs is N, the data size is K, and the link bandwidth is B, the overall communication time can be calculated as 2*(N-1)*K / (N*B). When the number of GPUs N is large enough, the overall communication time depends only on the data size K and the link bandwidth B. The link bandwidth B depends on existing external bus and network interconnection technologies and cannot be rapidly increased in the short term. However, the data size K can be effectively reduced through algorithm optimization.

[0004] The gradient matrix of a deep learning model is very large, and many of the gradients have relatively small absolute values, which have little effect on updating the weight matrix. Therefore, researchers have proposed a variety of methods to compress the gradient matrix so that these unimportant gradient values ​​can be removed during communication. In terms of gradient matrix compression, there are two commonly used technical solutions, including fine-grained sparse communication and coarse-grained sparse communication. Fine-grained sparse communication first selects a threshold, then traverses the gradient matrix, retaining only the gradients whose absolute values ​​exceed the threshold, and saves them in the form of key-value pairs of (value, index). Only the set of these key-value pairs is transmitted during communication. Coarse-grained sparse communication first divides the gradient matrix into several fixed-size blocks, generally dividing the entire gradient matrix into hundreds or thousands of blocks, then calculates the L1 norm of each block, and selects a certain proportion of blocks with larger L1 norms for communication.

[0005] However, the sparse matrix calculations used in fine-grained sparse communication are not well suited to GPU architectures, resulting in very high compression-related computational overhead. Furthermore, the number and location of gradients selected by each GPU vary, leading to unbalanced load and low network bandwidth utilization. While coarse-grained sparse communication can address the unsuitability of fine-grained sparse communication for GPU architectures, this technique uses a large granularity for partitioning the gradient matrix and a less detailed approach to partitioning. This results in the compressed gradient matrix not being sufficiently close to the original gradient matrix, and the communication process is prone to loss of precision.

[0006] It can be seen that how to reduce the sparsity of the compressed gradient matrix while ensuring that the compressed gradient matrix is ​​an effective approximation of the original gradient matrix is ​​a problem that those skilled in the art need to solve. Summary of the Invention

[0007] The purpose of the embodiments of the present application is to provide a data transmission method, apparatus, electronic device, and computer-readable storage medium that can reduce the sparsity of a compressed gradient matrix while ensuring that the compressed gradient matrix is ​​a valid approximation of the original gradient matrix.

[0008] To solve the above technical problems, an embodiment of the present application provides a data transmission method, including:

[0009] Divide the gradient matrix of the data to be processed into multiple data sets of fixed length;

[0010] According to the set screening granularity, a preset number of gradient values ​​are screened out from each of the data sets; and the screened gradient values ​​are combined into a target data set;

[0011] Determining a gradient mask corresponding to each target data set based on a position of the gradient value in the gradient matrix;

[0012] The target data set and its corresponding gradient mask are transmitted to the peer device.

[0013] Optionally, filtering out a preset number of gradient values ​​from each of the data sets according to a set filtering granularity includes:

[0014] Each thread of the data set determines the source thread number of the remaining data in the data set according to the set mask calculation method;

[0015] Based on the source thread number, obtaining a corresponding gradient value;

[0016] According to all the gradient values ​​contained in each of the data sets, a preset number of gradient values ​​with the largest values ​​are selected.

[0017] Optionally, the set screening granularity includes screening two gradient values ​​from four adjacent gradient values;

[0018] Accordingly, selecting a preset number of gradient values ​​with the largest values ​​based on all the gradient values ​​included in each data set includes:

[0019] Every four adjacent gradient values ​​are taken as a data set, and the two gradient values ​​with the largest values ​​are screened out from the four gradient values.

[0020] Optionally, determining the gradient mask corresponding to each target data set based on the position of the gradient value in the gradient matrix includes:

[0021] In binary form, the positions of the filtered gradient values ​​in the gradient matrix are set to 1, and the remaining positions in the gradient matrix are set to zero.

[0022] Optionally, transmitting the target data set and its corresponding gradient mask to the opposite device includes:

[0023] According to a set data length, merging the plurality of gradient masks into a gradient mask group;

[0024] The calling thread writes the gradient mask group into the global memory, so that the opposite device reads the gradient mask from the global memory.

[0025] Optionally, it also includes:

[0026] When the data set to be restored and its corresponding target gradient mask are obtained, the target gradient values ​​contained in the data set to be restored are filled in the corresponding positions according to the target gradient mask, and the positions without corresponding data are set to zero to obtain the restored gradient matrix.

[0027] Optionally, it also includes:

[0028] The set screening granularity is dynamically adjusted according to different stages of model training.

[0029] The embodiment of the present application further provides a data transmission device, comprising a dividing unit, a screening unit, a combining unit, a determining unit, and a transmitting unit;

[0030] The division unit is used to divide the gradient matrix of the data to be processed into multiple data sets of fixed length;

[0031] The screening unit is used to screen out a preset number of gradient values ​​from each of the data sets according to a set screening granularity;

[0032] The combining unit is configured to combine the screened gradient values ​​into a target data set;

[0033] The determining unit is configured to determine a gradient mask corresponding to each target data set based on a position of the gradient value in the gradient matrix;

[0034] The transmission unit is configured to transmit the target data set and its corresponding gradient mask to a peer device.

[0035] Optionally, the screening unit includes a determination subunit, an acquisition subunit and a selection subunit;

[0036] The determining subunit is configured to determine, by each thread of the data set, the source thread number of the remaining data in the data set according to a set mask calculation method;

[0037] The acquisition subunit is configured to acquire a corresponding gradient value based on the source thread number;

[0038] The selection subunit is configured to select a preset number of gradient values ​​having the largest values ​​based on all the gradient values ​​included in each of the data sets.

[0039] Optionally, the set screening granularity includes screening two gradient values ​​from four adjacent gradient values;

[0040] Correspondingly, the selection subunit is used to take every four adjacent gradient values ​​as a data set, and select the two gradient values ​​with the largest values ​​from the four gradient values.

[0041] Optionally, the determining unit is configured to set the positions of the filtered gradient values ​​in the gradient matrix to 1 and the remaining positions in the gradient matrix to zero in binary form.

[0042] Optionally, the transmission unit includes a merging subunit and a writing subunit;

[0043] The merging subunit is configured to merge the plurality of gradient masks into a gradient mask group according to a set data length;

[0044] The writing subunit is configured to call a thread to write the gradient mask group into a global memory, so that the peer device can read the gradient mask from the global memory.

[0045] Optionally, a reduction unit is further included;

[0046] The restoration unit is configured to, upon obtaining a data set to be restored and its corresponding target gradient mask, fill in target gradient values ​​contained in the data set to be restored to corresponding positions based on the target gradient mask, and set positions without corresponding data to zero, so as to obtain a restored gradient matrix.

[0047] Optionally, an adjustment unit is further included;

[0048] The adjustment unit is used to dynamically adjust the set screening granularity according to different stages of model training.

[0049] The present invention also provides a data transmission device, including:

[0050] memory for storing computer programs;

[0051] A processor is used to execute the computer program to implement the steps of the above-mentioned data transmission method.

[0052] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data transmission method are implemented.

[0053] It can be seen from the above technical solution that the gradient matrix of the data to be processed is divided into multiple data sets of fixed length; according to the set screening granularity, a preset number of gradient values ​​are screened out from each data set. By setting the screening granularity, the amount of data in the gradient matrix can be effectively reduced, thereby achieving the purpose of compressing the gradient matrix. The screened gradient values ​​are combined into a target data set; based on the position of the gradient values ​​in the gradient matrix, the gradient mask corresponding to each target data set is determined; the gradient mask characterizes the position of each gradient value in the target data set in the gradient matrix. The target data set and its corresponding gradient mask are transmitted to the opposite device, and the opposite device can restore the target data set to the corresponding position according to the gradient mask, so that the gradient matrix can be reconstructed, and the reconstructed gradient matrix has a high degree of approximation to the original gradient matrix. In this technical solution, the gradient matrix is ​​compressed by setting the screening granularity, and the target data set is restored based on the gradient mask, which not only reduces the sparsity of the compressed gradient matrix, but also ensures that the compressed gradient matrix is ​​an effective approximation of the original gradient matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0055] Figure 1 A flowchart of a data transmission method provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of a 2:4 fine-grained structured sparse structure provided in an embodiment of the present application;

[0057] Figure 3 A schematic diagram of gradient selection based on warp shuffle provided in an embodiment of the present application;

[0058] Figure 4 A schematic diagram of calculating a gradient mask provided in an embodiment of the present application;

[0059] Figure 5 A schematic diagram of the structure of a data transmission device provided in an embodiment of the present application;

[0060] Figure 6 A structural diagram of a data transmission device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0062] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not limited to the listed steps or elements and may include steps or elements that are not listed.

[0063] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0064] Next, a data transmission method provided by an embodiment of the present application is introduced in detail. Figure 1 A flowchart of a data transmission method provided in an embodiment of the present application, the method comprising:

[0065] S101: Divide the gradient matrix of the data to be processed into multiple data sets of fixed length.

[0066] The gradient matrix contains the gradient values. The gradient values ​​are used to represent the weights of the deep learning model.

[0067] In an embodiment of the present application, to reduce the amount of data in the gradient matrix, the gradient matrix can be compressed. Furthermore, to make the compressed gradient matrix approximate the original gradient matrix, the gradient matrix can be grouped to obtain data sets. Since the gradient values ​​contained in each data set are relatively similar elements, a portion of the gradient values ​​can be filtered out from each data set to achieve the purpose of compressing the gradient matrix.

[0068] In a specific implementation, a fixed length can be set for data set division. The gradient matrix is ​​divided into fixed lengths to obtain multiple data sets. The fixed length can be set based on actual needs. For example, four gradient values ​​can be set as one data set.

[0069] S102: According to the set screening granularity, a preset number of gradient values ​​are screened out from each data set, and the screened gradient values ​​are combined into a target data set.

[0070] The screening granularity refers to the number of gradient values ​​that are screened from each data set. The screening granularity can be a specific number or a proportional relationship.

[0071] Taking every 4 gradient values ​​as a data set as an example, the screening granularity can be 2, that is, 2 gradient values ​​are screened out from every 4 gradient values ​​as a target data set. The screening granularity can also be 1 / 2, that is, 4*1 / 2=2 gradient values ​​are screened out from every 4 gradient values ​​as a target data set.

[0072] For the sake of distinction, in the embodiment of the present application, the data set of the screened gradient value combination is referred to as the target data set.

[0073] When performing gradient value screening, a preset number of gradient values ​​can be screened out based on the magnitude of the gradient values ​​contained in each data set. In a specific implementation, each thread of a data set can determine the source thread number of the remaining data in its data set according to a set mask calculation method. The source thread number is used to indicate the storage location of the data, and the corresponding gradient value can be obtained based on the source thread number. Based on all the gradient values ​​contained in each data set, a preset number of gradient values ​​with the largest values ​​are selected.

[0074] Taking the set filtering granularity including filtering two gradient values ​​from four adjacent gradient values ​​as an example, each four adjacent gradient values ​​can be regarded as a data set, and the two gradient values ​​with the largest values ​​can be filtered out from the four gradient values.

[0075] like Figure 2 The figure shows a schematic diagram of a 2:4 fine-grained structured sparse structure provided by an embodiment of the present application. Figure 2The white box in the middle can be used to represent the unselected gradient values ​​in the gradient matrix, and the black box is used to represent the selected gradient values ​​in the gradient matrix. The selected gradient values ​​constitute Figure 2 The target dataset pointed to by the arrow on the right is in the form of.

[0076] S103: Determine a gradient mask corresponding to each target data set based on the position of the gradient value in the gradient matrix.

[0077] The gradient mask represents the position of each gradient value in the target dataset in the gradient matrix. In a specific implementation, the selected gradient values ​​can be set to 1 in binary form and the remaining positions in the gradient matrix can be set to 0.

[0078] See also Figure 2 The schematic diagram of 2:4 fine-grained structured sparsity is shown. Figure 2 The rightmost binary value represents the gradient mask corresponding to each target dataset. Taking the first row of data as an example, "01101001" means that the gradient values ​​of the 2nd, 3rd, 5th and 8th bits are selected from the gradient matrix as the target dataset.

[0079] S104: Transmit the target data set and its corresponding gradient mask to the peer device.

[0080] The target data set and its corresponding gradient mask are transmitted to the peer device. The peer device can restore the target data set to the corresponding position according to the gradient mask, so as to reconstruct the gradient matrix. The reconstructed gradient matrix has a high degree of approximation to the original gradient matrix.

[0081] Taking the efficient implementation of fine-grained structured sparse communication on a GPU as an example, in the GPU thread model, a grid consists of multiple blocks, a block consists of multiple threads, and every 32 adjacent threads form a warp. Threads within a warp simultaneously execute the same instruction, operating on different data, a technique known as single instruction multiple threading (SIMT). In the GPU memory hierarchy, off-chip global memory is slower than on-chip shared memory, which in turn is slower than registers. Global memory accesses by threads within a warp that fall within contiguous, 128-byte aligned regions are merged into a single memory transaction.

[0082] In 2:4 fine-grained structured sparse communication, threads within a warp load 32 adjacent data points from the original gradient matrix from global memory, with each thread fetching one data point. Each thread then calculates whether its data point is the largest in absolute value among the two in its warp (each warp consists of four adjacent threads).

[0083] Each thread needs to access all the data in the warp. This computation can usually be achieved by the warp accessing global memory again to load adjacent data, or the warp can first load the data into shared memory to optimize subsequent access.

[0084] Figure 3 A schematic diagram of gradient selection based on warp shuffle provided in an embodiment of the present application is provided. Figure 3 A diagram illustrating a method for exchanging data between threads within a warp without using shared memory, known as the warp shuffle primitive. This method allows threads to directly access registers of neighboring threads, resulting in lower latency and greater memory conservation.

[0085] In the specific implementation, by calling the shfl_xor_sync primitive, the current thread number and laneMask can be bitwise XORed to calculate the source thread number, and then the value of the corresponding variable on the source thread, that is, the gradient value, can be directly obtained. For the 2:4 gradient selection problem, calling the shfl_xor_sync primitive three times with laneMask equal to 1, 2, and 3 can complete the complete exchange of data within the thread bundle. Finally, the thread corresponding to the retained gradient value writes the gradient value to the global memory. Each writing thread can determine the write location by the thread number and the calculated gradient pressure mask. It is also possible to first write the retained data to the shared memory, and then the first half of the threads in the block write the data from the shared memory to the global memory, saving half of the global memory transactions.

[0086] Fine-grained structured sparse communication also requires calculating the gradient mask and transmitting it simultaneously during communication so that the receiving device (the other device) can recover the approximate gradient matrix. The process of calculating the gradient mask can also be efficiently implemented using the warp shuffle primitive.

[0087] Figure 4 A schematic diagram of gradient mask calculation provided by an embodiment of the present application, the shfl_down_sync primitive adds delta to the current thread number, calculates the source thread number, and then directly obtains the gradient value of the corresponding variable on the source thread. When the source thread number exceeds the warp boundary, the variable on the current thread remains unchanged. Figure 2It can be seen that a warp corresponds to a 32-bit or 4-byte gradient mask. Therefore, in the embodiment of the present application, each thread in the warp is allocated a 4-byte variable and initialized to 0. After the gradient selection is completed, each thread corresponding to the retained data sets the corresponding position in this variable. Finally, the shfl_down_sync primitive is called five times with delta equal to 16, 8, 4, 2, and 1 to perform parallel reduction, and the reduction operation is set to bitwise OR. Finally, thread number 0 calculates the complete gradient mask and writes it to shared memory. Then, some threads in the block write the gradient mask from shared memory to global memory.

[0088] Taking 2:4 fine-grained structured sparse processing as an example, 2 gradient values ​​are selected from every 4 gradient values ​​as the target data set. When the target data set and its corresponding gradient mask are transmitted to the peer device, multiple gradient masks can be merged into a gradient mask group according to the set data length; the calling thread writes the gradient mask group to the global memory so that the peer device can read the gradient mask from the global memory.

[0089] One thread can process 32-bit data, and the data length can be set to 32 bits. When a target data set contains 2-bit data, 16 target data sets can be merged into one gradient mask group.

[0090] When the peer device restores the approximate gradient matrix, some threads in the block first load the gradient mask corresponding to the block into shared memory. Then, threads in the warp retrieve the corresponding gradient mask from shared memory. Since all threads in the warp access the same 4-byte gradient mask, they can be merged into a single shared memory transaction using broadcast. Finally, based on the gradient mask, the warp fills the corresponding locations with the target gradient values ​​contained in the dataset to be restored and sets the locations without corresponding data to zero to obtain the restored gradient matrix. The restored gradient matrix can then be written to global memory. The process of retrieving the corresponding data can also reduce global memory transactions by using shared memory.

[0091] It can be seen from the above technical solution that the gradient matrix of the data to be processed is divided into multiple data sets of fixed length; according to the set screening granularity, a preset number of gradient values ​​are screened out from each data set. By setting the screening granularity, the amount of data in the gradient matrix can be effectively reduced, thereby achieving the purpose of compressing the gradient matrix. The screened gradient values ​​are combined into a target data set; based on the position of the gradient values ​​in the gradient matrix, the gradient mask corresponding to each target data set is determined; the gradient mask characterizes the position of each gradient value in the target data set in the gradient matrix. The target data set and its corresponding gradient mask are transmitted to the opposite device, and the opposite device can restore the target data set to the corresponding position according to the gradient mask, so that the gradient matrix can be reconstructed, and the reconstructed gradient matrix has a high degree of approximation to the original gradient matrix. In this technical solution, the gradient matrix is ​​compressed by setting the screening granularity, and the target data set is restored based on the gradient mask, which not only reduces the sparsity of the compressed gradient matrix, but also ensures that the compressed gradient matrix is ​​an effective approximation of the original gradient matrix.

[0092] For some deep learning models, even a 50% sparsity setting will still retain many unimportant gradients. In this case, a smaller sparsity setting should be used based on the characteristics of the deep learning model. Alternatively, different sparsities can be used at different stages of model training. This means dynamically adjusting the filtering granularity based on the different stages of model training.

[0093] For example, in the early stages of model training, using a larger sparsity can improve approximation and rapidly reduce the loss function. Subsequently, using a smaller sparsity can reduce communication traffic and increase the speed of model iteration. However, all of these implementations must maintain the characteristics of structured sparsity to achieve efficient implementation on GPUs.

[0094] Figure 5 A schematic structural diagram of a data transmission device provided in an embodiment of the present application, comprising a dividing unit 51, a screening unit 52, a combining unit 53, a determining unit 54, and a transmitting unit 55;

[0095] A division unit 51 is used to divide the gradient matrix of the data to be processed into multiple data sets of fixed length;

[0096] A screening unit 52 is used to screen out a preset number of gradient values ​​from each data set according to a set screening granularity;

[0097] a combining unit 53, configured to combine the screened gradient values ​​into a target data set;

[0098] a determination unit 54 for determining a gradient mask corresponding to each target data set based on the position of the gradient value in the gradient matrix;

[0099] The transmission unit 55 is configured to transmit the target data set and its corresponding gradient mask to the peer device.

[0100] Optionally, the screening unit includes a determination subunit, an acquisition subunit, and a selection subunit;

[0101] Determine a subunit, where each thread of the data set determines the source thread number of the remaining data in the data set according to the set mask calculation method;

[0102] The acquisition subunit is used to obtain the corresponding gradient value based on the source thread number;

[0103] The selection subunit is used to select a preset number of gradient values ​​with the largest values ​​based on all the gradient values ​​contained in each data set.

[0104] Optionally, the set screening granularity includes screening two gradient values ​​from four adjacent gradient values;

[0105] Accordingly, the selection subunit is used to take every four adjacent gradient values ​​as a data set, and to filter out the two gradient values ​​with the largest values ​​from the four gradient values.

[0106] Optionally, the determining unit is configured to set the positions of the filtered gradient values ​​in the gradient matrix to 1 and the remaining positions in the gradient matrix to zero in binary form.

[0107] Optionally, the transmission unit includes a merging subunit and a writing subunit;

[0108] A merging subunit, configured to merge multiple gradient masks into a gradient mask group according to a set data length;

[0109] The writing subunit is used to call the thread to write the gradient mask group into the global memory so that the peer device can read the gradient mask from the global memory.

[0110] Optionally, a reduction unit is further included;

[0111] The restoration unit is used to fill the target gradient values ​​contained in the to-be-restored dataset into the corresponding positions according to the target gradient mask after obtaining the to-be-restored dataset and its corresponding target gradient mask, and to set the positions without corresponding data to zero to obtain the restored gradient matrix.

[0112] Optionally, an adjustment unit is further included;

[0113] The adjustment unit is used to dynamically adjust the set filtering granularity according to different stages of model training.

[0114] Figure 5 The description of the features in the corresponding embodiment can be found in Figure 1The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0115] It can be seen from the above technical solution that the gradient matrix of the data to be processed is divided into multiple data sets of fixed length; according to the set screening granularity, a preset number of gradient values ​​are screened out from each data set. By setting the screening granularity, the amount of data in the gradient matrix can be effectively reduced, thereby achieving the purpose of compressing the gradient matrix. The screened gradient values ​​are combined into a target data set; based on the position of the gradient values ​​in the gradient matrix, the gradient mask corresponding to each target data set is determined; the gradient mask characterizes the position of each gradient value in the target data set in the gradient matrix. The target data set and its corresponding gradient mask are transmitted to the opposite device, and the opposite device can restore the target data set to the corresponding position according to the gradient mask, so that the gradient matrix can be reconstructed, and the reconstructed gradient matrix has a high degree of approximation to the original gradient matrix. In this technical solution, the gradient matrix is ​​compressed by setting the screening granularity, and the target data set is restored based on the gradient mask, which not only reduces the sparsity of the compressed gradient matrix, but also ensures that the compressed gradient matrix is ​​an effective approximation of the original gradient matrix.

[0116] Figure 6 A structural diagram of a data transmission device provided in an embodiment of the present application, such as Figure 6 As shown, the data transmission device includes: a memory 20 for storing computer programs;

[0117] The processor 21 is configured to implement the steps of the data transmission method in the above embodiment when executing a computer program.

[0118] The data transmission device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0119] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0120] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the data transmission method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include but is not limited to gradient matrices, screening granularity, target data sets, gradient masks, etc.

[0121] In some embodiments, the data transmission device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0122] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the data transmission device, and may include more or fewer components than shown in the figure.

[0123] It is understandable that if the data transmission method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0124] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data transmission method are implemented.

[0125] The functions of the various functional modules of the computer-readable storage medium described in the embodiment of the present invention can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment and will not be repeated here.

[0126] The above describes in detail a data transmission method, device, electronic device, and computer-readable storage medium provided by the embodiments of the present application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between the various embodiments can be referenced to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method section.

[0127] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0128] The above is a detailed introduction to a data transmission method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A data transmission method, characterized in that: include: Divide the gradient matrix of the data to be processed into multiple data sets of fixed length; According to the set screening granularity, a preset number of gradient values ​​are screened from each of the data sets; the screened gradient values ​​are combined into a target data set; and the set screening granularity is dynamically adjusted according to different stages of model training; Determining a gradient mask corresponding to each target data set based on a position of the gradient value in the gradient matrix; Transmitting the target data set and its corresponding gradient mask to a peer device; The step of filtering out a preset number of gradient values ​​from each of the data sets according to the set filtering granularity includes: Each thread of the data set determines the source thread number of the remaining data in the data set according to the set mask calculation method; the source thread number is used to indicate the storage location of the data; Based on the source thread number, obtaining a corresponding gradient value; Selecting a preset number of gradient values ​​with the largest values ​​according to all the gradient values ​​included in each data set; The set screening granularity includes screening two gradient values ​​from four adjacent gradient values; Accordingly, selecting a preset number of gradient values ​​with the largest values ​​based on all the gradient values ​​included in each data set includes: Every four adjacent gradient values ​​are taken as a data set, and the two gradient values ​​with the largest values ​​are screened out from the four gradient values.

2. The data transmission method according to claim 1, wherein: Determining the gradient mask corresponding to each target data set based on the position of the gradient value in the gradient matrix includes: In binary form, the positions of the filtered gradient values ​​in the gradient matrix are set to 1, and the remaining positions in the gradient matrix are set to zero.

3. The data transmission method according to claim 1, wherein: The transmitting the target data set and its corresponding gradient mask to the opposite end device includes: According to a set data length, merging the plurality of gradient masks into a gradient mask group; The calling thread writes the gradient mask group into the global memory, so that the opposite device reads the gradient mask from the global memory.

4. The data transmission method according to claim 1, wherein: Also includes: When the data set to be restored and its corresponding target gradient mask are obtained, the target gradient values ​​contained in the data set to be restored are filled in the corresponding positions according to the target gradient mask, and the positions without corresponding data are set to zero to obtain the restored gradient matrix.

5. A data transmission device, characterized in that: It includes a division unit, a screening unit, a combination unit, a determination unit and a transmission unit; The division unit is used to divide the gradient matrix of the data to be processed into multiple data sets of fixed length; The screening unit is used to screen out a preset number of gradient values ​​from each of the data sets according to a set screening granularity; The combining unit is configured to combine the screened gradient values ​​into a target data set; The determining unit is configured to determine a gradient mask corresponding to each target data set based on a position of the gradient value in the gradient matrix; The transmission unit is configured to transmit the target data set and its corresponding gradient mask to a peer device; The screening unit includes a determination subunit, an acquisition subunit, and a selection subunit; the determination subunit is used for each thread of the data set to determine the source thread number of the remaining data in the data set according to the set mask calculation method; The source thread number is used to indicate the storage location of the data; the acquisition subunit is used to acquire the corresponding gradient value based on the source thread number; the selection subunit is used to select a preset number of gradient values ​​with the largest values ​​based on all the gradient values ​​contained in each data set; the set screening granularity includes screening two gradient values ​​from four adjacent gradient values; accordingly, the selection subunit is used to treat each four adjacent gradient values ​​as a data set and screen out the two gradient values ​​with the largest values ​​from the four gradient values; It also includes an adjustment unit; the adjustment unit is used to dynamically adjust the set screening granularity according to different stages of model training.

6. A data transmission device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the data transmission method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data transmission method according to any one of claims 1 to 4.

Images