Hardware-oriented high-precision gradient compression method

Abstract

The invention relates to the field of data processing, and discloses a hardware-oriented high-precision gradient compression method, which comprises the following steps that gradient data are loaded by blocks, and each data block comprises a plurality of floating-point numbers; separating a sign bit, an index bit and a mantissa bit of each floating-point number, cutting off the mantissa bit, and retaining a high-order part of the mantissa bit; performing fixed value prediction or block average value prediction on the index bit, calculating a residual error, and performing inversion processing on a negative residual error; zero columns in the residual matrix are eliminated through zero mapping coding, and a compact non-zero matrix is generated; and finally, packaging the sign bit, the truncated mantissa bit, the zero mapping and the non-zero matrix to form a compressed output stream. According to the method, through the steps of separation, prediction, residual calculation, zero coding and the like, the storage and transmission requirements of gradient data are remarkably reduced, meanwhile, the precision of model training is kept, and the method is suitable for implementation of hardware such as an FPGA or a GPU and has the advantages of being low in resource consumption, high in compression rate and suitable for various deep learning models.

Description

Technical Field

[0001] The invention relates to the technical field of data processing, and in particular to a hardware-oriented high-precision gradient compression method. Background Art

[0002] As the scale of deep learning models continues to expand, distributed training has gradually become the main strategy for training large neural networks. In distributed training, data parallelism is a common method. The model is copied to multiple devices (i.e., worker nodes), and the data set is divided into several subsets and assigned to different devices for training. Each device performs gradient calculations based on processing its own data subset and updates the model parameters to global synchronization after each iteration. However, as the number of model parameters and devices increases, the amount of data communication between workers and aggregators also increases. The communication bandwidth becomes a key bottleneck in distributed training, which seriously limits the training efficiency of the system.

[0003] To alleviate this bottleneck, many studies have proposed gradient compression techniques to reduce the amount of data transmitted during model training. At present, compression technologies are mainly divided into two categories: lossy compression and lossless compression. Lossy compression reduces the amount of transmitted data by reducing data precision (such as quantization) or discarding unimportant information (such as sparsification), thereby significantly improving the compression ratio. However, these methods often bring a certain loss of accuracy, which may affect the convergence and final performance of the model. Although lossy compression methods can reduce the amount of communication, their computational complexity is high, especially when implemented on training devices such as GPUs, which may cause competition for hardware resources with the gradient calculation process, thereby affecting the overall performance of the system. In addition, system-level optimization methods such as ring reduction can improve data transmission efficiency by exploiting computation-communication overlap and optimizing network topology, which makes lossy compression methods not always bring significant speed advantages in practical applications.

[0004] Lossless compression technology reduces the amount of data by predicting and encoding while maintaining data accuracy. Common general-purpose lossless compression algorithms include LZ4 and Zstd. Some lossless compression algorithms are designed for floating-point numbers and use a two-stage compression process. First, the differential residual is generated through prediction technology, and then the residual is encoded. This type of algorithm can achieve higher accuracy, but its entropy coding process is usually computationally complex and difficult to implement through parallelization, resulting in limited hardware acceleration effects. In addition, most existing lossless compression methods are designed for structured scientific data sets, such as biological and medical data. The data dimensions and patterns are relatively fixed and suitable for specific prediction techniques. However, the gradient data generated during deep learning training comes from a variety of different models. The size and structure of the gradient are not fixed, and there is a lack of specific patterns, which makes it impossible to effectively apply existing lossless compression methods to gradient data.

[0005] In order to improve compression efficiency, some studies have introduced FPGA hardware to accelerate the compression process, especially implementing traditional compression algorithms and lossy compression algorithms on network interface cards in order to increase communication speed. However, due to the high complexity of existing algorithms, it is usually difficult to achieve high throughput on hardware, resulting in inefficient use of hardware resources. Especially in typical data center scenarios, gradient compression algorithms with high compression ratios are often difficult to scale linearly with the model size, but will introduce large computational overhead, which is not conducive to the scalability of the training process.

[0006] In summary, existing gradient compression methods have certain deficiencies in computational complexity, hardware adaptability, compression efficiency, and model accuracy. Therefore, there is an urgent need for a gradient compression method that can balance high accuracy and low complexity, can be efficiently implemented in hardware, and can reduce the amount of data transmission while maintaining the accuracy and convergence of model training. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides a hardware-oriented high-precision gradient compression method, which achieves efficient gradient data compression with low complexity while maintaining the accuracy of model training, reduces the communication overhead in distributed training and improves the efficiency of hardware implementation.

[0008] To achieve the above objectives, the present invention is implemented by the following technical scheme: a hardware-oriented high-precision gradient compression method, comprising the following steps: Divide the gradient data into multiple data blocks, each data block contains a number of floating point data; Separate the sign bit, exponent bit and mantissa bit of each floating point data, where the mantissa bit is truncated to remove the low bit and only retain the high bit; Apply the prediction method to the exponent bit to generate a predicted value, and calculate the residual between the exponent bit and the predicted value; Perform zero encoding on the residual to generate a zero map to eliminate zero columns; The sign bit, truncated mantissa bits, zero mapping, and non-zero residual data are packed into a compressed output stream.

[0009] Preferably, the step of applying a prediction method to the exponent bit to generate a predicted value comprises: Use fixed values ​​as predicted values, where the fixed values ​​are based on the results of statistical analysis of the data distribution.

[0010] Preferably, the fixed prediction value is 108, and the fixed prediction value is used to match the exponent range of the gradient data in a single-precision floating point number.

[0011] Preferably, the step of applying a prediction method to the exponent bit to generate a predicted value further comprises: Use the average index value of the previous data block as the predicted value.

[0012] Preferably, the average index value of the previous data block is stored in an external register, and the average value is used for prediction at the beginning of each data block.

[0013] Preferably, the step of calculating the residual between the exponent bit and the predicted value comprises: The exponent bits are subtracted from the predicted value to produce a residual.

[0014] Preferably, the step of calculating the residual between the exponent bit and the predicted value further comprises: checking the most significant bit of the residual; If the residual is negative, the residual is processed by two's complement to convert the negative value into a positive value close to zero; If the residual is non-negative, it is preserved in its original form.

[0015] Preferably, the step of performing zero encoding comprises: Perform a bitwise OR operation on each column of the residual matrix to generate a zero map to indicate whether each column contains a non-zero value; Based on the zero map, zero columns are removed and non-zero columns are transposed into a matrix of the specified format to simplify storage and transmission.

[0016] Preferably, the compressed output stream includes: A flag bit used to indicate the selected prediction method; The generated zero map is used to restore the zero column during decoding; Non-zero column matrix data, containing compressed non-zero exponential residual values; The mantissa and sign bits are truncated to ensure that the gradient direction information and the main precision are preserved.

[0017] The present invention also provides a hardware-oriented high-precision gradient compression device, comprising: Block loading module, used to split gradient data into data blocks; A prediction module, for applying a fixed value or a block average prediction method to the exponent bit to generate a prediction value and calculate a residual; An encoding module, used to generate a zero map for the residual and remove zero columns to form a non-zero matrix; The output module is used to pack the sign bit, truncated mantissa bits, zero map and non-zero matrix data into a compressed output stream.

[0018] The present invention provides a hardware-oriented high-precision gradient compression method. It has the following beneficial effects: 1. The present invention realizes effective data compression by performing differential processing on the sign bit, exponent bit and mantissa bit of floating point numbers. The truncation of mantissa bits and residual calculation of exponent bits reduce data redundancy, greatly reduce the transmission and storage requirements of gradient data, and is suitable for large-scale distributed deep learning training.

[0019] 2. When compressing gradient data, the present invention adopts a method combining fixed value prediction and block average prediction, so that the residual calculation of the exponential bit can keep the original information of the data as much as possible, avoiding the influence of information loss on the accuracy of model training, thereby maintaining the accuracy and convergence of the model at a high compression rate.

[0020] 3. The present invention adopts pipeline and parallel processing methods in hardware design, and gradually implements operations such as separation, prediction, residual calculation and zero coding through a modular structure, so that the method can be efficiently executed on hardware platforms such as FPGA or GPU, effectively reducing the implementation complexity and resource consumption.

[0021] 4. The present invention introduces zero mapping coding in the compression process to remove zero columns in the residual matrix, significantly reducing the amount of data stored and transmitted. Zero coding is implemented through simple bit mapping, with low consumption of hardware resources, while improving the compactness of compression and transmission efficiency.

[0022] 5. The present invention loads and processes gradient data in blocks, each of which contains multiple floating-point numbers, so that the processing can be parallelized. Data block division not only speeds up the processing flow, but also makes the method suitable for implementation in high-throughput hardware environments, such as data centers and large distributed training platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the method flow of the present invention; Figure 2 Schematic diagrams of gradient data distribution and exponential distribution of different deep learning models of the present invention, where (a), (b), and (c) are the gradient data distributions of ResNet50, ViT, and Bert, respectively, and (d) is the normalized exponential distribution of the gradient data of the four models; Figure 3 Schematic diagram of the encoding process of the algorithm of the present invention; Figure 4 It is a schematic diagram of the hardware architecture of the present invention; Figure 5 Schematic diagram of the change in training accuracy of different models of the present invention when the compression method (HHGC) of the present invention and the baseline method (Baseline) are applied, where (a) ResNet50, (b) VGG16; Figure 6 It is a schematic diagram of the device structure of the present invention.

[0024] Among them, 10, block loading module; 20, prediction module; 30, encoding module; 40, output module. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Please see attached Figure 1 -Attached Figure 5 , an embodiment of the present invention provides a hardware-oriented high-precision gradient compression method, which is suitable for gradient compression scenarios in large-scale distributed deep learning training. The present invention achieves high-precision gradient compression by processing gradient data in blocks, separating and truncating floating-point numbers, calculating the residual of exponent bits, and encoding zeros, taking into account both compression efficiency and hardware friendliness. The following is a detailed description of the specific implementation method of the method.

[0027] S1. Divide the gradient data into multiple data blocks The present invention realizes block compression of data by dividing the input gradient data into multiple data blocks and performing subsequent processing steps in units of fixed-size data blocks, thereby effectively improving the parallelism and efficiency of hardware implementation.

[0028] In this embodiment, the input gradient data is first divided into multiple data blocks, each data block containing a number of floating point data. In general, each data block contains 32 floating point numbers in FP32 format, specifically 1024 bits. The size of this data block is selected to meet the standard bit width of mainstream hardware processing, such as the cache and pipeline characteristics of computing hardware such as FPGA and GPU, so it can effectively support efficient parallel data processing. In the block process, the input gradient data stream is processed in a sequential read manner, and each 32 floating point numbers are divided into a data block in turn.

[0029] Specifically, the floating point data contained in each data block is represented by IEEE 754 single-precision floating point (FP32), which consists of 1 sign bit, 8 exponent bits, and 23 mantissa bits. Floating point data has an important influence on the update of model parameters during the training process, so the present invention separates and compresses each part of each FP32 floating point data in each data block.

[0030] In a possible implementation, the size of each data block can be appropriately adjusted according to hardware conditions, but in order to facilitate standardized processing, it is preferred to select a data block size of 32 FP32 floating point numbers.

[0031] As an option, if the size of the gradient data set is not an integer multiple of the data block size, the remaining data is directly output as boundary data to ensure the integrity of the input data. The boundary data is directly retained in the final compressed data stream without block processing, thereby reducing the complexity of boundary processing.

[0032] In some embodiments, boundary data may be directly written to the output stream during the compression process without undergoing separation and truncation operations, thereby ensuring that the data is not affected during block processing.

[0033] In a possible implementation, in order to support the synchronous processing of multiple data blocks, this embodiment designs a multi-stage pipeline structure, and each pipeline stage is responsible for the reading, segmentation and storage operations of a data block. Through this pipeline design, multiple data blocks can be read and processed in parallel, thereby further improving the processing efficiency of hardware implementation.

[0034] In some embodiments, the synchronous control of the pipeline is triggered by a clock signal, so that the reading and storing process of each data block can be effectively coordinated with the calculation steps.

[0035] In one possible implementation, in order to better adapt to the training requirements of different deep learning models, an adaptive data block partitioning scheme can be designed. For example, a smaller block size is used when the data volume is small, and a larger block size is used when the data volume is large, so as to achieve better hardware resource utilization efficiency under different data scales.

[0036] S2, separate the sign bit, exponent bit and mantissa bit for each floating point data In step S2, the segmented data blocks are processed one by one by separating the floating point numbers to extract the sign bit, exponent bit and mantissa bit of the single-precision floating point number, and differentiated processing is performed according to the importance of each part of the data to the model training accuracy. Among them, the mantissa bits are truncated to a part of the low bit, and only the high bit is retained to reduce data redundancy and storage pressure. This step lays the foundation for the subsequent compression step.

[0037] In this embodiment, the floating point numbers in each data block are represented by the IEEE 754 single-precision floating point (FP32) format. As mentioned above, the FP32 format consists of three parts: 1 sign bit, 8 exponent bits, and 23 mantissa bits. In general, the sign bit is used to indicate the positive or negative value of the floating point number, the exponent bit determines the numerical range of the floating point number, and the mantissa bit is used to indicate the precision of the floating point number.

[0038] In a possible implementation, the present embodiment first separates the data structure of each FP32 floating point number, and extracts the sign bit, exponent bit, and mantissa bit in sequence. Specifically, the separation operation is implemented through the shift and mask operation of the hardware pipeline, and three independent data streams are generated based on the bit width of the sign bit, exponent bit, and mantissa bit.

[0039] For the sign bit, generally, the amount of data of the sign bit is small and directly affects the update direction of the model parameters, so the sign bit does not participate in the compression process, but is directly retained. In some embodiments, the sign bit and the mantissa bit are directly packaged and stored in subsequent steps without further processing.

[0040] As for the exponent bit, the exponent bit is an important component of the floating point number and directly determines the numerical value range of the floating point number. In this embodiment, the exponent bit is extracted separately and used for prediction and residual calculation.

[0041] As an option, the exponent bits can be used to generate a residual through a subtraction operation in a subsequent step, further reducing their data redundancy.

[0042] As for the mantissa, since the mantissa accounts for more than 72% of the total bit width of FP32, it is the main target of data compression. In this embodiment, by truncating the low bits of the mantissa, only the highest 9-bit data stream is retained. In general, the low bits of the mantissa contribute less to the accuracy of model training, while the high bits retain the main information of the floating point number.

[0043] In a possible implementation, the truncation operation is implemented through bit shifting and masking, and the data from the 10th to the 23rd bits of the mantissa are truncated, leaving only the first 9 bits.

[0044] As an option, after the mantissa is truncated, the remaining mantissa bit data and the sign bit enter a subsequent compression processing link to improve the compactness of the data stream.

[0045] In some embodiments, the truncation range of the mantissa can be appropriately adjusted according to specific hardware requirements, for example, more high bits can be reserved in a specific scenario to further enhance the expression capability of the gradient.

[0046] Specifically, the separation process of FP32 format can be expressed as: The gradient data for a given input is:

[0047] The sign bit is , the exponent is , the last digit is By separating the operations, the sign bit can be extracted separately. , exponent bit , and the high digit of the mantissa .

[0048] In one implementation, the data parts of multiple floating point numbers can be extracted in parallel through pipeline design. In hardware implementation, the operation of separating the sign bit, exponent bit and mantissa bit is implemented by bit masking and shifting, for example: The sign bit is extracted by bitwise ANDing with the mask "10000000 00000000 00000000 00000000"; The exponent bits are extracted by bitwise ANDing with the mask "01111111 10000000 00000000 00000000" and then shifting right 23 bits; The high-order bits of the mantissa are extracted by bitwise ANDing with the mask "00000000 01111111 11110000 00000000" and then shifting right 14 bits.

[0049] Through the above operations, this embodiment can complete the separation and mantissa truncation operations of FP32 floating-point numbers with low hardware overhead, providing a streamlined input data stream for subsequent compression steps.

[0050] S3. Apply the prediction method to the exponent bit to generate the predicted value, and calculate the residual between the exponent bit and the predicted value This step further reduces the data redundancy of the exponent bit by predicting the exponent bit of the floating point number and calculating the residual, and provides a more compact data structure for the subsequent zero encoding step. The exponent bit directly determines the numerical range of the floating point number, so its accuracy is of great significance to the convergence of model training. The present invention combines the two methods of fixed value prediction and block-level average prediction to flexibly process the exponent bit, generate a residual value close to zero through subtraction, and specially encode the negative residual to adapt it to a hardware-friendly data structure.

[0051] In this embodiment, the separated floating point number exponent bits are first predicted to generate the corresponding predicted value. There are two methods for predicting the exponent bits: fixed value prediction and block average prediction. Specifically, the fixed value prediction selects a fixed median as the predicted value based on the statistical characteristics of the gradient data distribution; while the block average prediction calculates the average exponent value of the previous data block to generate the predicted value of the current data block.

[0052] As an option, the fixed value used in the fixed value prediction is obtained by data analysis, and the value of the gradient data is usually distributed between -0.001 and 0.001. According to the formula of IEEE 754 single-precision floating point numbers, when the gradient value is in the above interval, the exponent bits are mostly distributed between 90 and 127. Therefore, selecting the median value 108 of the interval as the fixed prediction value can significantly reduce the residual range of the exponent bit.

[0053] Specifically, the present invention analyzes the gradient data distribution (such as ResNet50, VGG16, Bert and ViT) of different models (such as Figure 2 As shown), it can be concluded that the rationality of the exponent bit prediction and residual calculation of the present invention is reasonable.

[0054] In the present invention, the distribution characteristics of the gradient data provide an important reference for the solution. Figure 2 The gradient data distribution of the ResNet50, ViT and Bert models shown in (a), (b) and (c) shows that most of the gradient values ​​are concentrated in a range close to zero, and their absolute values ​​do not exceed 0.001. This concentrated distribution indicates that the low bits of the mantissa contribute little to the accuracy of model training. Therefore, the present invention truncates the mantissa of the floating-point number, retaining only the highest 9 bits of data and truncating the remaining low-bit data. In general, the truncated mantissa data can still retain the main information of the gradient, thereby achieving a good balance between compression rate and accuracy.

[0055] also, Figure 2 (d) shows the normalized exponential distribution of gradient data of multiple deep learning models. The analysis results show that the exponential bits of the gradient data of different models are mainly concentrated in the range of 90 to 127. Based on this, the fixed value prediction of the present invention selects 108 as the prediction value, which is located in the middle of the range of 90 to 127, thereby effectively reducing the amplitude of the exponential bit residual and facilitating subsequent zero coding processing. The fixed value prediction method is simple and applicable to a variety of models, and can reduce computing resource usage in hardware implementation.

[0056] In one possible implementation, for block average prediction, all exponent bit data of the previous data block are first read from the external register, and then the average value is calculated as the predicted value of the current data block. In general, this method can adapt to the distribution characteristics of different gradient data and is suitable for data sets with large distribution changes.

[0057] Next, the exponent is subtracted from the predicted value to generate a residual value. In general, the residual value is calculated as follows:

[0058] As an option, in order to make the residual value more suitable for hardware processing, this embodiment performs an inversion operation on the negative residual. Specifically, the positive or negative value of the residual is determined by checking the most significant bit (sign bit) of the residual: If the highest bit of the residual is 1, indicating a negative value, the remaining bits of the residual are bitwise inverted to complete the inversion operation; If the highest bit of the residual is 0, indicating a non-negative value, the original form of the residual is retained directly.

[0059] In one possible implementation, the residual value after complement coding is further normalized to ensure that the negative residual can be mapped into a positive range close to zero. This processing method can effectively reduce the complexity of hardware implementation and provide a more compact data structure for the subsequent zero encoding step.

[0060] In some embodiments, the result of residual calculation is generated step by step through pipeline processing. For example, for the exponent bit data of 32 floating point numbers, the reading of the prediction value, the subtraction operation and the complement processing of the negative residual can be completed simultaneously in each pipeline stage. Specifically, in the scenario of fixed prediction value, all exponent bits are subtracted from the fixed value 108 at the same time; in the scenario of block average prediction, the calculation of the average value can be completed in parallel with the processing of the previous data block, thereby reducing the prediction delay.

[0061] Through the above processing, the present invention can compress the exponent bit into a residual value close to zero, while maintaining the main numerical characteristics of the floating point number, providing optimized input data for the subsequent compression step. The overall design greatly reduces the complexity of hardware implementation while ensuring high precision.

[0062] S4. Perform zero encoding on the residual to generate a zero map to eliminate zero columns This step achieves further data compression by zero-coding the exponential bit residual. In the zero-coding process, a zero map is generated to identify the zero columns in the residual matrix, thereby effectively eliminating redundant data. The zero-coding step not only reduces the demand for data storage and transmission, but also provides a data structure that is easy to operate for subsequent hardware implementation. Specifically, the present invention uses a bitwise OR operation to generate a zero map, and combines the transposition operation of the matrix to compactly store non-zero columns, which significantly improves the compression efficiency.

[0063] In this embodiment, the exponent bit residual values ​​are first arranged into a 32×8 matrix, where 32 rows correspond to 32 floating point numbers in each data block, and 8 columns correspond to each bit of the exponent bit residual. In general, most columns of the residual matrix may contain only zero values, so these columns can be identified by zero mapping and directly removed in subsequent processing.

[0064] Specifically, an 8-bit zero map is generated by performing a bitwise OR operation on each column of the matrix. Alternatively, each bit in the zero map is used to identify whether the corresponding column is a zero value, for example: If all bits in a column are zero, the corresponding bit in the zero map is 0; If there is a non-zero value in a column, the corresponding bit in the zero map is 1.

[0065] In a possible implementation, the bitwise OR operation process is implemented by hardware, and the result of the bitwise OR operation of each column can be generated within a single clock cycle, thereby supporting efficient pipeline processing.

[0066] After generating the zero map, this embodiment further performs zero column removal and non-zero column storage operations on the residual matrix. Generally, only the columns whose corresponding bits in the zero map are 1 are retained, and these non-zero columns are transposed. Through transposition, the non-zero columns are stored as a compact N×32 matrix, where N is the number of 1s in the zero map.

[0067] As an option, the transposed matrix can be output together with the zero map to support subsequent decoding operations. For example, in the case of a zero map of "11010101", the non-zero columns correspond to the 1st, 2nd, 4th, 6th, and 8th columns of the matrix, which will be extracted and transposed to N×32 format for storage.

[0068] In some embodiments, in order to further optimize the hardware implementation, the present invention uses a buffer register to store non-zero column data when removing zero columns. Specifically, the buffer register scans the zero map bit by bit, and when a column is detected as a non-zero column, the data of the corresponding column is written into the register, and the register data is stored in the output buffer in sequence. This design can reduce the number of accesses to the matrix storage and improve the operation efficiency.

[0069] In one possible implementation, the generated zero map and non-zero matrix are packaged and output together. The zero map occupies a fixed 8-bit storage space, while the storage space of the non-zero matrix is ​​directly related to the number of zero columns. Through this step, the present invention significantly reduces the storage requirements of the exponential bit residual and provides an efficient processing mechanism for the overall compression method. The above design is easy to implement in hardware and can support large-scale compression operations of gradient data in distributed training.

[0070] S5, pack the sign bit, the truncated mantissa bit, the zero mapping and the non-zero residual data into a compressed output stream After completing the truncation of the sign bit and the mantissa bit, the prediction of the exponent bit and the residual calculation, and the zero coding of the residual, the present invention forms a unified compressed output stream by packaging the various parts of the data. This step is intended to organize the compressed gradient data into a format that is convenient for storage and transmission, while ensuring the operability and data integrity of the decoding process. The structural design of the output stream fully considers the characteristics of each part of the gradient data and the resource utilization efficiency in hardware implementation.

[0071] In this embodiment, the compressed output stream consists of the following parts: zero map, non-zero matrix, sign bit data and truncated mantissa bit data. Generally, each part of the data is packaged and arranged according to a fixed structure to facilitate the step-by-step reading and restoration of the original gradient data during decoding.

[0072] Specifically, the zero map data is first stored in the output stream. The zero map is used to mark whether the column of the exponential bit residual matrix contains valid data, and generally occupies 8 bits of storage space. For example, when the zero map is "11010101", the 1st, 2nd, 4th, 6th, and 8th columns in the matrix are identified as non-zero columns, and the remaining columns are all zero values. As an option, the zero map can be directly packed into the first 8 bits of the output stream through bit operations, thereby reducing additional operation steps.

[0073] Secondly, the transposed non-zero matrix data is stored in the output stream. The non-zero matrix consists of non-zero columns of exponent bit residues and is stored in an N×32 matrix format, where N is the number of 1s in the zero map. For example, in the case where the zero map is "11010101", the N value of the non-zero matrix is ​​5. In general, the data of the non-zero matrix is ​​arranged in column-major order and written directly to the output stream.

[0074] In a possible implementation, in order to improve the writing efficiency of non-zero matrices, the present invention uses a buffer register to temporarily store matrix data. The buffer register writes matrix data row by row, and writes the data to the output stream at one time after the register is fully loaded. This design reduces frequent access to the memory and improves the efficiency of data transmission.

[0075] Next, the sign bit data is stored in the output stream. In general, the sign bit directly affects the directionality of the gradient, so it is not compressed but directly stored in a 32-bit structure. For example, the sign bits of the 32 floating-point numbers in the data block can be arranged in sequence as a 32-bit binary number and written to the corresponding position of the output stream.

[0076] Finally, the truncated mantissa data is written to the output stream. Specifically, only the highest 9 bits of the mantissa are retained to reduce the amount of data while retaining the main accuracy of the gradient. As an option, the mantissa data can be combined and packed with the sign bit data to further reduce the redundant storage space of the output stream.

[0077] In one possible implementation, the data structure of the output stream is as follows: The first 8 bits store zero mapping; The next N×32 bits store the non-zero matrix; The subsequent 32 bits store the sign bit; Finally, the truncated mantissa data is stored.

[0078] In some embodiments, in order to support continuous compression of multiple data blocks, the present invention adds a flag bit in the output stream to indicate the selected exponent bit prediction method. For example, the flag bit "0" indicates that a fixed value prediction is used, and "1" indicates that a block average prediction is used. The flag bit is stored in the header area of ​​the output stream and is arranged adjacent to the zero mapping data.

[0079] Through the above design, this embodiment can effectively organize the compressed gradient data to form a clearly structured output stream. This output stream is not only convenient for decoding operations, but also adapts to the needs of hardware pipeline processing, providing an efficient solution for gradient communication in distributed deep learning training. The above design is easy to implement in hardware and has high scalability and adaptability.

[0080] The present invention realizes efficient data compression by differentially processing the sign bit, exponent bit and mantissa bit of the floating point number. Specifically, the method first processes the gradient data in blocks, and truncates the mantissa bits, retaining only the high bits to reduce redundancy, while generating residuals for the exponent bits through fixed value prediction or block average prediction, and further compressing the data using zero mapping coding. While maintaining the main information of the gradient data, the method significantly reduces the storage and transmission requirements, is suitable for implementation in hardware environments such as FPGA or GPU, can maintain the accuracy and convergence of model training at a high compression rate, and is suitable for large-scale distributed deep learning tasks.

[0081] Comparative experiment: In a specific implementation of the present invention, by comparing the accuracy performance of different models under the baseline method and the compression method of the present invention (HHGC), the effectiveness and applicability of the compression method can be further verified. Figure 5 (a) and (b) show the Top-1 validation accuracy of the ResNet50 and VGG16 models as the number of training steps changes, respectively.

[0082] In this embodiment, by Figure 5 (a) shows that the accuracy comparison curve of the ResNet50 model shows that the accuracy curve of the HHGC method is basically consistent with that of the Baseline method in the early training stage, indicating that HHGC compression did not have an adverse effect on the gradient update of the model during the early training process. As the number of training steps increases, the accuracy performance of the HHGC method gradually becomes consistent with that of the Baseline method, verifying that the present invention can still maintain the stability and accuracy of training at a high compression rate.

[0083] Specifically, Figure 5 The VGG16 model accuracy curve in (b) further shows that there is almost no difference in the accuracy performance of the HHGC method and the Baseline method at different training steps. In general, the training of the VGG16 model is more sensitive to gradient accuracy, but after using the HHGC compression method, its accuracy is not affected, indicating that the method of the present invention can well adapt to deep learning models with different structures.

[0084] In some embodiments, by adjusting the parameters of the mantissa truncation and exponent residual calculation in the present invention, a wider range of models and data sets can be adapted, while ensuring that the retention of gradient information does not affect the convergence of model training. The HHGC method shows an accuracy curve consistent with the Baseline in different models, verifying that the compression method of the present invention not only has a high compression rate, but also can maintain stable model accuracy, and is suitable for various deep learning tasks.

[0085] The hardware-oriented high-precision gradient compression device described below and the hardware-oriented high-precision gradient compression method described above can be referred to each other.

[0086] Please refer to the attached Figure 6 The present invention also provides a hardware-oriented high-precision gradient compression device, comprising: A block loading module 10, used for dividing the gradient data into data blocks; A prediction module 20, for applying a fixed value or a block average prediction method to the exponent bit to generate a prediction value and calculate a residual; An encoding module 30, configured to generate a zero map for the residual and remove zero columns to form a non-zero matrix; The output module 40 is used to pack the sign bit, the truncated mantissa bit, the zero map and the non-zero matrix data into a compressed output stream.

[0087] The device of this embodiment can be used to execute the above method embodiment, and its principles and technical effects are similar, which will not be repeated here.

[0088] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A hardware-oriented high-precision gradient compression method, characterized in that: The following steps are involved: Divide the gradient data into multiple data blocks, each data block contains a number of floating point data; Separate the sign bit, exponent bit and mantissa bit of each floating point data, where the mantissa bit is truncated to remove the low bit and only retain the high bit; Apply the prediction method to the exponent bit to generate a predicted value, and calculate the residual between the exponent bit and the predicted value; Perform zero encoding on the residual to generate a zero map to eliminate zero columns; The sign bit, truncated mantissa bits, zero mapping, and non-zero residual data are packed into a compressed output stream.

2. A hardware-oriented high-precision gradient compression method according to claim 1, characterized in that: The step of applying the prediction method to the exponent bit to generate a predicted value comprises: Use fixed values ​​as predicted values, where the fixed values ​​are based on the results of statistical analysis of the data distribution.

3. A hardware-oriented high-precision gradient compression method according to claim 2, characterized in that: The fixed prediction value is 108, and the fixed prediction value is used to match the exponent range of the gradient data in the single-precision floating point number.

4. The hardware-oriented high-precision gradient compression method according to claim 1, characterized in that: The step of applying the prediction method to the exponent bit to generate the predicted value also includes: Use the average index value of the previous data block as the predicted value.

5. A hardware-oriented high-precision gradient compression method according to claim 4, characterized in that: The average index value of the previous data block is stored in an external register, and is used for prediction at the beginning of each data block.

6. The hardware-oriented high-precision gradient compression method according to claim 1, characterized in that: The step of calculating the residual between the exponent bit and the predicted value comprises: The exponent bits are subtracted from the predicted value to produce a residual.

7. A hardware-oriented high-precision gradient compression method according to claim 6, characterized in that: The step of calculating the residual between the exponent bit and the predicted value also includes: checking the most significant bit of the residual; If the residual is negative, the residual is processed by two's complement to convert the negative value into a positive value close to zero; If the residual is non-negative, it is preserved in its original form.

8. The hardware-oriented high-precision gradient compression method according to claim 1, characterized in that: The step of performing zero encoding comprises: Perform a bitwise OR operation on each column of the residual matrix to generate a zero map to indicate whether each column contains a non-zero value; Based on the zero map, zero columns are removed and non-zero columns are transposed into a matrix of the specified format to simplify storage and transmission.

9. The hardware-oriented high-precision gradient compression method according to claim 1, characterized in that: The compressed output stream includes: A flag bit used to indicate the selected prediction method; The generated zero map is used to restore the zero column during decoding; Non-zero column matrix data, containing compressed non-zero exponential residual values; The mantissa and sign bits are truncated to ensure that the gradient direction information and the main precision are preserved.

10. A hardware-oriented high-precision gradient compression device, used to execute the method according to any one of claims 1 to 9, characterized in that: include: Block loading module, used to split gradient data into data blocks; A prediction module, for applying a fixed value or a block average prediction method to the exponent bit to generate a prediction value and calculate a residual; An encoding module, used to generate a zero map for the residual and remove zero columns to form a non-zero matrix; The output module is used to pack the sign bit, truncated mantissa bits, zero map and non-zero matrix data into a compressed output stream.

Images