Data compression and decompression method, device and system

Abstract

The embodiment of the invention discloses a data compression and decompression method, device and system, is applied to the technical field of data processing, and is used for selecting the optimal compressed data block length when the compression ratios are different so as to realize optimization of data compression and decompression. The method comprises the steps: when data compression is carried out, obtaining to-be-compressed target data by a compression engine and then compressing data blocks with the length of L bytes into compressed data blocks with the length of I bytes in sequence, and obtaining a compression result, wherein L and l are the maximum bandwidth length of a decompression outlet and the maximum bandwidth length of a decompression inlet of the decompression engine respectively; sequentially changing l into l1 to ln, repeatedly executing the compression step to obtain n compression results, acquiring the compressed data block length corresponding to the compression result with the minimum data total length in the n compression results, storing the compressed data block length into compression information, and sending the compression information and the compression result with the minimum data total length to the decompression engine.

Description

technical field [0001] The present application relates to the technical field of data processing, in particular to a data compression and decompression method, device and system. Background technique [0002] With the rapid development of artificial intelligence (AI) technology, more and more AI hardware accelerators (that is, AI hardware integrated chips) are used to accelerate operations with large amounts of data. For example, AI hardware accelerators can be used to The process of image processing realized by convolutional neural networks (CNN) is accelerated. [0003] However, when using AI hardware accelerators to run machine learning algorithms such as convolutional neural networks, a large number of parameters are often required, and these parameters cannot be completely stored inside the chip of the AI ​​hardware accelerator. Therefore, when running machine learning algorithms, it is necessary to import a large number of parameters to complete the calculation. Sinc...

AI Technical Summary

Problems solved by technology

Although this decompression method can ensure that data of a fixed length is read in each clock cycle during decompression, there is no coupling relationship between the data read in two clock cycles before and after, and the data length after decompression will not be greater than the data bit width of the decompression exit, which is convenient However, when this compression / decompression method processes data with a low sparse rate, due to the high compression rate, the length of the data contained in the decompression result corresponding to each clock cycle will be relatively short, which requires more Only after the decompression results of one clock cycle are concatenated can the L-length data be assembled, which leads to a low utilization rate of the decompression outlet bandwidth. In addition, repeated data splicing operations will also frequently shift the data and consume a lot of power.

When processing data with a high sparse rate, due to the low compression rate, 0 values ​​are filled during compression, which reduces the data compression rate in disguise.

It can be seen that when the data compression / decompression method commonly used at present processes data with different compression rates, it will either reduce the data compression rate, or reduce the utilization rate of the decompression outlet bandwidth and consume a lot of power.

Images