High-speed lossless data compression method based on GPU-CPU hybrid platform

Abstract

The invention discloses a high-speed lossless data compression method based on a GPU-CPU hybrid platform. The method includes: allowing a CPU to read a data file to be compressed, copying the data file to be compressed from a memory to a global memory of a GPU, setting a thread block set bk(a) of the GPU, setting the number b of threads in each thread block, setting a length c of a compression dictionary window, setting a head hand p_dic_h pointing to a first compression dictionary window, setting the size d of a pre-reading window, setting a hand p_pre_r pointing to a first pre-reading window, setting an initial value p_dic_h-c of the hand, initializing an operating thread set threads(a*b), c*d in size, and (a*b / 2) / c matrixes gMatrix, calling the (a*b / 2)-th thread in the operating thread set threads(a*b) to process q=(a*b / 2) / c data, c+d in length, in the data file to be compressed, searching for oblique segments having most continuous 1 from each of the q result matrixes gMatrix, and determining a three-element result array locations(p) of each result matrix. The method has the advantage that compression rate of massive data can be increased greatly.

Description

technical field [0001] The invention belongs to the technical field of computer data compression, and more specifically relates to a high-speed lossless data compression method based on a mixed platform of GPU and CPU. Background technique [0002] According to the research of International Data Corporation (IDC), in the past 10 years, the total amount of global information has doubled every two years. In 2011, the total amount of data created and copied globally was 1.8ZB (1800EB), and by 2015 it will reach 8ZB (8000EB), and in the next ten years, around 2020, the global data will be 50 times higher than it is now . At the same time, although network bandwidth and new storage technologies have also been rapidly developed, they are still far from meeting the performance requirements of current massive data transmission and storage. One of the key technologies to solve the challenges of mass data transmission and storage is data compression. Through data compression techno...

AI Technical Summary

Problems solved by technology

However, since these algorithms are invented based on the CPU platform, they are not suitable for the different hardware structure of the GPU in essence. Therefore, in practical applications, the improvement of the compression rate still needs to be improved.

Images