Method for quickly realizing GZIP compression based on hardware and application thereof

A hardware and fast technology, applied in electrical components, code conversion, etc., can solve problems such as inability to meet real-time compression processing requirements, consumption of large CPU and memory resources, low processing rate, etc., to improve data parallel processing capabilities and improve data throughput. rate, the effect of improving the throughput rate

Active Publication Date: 2022-03-08
HUAZHONG UNIV OF SCI & TECH
View PDF5 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In traditional applications, software-based solutions are widely used in data compression and decompression, but large-scale data compression and decompression will consume a lot of CPU and memory resources, and there is a performance bottleneck of low processing rate, which cannot meet the needs of big data environments. Real-time compression processing requirements under

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for quickly realizing GZIP compression based on hardware and application thereof
  • Method for quickly realizing GZIP compression based on hardware and application thereof
  • Method for quickly realizing GZIP compression based on hardware and application thereof

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0043] A method for quickly implementing GZIP compression based on hardware, including:

[0044] The text to be compressed is divided into multiple processing windows according to bytes, and the number of bytes m of each processing window is equal to the length that the hardware can process strings in parallel;

[0045] Simultaneously calculate the hash value at multiple adjacent locations. The implementation method is: according to the sequence of context, within one clock cycle, the current processing window and the next processing window adjacent to it form a cache window, and from the cache window Each byte of the current processing window is the starting byte and extends to build m character strings with a length of m; calculate the hash values ​​of the m character strings simultaneously within this clock cycle;

[0046] In one clock cycle, the m character strings are evenly stored in K single-port rams configured on the hardware to build a hash table entry dictionary; wh...

Embodiment 2

[0066] A hardware-based fast implementation of GZIP compression system, such as figure 2 As shown, it includes K single-port rams, wherein the ram of each address adopts a three-level cache structure, and also includes: data feeding module, hash value calculation module, string matching module, matching pruning module, Huffman coding module , and packaged modules.

[0067]The data feed module is used to divide the text to be compressed into multiple non-overlapping processing windows with a size of m bytes; within one clock cycle, the current processing window and its adjacent next processing window are transmitted to the hash value calculation module; the hash value calculation module is used to perform the hash value calculation and the read and write operation of the string in a kind of hardware-based method for quickly realizing GZIP compression as described in Embodiment 1; the string matching module is used to perform such as implementing The character string matching ...

Embodiment 3

[0078] A computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein, when the computer program is run by a processor, the device where the storage medium is located is controlled to execute one of the methods described in Embodiment 1. A hardware-based fast implementation of GZIP compression. The relevant technical solutions are the same as those in Embodiment 1, and will not be repeated here.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention belongs to the field of data compression, and particularly relates to a method for quickly realizing GZIP compression based on hardware and application thereof, and the method comprises the following steps: dividing a to-be-processed text into a plurality of processing windows with the size of m bytes, carrying out parallel calculation on m character string hash values starting from each byte in one processing window, and taking the hash values as index addresses of a dictionary; storing the current to-be-processed character string into the dictionary, and reading the historical candidate character string to complete the initial matching of the character string; the read historical character string and the current character string to be processed are compared byte by byte to finish fine matching, and the matching length and the matching distance of each character string are obtained. A matching pruning algorithm is introduced to eliminate matching overlapping of character strings in parallel processing windows and between the windows, the dependency between algorithms is solved, and the data parallel processing capacity is improved. According to the invention, a high-bandwidth full-pipeline extensible lossless compression data path suitable for hardware is realized, and the size of a processing window is dynamically adjusted according to the tradeoff among hardware resources, throughput rate and compression rate.

Description

technical field [0001] The invention belongs to the field of data compression, and more specifically relates to a method for quickly realizing GZIP compression based on hardware and an application thereof. Background technique [0002] With the continuous development of the current network technology, the amount of data produced by humans is increasing exponentially, and network-based applications are also increasing. A large amount of data needs to be transmitted on the network, which puts forward higher requirements for data storage and transmission. Require. It is necessary to realize data storage under limited hardware resources and transmit as much data as possible under limited bandwidth, so it is very necessary to effectively compress data before data transmission and storage. [0003] GZIP is a lossless compression algorithm. First, the lz77 algorithm is used to find a string that matches the context. For the matching string, the length distance is used to replace i...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H03M7/30
CPCH03M7/3086
Inventor 刘政林杨东海于润泽
Owner HUAZHONG UNIV OF SCI & TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap