FSE coding method and device, equipment and storage medium

Abstract

The invention discloses an FSE coding method and device, equipment and a storage medium, and the method comprises the steps: obtaining a to-be-coded symbol sequence corresponding to to-be-coded data, and determining a to-be-coded symbol of which the symbol state does not depend on other symbol states from the to-be-coded symbol sequence, so as to obtain a target to-be-coded symbol; splitting the to-be-coded symbol sequence based on the position of the target to-be-coded symbol in the to-be-coded symbol sequence to obtain a plurality of to-be-coded subsequences corresponding to the to-be-coded symbol sequence; and separately coding the plurality of sub-sequences to be coded to obtain the coded data to be coded. According to the method, the symbol with the unique state is taken as the splitting point to split the to-be-coded symbol sequence to obtain the plurality of subsequences, and compared with the mode that the whole to-be-coded symbol sequence is subjected to serial coding and splitting to realize parallel coding of the subsequences, the coding rate is improved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to an FSE encoding method, device, equipment and storage medium. Background technique [0002] With the rapid development of cutting-edge science and technology such as big data, the explosive growth of data has been generated, and massive data has brought enormous pressure to existing storage devices. In the face of continuous growth of massive data, data compression has become the best and most effective way to reduce the storage burden on servers and reduce storage costs. Data compression reduces the amount of data to reduce storage space and improve transmission, storage and processing efficiency without losing useful information. Lossless data compression is generally achieved through two methods: one is the compression algorithm through the dictionary, including the LZ series algorithm, which can realize the search function of repeated data; the other is the c...

AI Technical Summary

Problems solved by technology

On this basis, the hybrid compression algorithm Zstandard (zstd) composed of LZ77 encoding, Hufman encoding and FSE has better compression performance than other compression algorithms (such as DEFLATE algorithm, LZ4 algorithm), but due to its interlocking The serial stream encoding method limits its encoding rate to a large extent, making its compression performance unable to achieve

Images