A FSE encoding method, device, apparatus and storage medium
By splitting the symbol sequence into independent subsequences for parallel encoding in the FSE encoding method, the problem of limited encoding rate in serial stream encoding is solved, thereby improving encoding rate and compression performance.
Patent Information
- Application Number
- CN202111608151.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-23
- Publication Date
- 2026-01-27
- Estimated Expiration
- 2041-12-23
AI Technical Summary
Existing FSE encoding methods cannot achieve ideal compression performance due to the limitation of encoding rate caused by serial stream encoding.
By identifying symbols in the sequence of symbols to be encoded whose states are independent of other symbol states, and splitting them based on their positions to form multiple independent subsequences, these subsequences are encoded in parallel using multiple encoders.
Parallel encoding of subsequences was achieved, improving encoding rate and compression performance.
Smart Images
Figure CN114301468B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to an FSE encoding method, apparatus, device, and storage medium. Background Technology
[0002] With the rapid development of cutting-edge technologies such as big data, data is experiencing explosive growth, placing enormous pressure on existing storage devices. Faced with this continuously increasing volume of data, data compression has become the optimal and effective method to alleviate the storage burden on servers and reduce storage costs. Data compression reduces data volume to decrease storage space and improve transmission, storage, and processing efficiency without losing useful information. Lossless data compression is generally achieved through two methods: one is through dictionary-based compression algorithms, including the LZ series algorithms, which can perform duplicate data search; the other is based on statistical model compression algorithms, such as Huffman codes and arithmetic coding. The core idea of these algorithms is to allocate code length according to the frequency of symbol occurrence; the higher the symbol frequency, the shorter the code length.
[0003] However, Huffman coding can only allocate codewords of integer length. While arithmetic coding can address this limitation, its implementation is complex and its coding efficiency is very low (approximately 1 / 10 that of Huffman coding), making it unsuitable for data compression. Currently, the Finite State Entropy (FSE) entropy coding algorithm combines the advantages of both Huffman and arithmetic coding. Based on this, the hybrid compression algorithm Zstandard (zstd), composed of LZ77 coding, Hufman coding, and FSE, exhibits better compression performance compared to other compression algorithms (such as DEFLATE and LZ4). However, its interlocking serial stream coding method significantly limits its coding rate, preventing it from achieving its full compression potential.
[0004] Therefore, how to provide an FSE encoding method with a higher encoding rate is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of this, the purpose of this invention is to provide an FSE encoding method, apparatus, device, and storage medium capable of parallel encoding of subsequences and improving the encoding rate. The specific solution is as follows:
[0006] The first aspect of this application provides an FSE encoding method, comprising:
[0007] Obtain the sequence of symbols to be encoded corresponding to the data to be encoded, and determine the symbols to be encoded from the sequence of symbols whose symbol states do not depend on other symbol states, so as to obtain the target symbols to be encoded;
[0008] The sequence of symbols to be encoded is split based on the position of the target symbol in the sequence of symbols to be encoded to obtain multiple sub-sequences to be encoded corresponding to the sequence of symbols to be encoded;
[0009] Each of the multiple subsequences to be encoded is encoded individually to obtain the encoded data to be encoded.
[0010] Optionally, determining the symbols to be encoded from the sequence of symbols to be encoded whose symbol states do not depend on other symbol states, in order to obtain the target symbols to be encoded, includes:
[0011] Determine the frequency of occurrence of each symbol to be encoded in the sequence of symbols to be encoded, so as to obtain the normalized probability corresponding to each symbol to be encoded;
[0012] Based on the normalized probability, the target symbol to be encoded is determined from the sequence of symbols to be encoded in which the symbol state has no dependency on other symbol states.
[0013] Optionally, after determining the frequency of occurrence of each symbol to be encoded in the symbol sequence to be encoded, the method further includes:
[0014] The normalized probability corresponding to each symbol to be encoded is calculated based on the occurrence frequency of each symbol to be encoded, the total number of symbols in the sequence of symbols to be encoded, and the length of the preset FSE code table used for encoding.
[0015] Optionally, determining the target symbol to be encoded from the sequence of symbols to be encoded based on normalized probabilities, where the symbol state has no dependency on other symbol states, includes:
[0016] The target symbol to be encoded is obtained by determining the character to be encoded with a normalized probability absolute value of 1 from the sequence of symbols to be encoded.
[0017] Optionally, the step of splitting the symbol sequence to be encoded based on the position of the target symbol in the symbol sequence to be encoded to obtain multiple sub-sequences to be encoded corresponding to the symbol sequence to be encoded includes:
[0018] Each target symbol to be encoded is split at the beginning position of the symbol sequence to be encoded, so as to split each target symbol to be encoded from the symbol sequence to obtain a first type of subsequence to be encoded with the same number of subsequences as the number of target symbols to be encoded, and a second type of subsequence to be encoded with one more subsequence than the number of target symbols to be encoded; wherein each first type of subsequence to be encoded corresponds to one target symbol to be encoded, and each second type of symbol to be encoded is a broken chain formed by a gap at the split position.
[0019] Optionally, the step of individually encoding the multiple sub-sequences to be encoded to obtain the encoded data to be encoded includes:
[0020] Multiple subsequences to be encoded are individually encoded using a preset FSE code table to obtain multiple encoded subsequences to be encoded. The multiple encoded subsequences to be encoded are then concatenated according to the order of the symbols to be encoded to obtain the encoded data to be encoded.
[0021] Optionally, encoding the plurality of subsequences to be encoded individually includes:
[0022] Multiple encoders are used to encode multiple sub-sequences to be encoded individually in parallel; wherein, one encoder corresponds to one sub-sequence to be encoded.
[0023] A second aspect of this application provides an FSE encoding device, comprising:
[0024] The acquisition module is used to acquire the sequence of symbols to be encoded corresponding to the data to be encoded, and to determine the symbols to be encoded from the sequence of symbols whose symbol states do not depend on other symbol states, so as to obtain the target symbols to be encoded.
[0025] The splitting module is used to split the symbol sequence to be encoded based on the position of the target symbol in the symbol sequence to be encoded, so as to obtain multiple sub-sequences to be encoded corresponding to the symbol sequence to be encoded;
[0026] The encoding module is used to encode each of the multiple sub-sequences to be encoded individually to obtain the encoded data to be encoded.
[0027] A third aspect of this application provides an electronic device including a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the aforementioned FSE encoding method.
[0028] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions that, when loaded and executed by a processor, implement the aforementioned FSE encoding method.
[0029] In this application, a sequence of symbols to be encoded corresponding to the data to be encoded is first obtained, and symbols whose states do not depend on other symbol states are identified from the sequence to obtain target symbols to be encoded. Then, based on the position of the target symbols in the sequence, the sequence is split to obtain multiple sub-sequences to be encoded. Finally, each sub-sequence is encoded individually to obtain the encoded data. This application obtains multiple sub-sequences by finding symbols in the sequence that do not depend on other symbol states, i.e., symbols with unique states, as the splitting points. Compared to serial encoding of the entire sequence, this allows for parallel encoding of sub-sequences, improving the encoding rate. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0031] Figure 1 A flowchart of an FSE encoding method is provided in this application;
[0032] Figure 2 A schematic diagram of a specific FSE encoding method provided in this application;
[0033] Figure 3 An example diagram of an FSE code representation provided in this application;
[0034] Figure 4 An example diagram illustrating the encoding process of a specific subsequence to be encoded provided in this application;
[0035] Figure 5 Example diagram of another specific encoding process for the subsequence to be encoded provided in this application;
[0036] Figure 6 This application provides a schematic diagram of the structure of an FSE encoding device;
[0037] Figure 7 This application provides a structural diagram of an FSE-encoded electronic device. Detailed Implementation
[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0039] Existing FSE entropy coding algorithms combine the advantages of Huffman and arithmetic coding. Based on this, the hybrid compression algorithm Zstandard (zstd), composed of LZ77 coding, Hufman coding, and FSE, exhibits better compression performance compared to other compression algorithms (such as DEFLATE and LZ4). However, its interlocking serial stream coding method significantly limits its coding rate, preventing it from achieving its full compression potential. To address these shortcomings, this application provides an FSE coding scheme that enables parallel coding of subsequences, thereby improving the coding rate.
[0040] Figure 1 A flowchart illustrating an FSE encoding method provided in an embodiment of this application. See also... Figure 1 As shown, the FSE encoding method includes:
[0041] S11: Obtain the sequence of symbols to be encoded corresponding to the data to be encoded, and determine the symbols to be encoded from the sequence of symbols whose symbol states do not depend on other symbol states, so as to obtain the target symbols to be encoded.
[0042] In this embodiment, when encoding the data to be encoded, a sequence of symbols to be encoded corresponding to the data to be encoded is obtained, and symbols whose states do not depend on other symbol states are determined from the sequence of symbols to be encoded, so as to obtain the target symbols to be encoded. The target symbols to be encoded have unique states in the FSE code table. The symbols to be encoded are symbols of the Zstandard protocol, which refer to the output symbols of the source data after deduplication by the LZ77 algorithm, including four types of symbols: literals (original text letters), literal_length (original text segment length), match_length (repetition length), and offset (offset distance, the distance between repeated data). Among them, literals use Hufffman coding, and the other three types of symbols use FSE coding.
[0043] Specifically, the frequency of occurrence of each symbol to be encoded in the sequence is first determined to obtain the normalized probability corresponding to each symbol. This normalized probability can be calculated based on the frequency of occurrence of each symbol, the total number of symbols in the sequence, and the length of the preset FSE code table used for encoding. Then, based on the normalized probability, symbols whose states do not depend on other symbol states are identified from the sequence to be encoded, thus obtaining the target symbols to be encoded.
[0044] It's easy to understand that the length of the FSE code table is an integer power of 2. The ZSTD protocol calls the index of the FSE code table a state, and each state corresponds to a symbol. The number of states corresponding to each symbol in the FSE code table is related to the normalized probability of that symbol. Therefore, it is necessary to normalize the frequency of the symbols to generate a probability table. The normalization calculation formula is as follows:
[0045]
[0046] The normalized probability of a symbol is related to its frequency relative to the total number of symbols. The normalization algorithm needs to satisfy the following two principles: (1) the sum of the normalized probabilities must be equal to Table_size; (2) as long as a symbol appears, the normalized probability is at least 1. Assuming the total number of source symbols is 1024, the length of the FSE code table is 32, the frequency of symbol s1 is 32, and the frequency of symbol s2 is 2, the normalized probabilities of the two symbols can be calculated by the above formula:
[0047]
[0048]
[0049] Due to the requirements of the normalization principle, the value of norm(s2) needs to be set to the value of the Zstandard protocol. To distinguish between the two cases where the normalization probability is equal to 1 (represented by s1 and s2), the normalization probability of case s2 is marked with "-1", and the normalization probability of case s1 is marked with "1". Therefore, this embodiment takes the normalization probability of "1" or "-1" as the starting point, and determines the character to be encoded with a normalization probability absolute value of 1 from the sequence of symbols to be encoded, so as to obtain the target symbol to be encoded.
[0050] To explain the rationale for obtaining the target symbols to be encoded in the above manner, in conjunction with the existing FSE code table construction process, we first assign states to symbols with a probability of "-1", starting from the reverse order of states (Table_size-1). The pseudocode is as follows:
[0051]
[0052] Then, assign states to other symbols. The pseudo-code is as follows:
[0053]
[0054] In the FSE code table, each state corresponds to a symbol and a state range (State-range). The State-range is described as (Baseline, number-bits). In the FSE encoding and decoding process, the State-range of each state is used to describe the range of state values of the next symbol [Baseline, Baseline+(1<<number-bits)]. For a symbol, there are several corresponding states in the FSE code table. The State-ranges corresponding to these states do not overlap, and the sum of the lengths of all State-ranges is equal to the total number of states (Table-size) in the FSE code table. The length value of the State-range must be an integer power of 2 (the integer is number-bits), and the length of the State-ranges of all states of the same symbol may take 1-2 values. If the probability of the symbol is exactly an integer power of 2 (norm = 2^n), then the length of the State-range has a unique value:
[0055] number_bits = table_log - n
[0056]
[0057] In other cases, the State-range has two length values:
[0058]
[0059]
[0060] For the same symbol, all states need to be sorted first. The length of the State-range corresponding to the smaller state is num_bits1, and the length of the larger state is num_bits2. Assume N = 2 ceil(log(norm)) , then the number of states corresponding to num_bits1 is N - norm, and the number of states corresponding to num_bits2 is 2 * norm - N. The baseline (Baseline) of the State-range is assigned starting from the larger state.
[0061] Assuming the FSE code table has a length of 64 and the probability of a symbol s is 5, if we start assigning from state 0, then the symbols corresponding to states [0, 43, 22, 1, 44] are all s, and the sorted result is [0, 1, 22, 43, 44]. N = 2^3 = 8, so the number of states with a Number-bits value of 4 is 3, i.e., [0, 1, 22]. The number of states with a Number-bits value of 3 is 2, i.e., [43, 44]. The baseline of State(43) is 0, the baseline of State(44) is 8 (0+8), the baseline of State(0) is 16 (8+8), the baseline of State(1) is 32 (16+16), and the baseline of State(22) is 48 (32+16). The above calculation process is summarized in the table below:
[0062] state_order 0 1 2 3 4 State 0 1 22 43 44 Number_bit 4 4 4 3 3 Baseline 16 32 48 0 8 State_range 16-31 32-47 48-63 0-7 8-15
[0063] During encoding, assuming the previous symbol s1 selects the state as State
[39] , and the next symbol s2, and s2 can be encoded using Table 1, during FSE encoding, s2 needs to select the corresponding state of the State-range where State
[39] is located. In this example, State[1] is obviously selected. After the state of s2 is determined, the state of s1 can be encoded. The bit width of the State
[39] encoding is the number_bits(4) of State[1], and the encoding value is 7 (39–32, 32 is the baseline of State[1]). Thus, the encoding of s1 is completed. Assuming that s3 is exactly the same as s2, then the state of s3 selects the state
[43] corresponding to the State-range(0-7) where State[1] is located. The bit width of the s2 encoding is the number_bits(3) of state
[43] , and the encoding value is 1 (1–0, 0 is the baseline of State
[43] ).
[0064] Therefore, the state of the first symbol determines the state of the second symbol, and the codeword of the first symbol is determined by the range of the second state; the state of the second symbol determines the state of the third symbol, and the codeword of the second symbol is determined by the range of the third state; and so on, with the state of the second-to-last symbol determining the state of the last symbol, and the codeword of the second-to-last symbol being determined by the range of the last symbol. In summary, the state of the preceding symbol determines the state of the following symbol, but the encoded content (code length, codeword) of the preceding symbol is determined by the following symbol. In the FSE encoding process, the state of each symbol in the entire symbol sequence depends on the state of the preceding symbol. The entire encoding process is interconnected, flowing sequentially step by step. From a process perspective, it is difficult to implement FSE encoding of symbol sequences in parallel. Therefore, for certain symbols with unique states, i.e., symbols with a normalized probability of "1" or "-1", their chosen state can be predicted in advance.
[0065] S12: Based on the position of the target symbol to be encoded in the symbol sequence to be encoded, the symbol sequence to be encoded is split to obtain multiple sub-sequences to be encoded corresponding to the symbol sequence to be encoded.
[0066] In this embodiment, after the target symbol to be encoded is found, the sequence of symbols to be encoded is split based on the position of the target symbol in the sequence of symbols to be encoded to obtain multiple sub-sequences corresponding to the sequence of symbols to be encoded. Specifically, each target symbol to be encoded is split at the beginning position in the sequence of symbols to be encoded to obtain a first type of sub-sequence with the same number of sub-sequences as the number of target symbols to be encoded, and a second type of sub-sequence with one more sub-sequence than the number of target symbols to be encoded; wherein each first type of sub-sequence corresponds to one target symbol to be encoded, and each second type of symbol to be encoded is a broken chain formed by a gap at the split position.
[0067] S13: Encode each of the multiple subsequences to be encoded individually to obtain the encoded data to be encoded.
[0068] In this embodiment, multiple sub-sequences to be encoded are encoded individually to obtain encoded data. Multiple encoders can be used to encode the multiple sub-sequences individually in parallel. Each encoder corresponds to one sub-sequence, thereby accelerating the FSE encoding rate in hardware. Based on this, a preset FSE code table is used to encode the multiple sub-sequences individually to obtain multiple encoded sub-sequences, and these encoded sub-sequences are then concatenated according to the order of the symbols to be encoded to obtain the encoded data.
[0069] The specific implementation process of this embodiment is as follows: Figure 2 As shown, the entire symbol sequence is first searched to find the positions of all symbols with a probability of "1" or "-1". Then, using these positions as the first and last positions, the entire symbol sequence is divided into several sub-sequences. Each sub-sequence is encoded using the FSE code table, generating n FSE-coded data segments. All sub-sequences can be encoded using parallel FSE. Finally, the n FSE-coded data segments are concatenated according to the natural order of the source data, and the encoded bitstream is output. More specifically, assuming the symbol sequence to be encoded is "fdgecaedcfcegeafeeadedgfegcafgegedgegfecebfecgegcgefbcdgaefdebgfdefagfecfgedeaegfedgfbdcdfefefadgecgfgdcgcbfefdegcgefacefdgcdefd", the symbol frequency statistics are as follows:
[0070] a b c d e f g 9 5 16 17 32 24 24
[0071] The normalization results are as follows:
[0072] a b c d e f g 1 -1 2 -2 4 3 3
[0073] Assuming the FSE code table has a length of 16, then it will be constructed Figure 3The code table is shown. Finding the characters 'a' and 'b', the above string can be divided into the following 15 subsequences: “fdgec”, “aedcfcege”, “afee”, “adedgfegc”, “afgegedgegfece”, “bfecgegcgef”, “bcdg”, “aefde”, “bgfdef”, “agfecfgede”, “aegfedgf”, “bdcdfefef”, “adgecgfgdcgc”, “bfefdegcgef”, and “acefdgcdefd”. The encoding process is explained using “afgegedgegfece” and “bfecgegcgef” as examples.
[0074] For "afgegedgegfece", the state of 'a' is 0. The next symbol is 'f'. The states corresponding to 'f' are 2 (8-15), 5 (0-3), and 12 (4-7). Since '0' falls within the range of state 5, the state of 'f' is 5, and the bit width of 'a' is 2, resulting in a value of 0. The next symbol is 'g'. The states corresponding to 'g' are 3 (8-15), 6 (0-3), and 9 (4-7). Here, state 9 is chosen, and the bit width of 'f' is 2, resulting in a value of 1 (5-4). The next symbol is 'e'. The states corresponding to 'e' are 1 (0-3), 8 (4-7), 11 (8-11), and 14 (12-15). Here, state 11 is chosen, and the bit width of 'g' is 2, resulting in a value of 1 (9-8). The next symbol is 'g'. The states corresponding to 'g' are 3 (8-15), 6 (0-3), and 9 (4-7). Here, state 3 is chosen, and the bit width of 'f' is 3, resulting in a value of 3 (11-8). And so on, the entire process is as follows: Figure 4 As shown, the final binary encoded bitstream is 0001010111000011 010111101000011110. Similarly, the process for "bfecgegcgef" is as follows. Figure 5 As shown, the final binary encoded bitstream is 11110001010110111001011010101. The two bitstreams above, when combined, become 0001010111000011 0101111010000111101111000101011011100101101010101.
[0075] As can be seen, this embodiment first obtains the sequence of symbols to be encoded corresponding to the data to be encoded, and determines the symbols to be encoded whose states do not depend on other symbol states from the sequence to obtain the target symbols to be encoded; then, based on the position of the target symbols to be encoded in the sequence, the sequence is split to obtain multiple sub-sequences to be encoded corresponding to the sequence; finally, each of the multiple sub-sequences to be encoded is encoded separately to obtain the encoded data. This embodiment obtains multiple sub-sequences by finding symbols in the sequence to be encoded that do not depend on other symbol states, that is, by using symbols with unique states as splitting points to split the sequence, which allows for parallel encoding of sub-sequences compared to serial encoding of the entire sequence, thus improving the encoding rate.
[0076] See Figure 6 As shown in the figure, this application also discloses an FSE encoding device, including:
[0077] The acquisition module 11 is used to acquire the sequence of symbols to be encoded corresponding to the data to be encoded, and to determine the symbols to be encoded from the sequence of symbols to be encoded whose symbol states do not depend on other symbol states, so as to obtain the target symbols to be encoded.
[0078] The splitting module 12 is used to split the symbol sequence to be encoded based on the position of the target symbol to be encoded in the symbol sequence to be encoded, so as to obtain multiple sub-sequences to be encoded corresponding to the symbol sequence to be encoded;
[0079] The encoding module 13 is used to encode the multiple sub-sequences to be encoded individually to obtain the encoded data to be encoded.
[0080] As can be seen, this embodiment first obtains the sequence of symbols to be encoded corresponding to the data to be encoded, and determines the symbols to be encoded whose states do not depend on other symbol states from the sequence to obtain the target symbols to be encoded; then, based on the position of the target symbols to be encoded in the sequence, the sequence is split to obtain multiple sub-sequences to be encoded corresponding to the sequence; finally, each of the multiple sub-sequences to be encoded is encoded separately to obtain the encoded data. This embodiment obtains multiple sub-sequences by finding symbols in the sequence to be encoded that do not depend on other symbol states, that is, by using symbols with unique states as splitting points to split the sequence, which allows for parallel encoding of sub-sequences compared to serial encoding of the entire sequence, thus improving the encoding rate.
[0081] In some specific embodiments, the acquisition module 11 specifically includes:
[0082] A frequency determination unit is used to determine the frequency of occurrence of each symbol to be encoded in the symbol sequence to be encoded.
[0083] The probability determination unit is used to calculate the normalized probability corresponding to each symbol to be encoded based on the occurrence frequency of each symbol to be encoded, the total number of symbols in the sequence of symbols to be encoded, and the length of the preset FSE code table used for encoding.
[0084] The symbol determination unit is used to determine, based on normalized probabilities, the symbols to be encoded from the sequence of symbols to be encoded in which the symbol states are not dependent on other symbol states, so as to obtain the target symbols to be encoded.
[0085] In some specific embodiments, the symbol determination unit is specifically used to determine the character to be encoded from the sequence of symbols to be encoded with a normalized probability absolute value of 1, so as to obtain the target symbol to be encoded.
[0086] In some specific embodiments, the splitting module 12 is specifically used to split each target symbol to be encoded at the beginning position of the symbol sequence to be encoded, so as to split each target symbol to be encoded from the symbol sequence to be encoded to obtain a first type of subsequence to be encoded with the same number of subsequences as the number of target symbols to be encoded, and a second type of subsequence to be encoded with one more subsequence than the number of target symbols to be encoded; wherein each first type of subsequence to be encoded corresponds to one target symbol to be encoded, and each second type of symbol to be encoded is a broken chain formed by a gap at the splitting position.
[0087] In some specific embodiments, the encoding module 13 specifically includes:
[0088] The encoding unit is used to encode multiple sub-sequences to be encoded individually using a preset FSE code table to obtain multiple encoded sub-sequences to be encoded.
[0089] The splicing unit is used to splice multiple encoded subsequences of the symbols to be encoded in the order of their arrangement to obtain the encoded data.
[0090] In some specific embodiments, the encoding unit is further configured to encode multiple sub-sequences to be encoded individually in parallel using multiple encoders; wherein one encoder corresponds to one sub-sequence to be encoded.
[0091] Furthermore, embodiments of this application also provide an electronic device. Figure 7This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0092] Figure 7 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the FSE encoding method disclosed in any of the foregoing embodiments.
[0093] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0094] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.
[0095] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the FSE encoding method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks. The data 223 may include the data to be encoded collected by the electronic device 20.
[0096] Furthermore, embodiments of this application also disclose a storage medium storing a computer program, which, when loaded and executed by a processor, implements the FSE encoding method steps disclosed in any of the foregoing embodiments.
[0097] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0098] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0099] The FSE encoding method, apparatus, device, and storage medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. An FSE encoding method, characterized in that, include: Obtain the sequence of symbols to be encoded corresponding to the data to be encoded, and determine the symbols to be encoded from the sequence of symbols whose symbol states do not depend on other symbol states, so as to obtain the target symbols to be encoded; The sequence of symbols to be encoded is split based on the position of the target symbol in the sequence of symbols to be encoded to obtain multiple sub-sequences to be encoded corresponding to the sequence of symbols to be encoded; Each of the multiple subsequences to be encoded is encoded individually to obtain the encoded data to be encoded. The step of splitting the symbol sequence to be encoded based on the position of the target symbol in the symbol sequence to be encoded to obtain multiple sub-sequences to be encoded corresponding to the symbol sequence to be encoded includes: Each target symbol to be encoded is split at the beginning position of the symbol sequence to be encoded, so as to split each target symbol to be encoded from the symbol sequence to obtain a first type of subsequence to be encoded with the same number of subsequences as the number of target symbols to be encoded, and a second type of subsequence to be encoded with one more subsequence than the number of target symbols to be encoded; wherein each first type of subsequence to be encoded corresponds to one target symbol to be encoded, and each second type of subsequence to be encoded is a broken chain formed by a gap at the split position.
2. The FSE encoding method according to claim 1, characterized in that, The step of determining, from the sequence of symbols to be encoded, symbols whose symbol states have no dependency relationship with other symbol states, to obtain the target symbol to be encoded, includes: Determine the frequency of occurrence of each symbol to be encoded in the sequence of symbols to be encoded, so as to obtain the normalized probability corresponding to each symbol to be encoded; Based on the normalized probability, the target symbol to be encoded is determined from the sequence of symbols to be encoded in which the symbol state has no dependency on other symbol states.
3. The FSE encoding method according to claim 2, characterized in that, After determining the frequency of occurrence of each symbol to be encoded in the symbol sequence to be encoded, the method further includes: The normalized probability corresponding to each symbol to be encoded is calculated based on the occurrence frequency of each symbol to be encoded, the total number of symbols in the sequence of symbols to be encoded, and the length of the preset FSE code table used for encoding.
4. The FSE encoding method according to claim 3, characterized in that, The step of determining, based on normalized probabilities, the symbols to be encoded from the sequence of symbols to be encoded in which the symbol states are independent of other symbol states, in order to obtain the target symbol to be encoded, includes: The target symbol to be encoded is obtained by determining the character to be encoded with a normalized probability absolute value of 1 from the sequence of symbols to be encoded.
5. The FSE encoding method according to claim 1, characterized in that, The step of individually encoding the multiple sub-sequences to be encoded to obtain the encoded data to be encoded includes: Multiple subsequences to be encoded are individually encoded using a preset FSE code table to obtain multiple encoded subsequences to be encoded. The multiple encoded subsequences to be encoded are then concatenated according to the order of the symbols to be encoded to obtain the encoded data to be encoded.
6. The FSE encoding method according to any one of claims 1 to 5, characterized in that, The step of individually encoding the multiple sub-sequences to be encoded includes: Multiple encoders are used to encode multiple sub-sequences to be encoded individually in parallel; wherein, one encoder corresponds to one sub-sequence to be encoded.
7. An FSE encoding device, characterized in that, include: The acquisition module is used to acquire the sequence of symbols to be encoded corresponding to the data to be encoded, and to determine the symbols to be encoded from the sequence of symbols whose symbol states do not depend on other symbol states, so as to obtain the target symbols to be encoded. The splitting module is used to split the symbol sequence to be encoded based on the position of the target symbol in the symbol sequence to be encoded, so as to obtain multiple sub-sequences to be encoded corresponding to the symbol sequence to be encoded; The encoding module is used to encode the multiple sub-sequences to be encoded individually to obtain the encoded data to be encoded. The splitting module is specifically used for: Each target symbol to be encoded is split at the beginning position of the symbol sequence to be encoded, so as to split each target symbol to be encoded from the symbol sequence to obtain a first type of subsequence to be encoded with the same number of subsequences as the number of target symbols to be encoded, and a second type of subsequence to be encoded with one more subsequence than the number of target symbols to be encoded; wherein each first type of subsequence to be encoded corresponds to one target symbol to be encoded, and each second type of subsequence to be encoded is a broken chain formed by a gap at the split position.
8. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the FSE encoding method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, Used to store computer-executable instructions, which, when loaded and executed by a processor, implement the FSE encoding method as described in any one of claims 1 to 6.