A multi-character limited entropy encoding method, device, equipment and readable medium

Abstract

This invention discloses a multi-character finite entropy encoding method, comprising: statistically analyzing the frequency of occurrence of all characters in a data block to be compressed and sorting them from largest to smallest; calculating the normalized state number of each character based on the frequency of occurrence, and setting a preset mark for the normalized state number within a preset range; creating a finite state entropy code table based on the normalized state number and sorting result of all characters, wherein the number of columns in each row of the finite state entropy code table is set to the absolute value of the normalized state number of the corresponding character in each row; filling the state number of the characters with the preset mark in reverse order at their corresponding positions in the finite state entropy code table, and calculating the state number of other characters at their corresponding positions in the finite state entropy code table using prime number expansion; and performing finite entropy encoding based on the finite state entropy code table. This invention also discloses a multi-character finite entropy encoding device, a computer device, and a readable storage medium.

Description

Technical Field

[0001] This invention relates to the field of encoding technology, and in particular to a multi-character finite entropy encoding method, apparatus, device, and readable medium. Background Technology

[0002] With the advent of the big data era, the demand for low latency in processing massive amounts of data is constantly increasing in specific application areas such as the Internet of Things and artificial intelligence, making lossless data compression technology increasingly important. Lossless data compression can be categorized into algorithms based on data statistics and algorithms based on dictionaries, according to their compression principles. Algorithms based on data statistics include Shannon-Fano coding, Huffman coding, arithmetic coding (AC), run-length coding (RLE), and finite-state entropy coding (FSE), among others. Algorithms based on dictionaries include LZ77 coding (Lempel-Ziv77) and LZ78 coding (Lempel-Ziv 78), among others.

[0003] To improve versatility, data compression schemes typically employ a hybrid approach using two or more compression algorithms. Zstandard (zstd), an open-source fast lossless compression algorithm from Facebook, is a hybrid algorithm composed of LZ77 encoding, Huffman coding, and finite-state entropy coding. Compared to other compression algorithms, zstd offers superior compression performance. Furthermore, zstd provides 22 compression levels to balance compression speed and compression ratio, and it has been widely adopted in applications such as the Linux kernel, FreeBSD operating system, and AWS Redshift data warehouse.

[0004] Lossless compression technology based on software implementation offers advantages such as high flexibility, universality, and low cost. However, software execution can only be performed sequentially, causing the central processing unit (CPU) to be occupied for extended periods when processing massive amounts of data. This significantly reduces compression speed and makes it difficult to meet the real-time compression processing needs of specific application areas. Hardware implementation is an effective way to solve these problems. Benefiting from the inherent parallel processing capabilities of hardware, it can improve transmission speed, resource utilization, and security.

[0005] Statistical analysis of the three components of the zstd algorithm—LZ77 encoding, Huffman encoding, and finite-state entropy encoding—shows a compression time ratio of approximately 4:1:1. While the proportion of finite-state entropy encoding is relatively small, it has a significant impact on zstd performance. Due to its use, zstd exhibits better compression performance than other hybrid compression algorithms. Furthermore, hardware acceleration solutions for LZ77 and Huffman encoding are relatively mature. Finite-state entropy encoding, as a novel compression algorithm, combines the precision of arithmetic encoding with the compression speed of Huffman encoding. Moreover, it can re-encode characters with decimal point accuracy and eliminates the need for multiplication and division to update the state during computation. Therefore, a hardware acceleration architecture for finite-state entropy encoding is crucial for achieving overall acceleration of the zstd algorithm and is an effective method to meet the needs of specific application domains. Summary of the Invention

[0006] In view of this, the purpose of this invention is to propose a multi-character finite entropy encoding method, apparatus, device, and readable medium. To meet the increasing demands for compression performance in specific application areas, this invention focuses on FSE in Zstd, employing a hardware-software co-implementation approach to comprehensively improve the compression and decompression speed of the Zstd algorithm. Following the Zstd specification standard, a hardware-compatible FSE compression and decompression architecture design is proposed, effectively reducing hardware overhead and improving hardware utilization. This solution for Zstd hardware acceleration design and implementation possesses theoretical foresight, feasibility, and practical application value.

[0007] To achieve the above objectives, one aspect of this invention provides a multi-character finite entropy encoding method, comprising the following steps: statistically analyzing the frequency of occurrence of all characters in the data block to be compressed and sorting them from largest to smallest; calculating the normalized state number of each character based on the frequency of occurrence, and setting a preset mark for the normalized state number within a preset range; creating a finite state entropy code table based on the normalized state number and sorting result of all characters, wherein the number of columns in each row of the finite state entropy code table is set to the absolute value of the normalized state number of the corresponding character in each row; filling the state number of the character with the preset mark in reverse order at the corresponding position in the finite state entropy code table, and calculating the state number of other characters at the corresponding position in the finite state entropy code table using prime number expansion; and performing finite entropy encoding based on the finite state entropy code table.

[0008] In some implementations, calculating the normalized state number of all characters based on their occurrence frequency includes: calculating the total number of states based on the total number of characters, and multiplying the ratio of the occurrence frequency of each character to the occurrence frequency of all characters by the total number of states to obtain the normalized state number corresponding to each character.

[0009] In some implementations, calculating the total number of states based on the total number of all characters includes calculating the total number of states using the following formula:

[0010]

[0011] Where, N state N represents the total number of states. symbol The number of all the characters.

[0012] In some implementations, setting a preset flag for the normalized state number within a preset range includes marking values ​​with a normalized state number less than 1 as -1.

[0013] In some implementations, filling the state number in reverse order at the corresponding position of the character with the preset mark in the finite state entropy code table includes: calculating the total number of states based on the total number of all characters, and filling the state number in reverse order at the corresponding position of the character with the preset mark in the finite state entropy code table based on the total number of states.

[0014] In some implementations, calculating the state number of other characters at their corresponding positions in the finite state entropy code table using prime number expansion includes: calculating the total number of states based on the total number of all characters, and calculating the diffusion interval based on the total number of states using the following formula:

[0015]

[0016] Where step is the diffusion interval, ts is the total number of states, and n is a preset prime number; the state sequence is diffused based on the diffusion interval to obtain a first sequence in which each value is less than the total number of states; the state numbers that have been filled in reverse order in the first sequence are deleted to generate a second sequence; the state numbers are taken from the second sequence based on the number of columns in each row of the finite state entropy code table and filled into the corresponding positions.

[0017] In some implementations, retrieving state numbers from the second sequence and filling them into corresponding positions based on the number of columns in each row of the finite state entropy code table includes: retrieving state numbers from the second sequence in ascending order of row number and deleting the retrieved state numbers from the second sequence; determining the number of state numbers to be retrieved based on the number of columns in each row, sorting the retrieved state numbers in ascending order, and filling them into the positions of each row sequentially.

[0018] In another aspect, this invention provides a multi-character finite entropy encoding apparatus, comprising: a first module configured to statistically analyze the frequency of occurrence of all characters in a data block to be compressed and encoded, and sort them from largest to smallest; a second module configured to calculate the normalized state number of each character based on the frequency of occurrence, and set a preset mark for the normalized state number within a preset range; a third module configured to create a finite state entropy code table based on the normalized state number and sorting result of all characters, wherein the number of columns in each row of the finite state entropy code table is set to the absolute value of the normalized state number of the corresponding character in each row; a fourth module configured to fill in the state number of the characters with the preset mark in reverse order at their corresponding positions in the finite state entropy code table, and calculate the state number of other characters at their corresponding positions in the finite state entropy code table using prime number expansion; and a fifth module configured to perform finite entropy encoding based on the finite state entropy code table.

[0019] In another aspect of the present invention, a computer device is provided, comprising: at least one processor; and a memory storing computer instructions executable on the processor, the instructions, when executed by the processor, implementing the steps of the above-described method.

[0020] In another aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0021] This invention offers at least the following beneficial technical effects: Addressing the current situation where finite state entropy (FSE) coding compression algorithms are relatively new and under-researched, this invention focuses on rapid implementation and convenient hardware mounting, effectively generating a finite state entropy code table. First, the data storage format is simple and convenient, eliminating the need for Huffman trees, binary trees, or linked lists, significantly reducing memory space. Second, the finite entropy coding algorithm designed in this scheme has simple computational steps, involving only comparison, addition, and shift operations, without other complex computational operations. Finally, all the computation and storage methods involved in this scheme not only improve the efficiency of software computation but are also hardware-encapsulated, offering multiple implementation options for greater application flexibility. If implemented in hardware, it can become a hardware acceleration technology for network data storage, accelerating the compression of finite entropy-coded data and effectively reducing the load on server CPUs. This solution conveniently solves the problem of finite state entropy code table generation, using only limited computation time and fewer adders and comparators, greatly reducing the hardware resource consumption for generating the finite state entropy coding code table. It allows for focused data compression acceleration, contributing to improved data center performance. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram illustrating an embodiment of the multi-character finite entropy encoding method provided by the present invention;

[0024] Figure 2 This invention provides a character probability statistics table for data to be compressed and encoded.

[0025] Figure 3 A schematic diagram of the table shape of the finite state entropy code table provided by the present invention;

[0026] Figure 4 A schematic diagram of the table portion of the finite state entropy code table provided by the present invention;

[0027] Figure 5 A schematic diagram of the final result of the finite state entropy code table provided by the present invention;

[0028] Figure 6 A schematic diagram of an embodiment of the multi-character finite entropy encoding device provided by the present invention;

[0029] Figure 7 A schematic diagram of an embodiment of the computer device provided by the present invention;

[0030] Figure 8 A schematic diagram illustrating an embodiment of the computer-readable storage medium provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0032] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0033] Finite-state entropy coding is a type of entropy coding for tANS (table asymmetric numeral systems) within asymmetric numeral systems (ANS). Existing research mainly focuses on tANS and uABS (uniform asymmetric binary systems), an important component of ANS, while research on FSE hardware architecture is scarce.

[0034] Based on the above objectives, a first aspect of the present invention proposes an embodiment of a multi-character finite entropy encoding method. Figure 1 This diagram illustrates an embodiment of the multi-character finite entropy encoding method provided by the present invention. Figure 1 As shown, the multi-character finite entropy encoding method of this invention includes the following steps:

[0035] 001. Count the frequency of occurrence of all characters in the data block to be compressed and encode, and sort them from largest to smallest.

[0036] 002. Calculate the normalized state number of all characters based on their frequency of occurrence, and set a preset flag for the normalized state number within a preset range;

[0037] 003. Create a finite state entropy code table based on the normalized state number and sorting result of all characters. The number of columns in each row of the finite state entropy code table is set to the absolute value of the normalized state number of the corresponding character in each row.

[0038] 004. Fill in the state numbers of the characters with preset markers in reverse order at their corresponding positions in the finite state entropy code table, and calculate the state numbers of other characters at their corresponding positions in the finite state entropy code table using prime number extension; and

[0039] 005. Finite entropy coding based on finite state entropy code table.

[0040] In some embodiments of the present invention, calculating the normalized state number of all characters based on their occurrence frequency includes: calculating the total number of states based on the total number of characters, and multiplying the ratio of the occurrence frequency of each character to the occurrence frequency of all characters by the total number of states to obtain the normalized state number corresponding to each character.

[0041] In some embodiments of the present invention, calculating the total number of states based on the total number of all characters includes: calculating the total number of states using the following formula:

[0042]

[0043] Where, N state N represents the total number of states. symbolThe total number of characters.

[0044] In some embodiments of the present invention, setting a preset flag for the normalized state number within a preset range includes: marking values ​​with a normalized state number less than 1 as -1.

[0045] In some embodiments of the present invention, filling the state number in reverse order at the corresponding position of the character with the preset mark in the finite state entropy code table includes: calculating the total number of states based on the total number of all characters, and filling the state number in reverse order at the corresponding position of the character with the preset mark in the finite state entropy code table based on the total number of states.

[0046] In some embodiments of the present invention, calculating the number of states of other characters at their corresponding positions in the finite state entropy code table through prime number extension includes: calculating the total number of states based on the total number of all characters, and calculating the diffusion interval based on the total number of states using the following formula:

[0047]

[0048] Where step is the diffusion interval, ts is the total number of states, and n is a preset prime number; the state sequence is diffused based on the diffusion interval to obtain a first sequence in which each value is less than the total number of states; the state numbers that have been filled in reverse order in the first sequence are deleted to generate a second sequence; the state numbers are taken from the second sequence based on the number of columns in each row of the finite state entropy code table and filled into the corresponding positions.

[0049] In some embodiments of the present invention, retrieving state numbers from the second sequence and filling them into corresponding positions based on the number of columns in each row of the finite state entropy code table includes: retrieving state numbers from the second sequence in ascending order of row number and deleting the retrieved state numbers from the second sequence; determining the number of state numbers to be retrieved based on the number of columns in each row, sorting the retrieved state numbers in ascending order, and filling them into the positions of each row sequentially.

[0050] The specific embodiments of the present invention are further described below with reference to specific examples. Taking the following example of a data block to be compressed and encoded:

[0051] They had hardly begun to do so,and the trembling people who hadhidden themselves were scarcely at home again,when Edward,the elder of thetwo exiled Princes,came over from Normandy with a few followers,to claim theEnglish Crown.His mother Emma,however,who only cared for her last sonHardicanute,instead of assisting him,as he expected,opposed him so stronglywith all her influence that he was very soon glad to get safely back.Hisbrother Alfred was not so fortunate.Believing in an affectionate letter,written some time afterwards to him and his brother,in his mother's name(butwhether really with or without his mother's knowledge is now uncertain),heallowed himself to be tempted over to England,with a good force of soldiers,and landing on the Kentish coast,and being met and welcomed by Earl Godwin,proceeded into Surrey,as far as the town of Guildford.Here,he and his menhalted in the evening to rest,having still the Earl in their company;who hadordered lodgings and good cheer for them.But, in the dead of the night, when they were off their guard, being divided into small parties sleeping soundly after a long march and a plentiful supper in different houses, they were setupon by the King's troops, and taken prisoners. Next morning they were drawn out in a line, to the number of six hundred men, and were barbarously tortured and killed; with the exception of every tenth man, who was sold into slavery. Asto the wretched Prince Alfred, he was stripped naked, tied to a horse and sentaway into the Isle of Ely, where his eyes were torn out of his head, and wherein a few days he miserably died. I am not sure that the Earl had willfully entrapped him, but I suspect it strongly.

[0052] First, by counting the frequency of characters appearing in the data block, we can obtain the following table:

[0053]

[0054]

[0055] Table 1. Character Frequency Statistics of Data to be Compressed

[0056] After calculating the probability of occurrence of all characters in the text, they are sorted, and then the normalized number of states is calculated. In this embodiment, a total of 44 characters are used, and the formula for calculating the total number of states is as follows:

[0057]

[0058] The normalization algorithm for each symbol is as follows:

[0059] In the example, it is 1765, N state Substituting 128 into the above formula, we get the following formula:

[0060]

[0061] After normalizing the character probabilities, the number of states in the table is obtained. At this point, states with normalized values ​​less than 1 are marked as -1. The resulting character probability statistics table for the data to be compressed is as follows: Figure 2 As shown, a schematic diagram of the shape of the finite-state entropy code table can be determined. Figure 3 As shown. Each row represents the state of a different character. The first row will contain the 20 states for the first character, the second row will contain the 11 states for the second character, and so on. Note that characters marked with -1 have a state count of 1 in the table, but the calculation rules are different from those for characters with a state count of 1.

[0062] For characters with a state count of -1, the filling scheme is to fill them in reverse order starting from the maximum state count. In this embodiment, the state counts of characters 25 to 44 are all marked as -1. Filling them in according to this pattern yields... Figure 4 The diagram shows a partial table of the finite state entropy code table.

[0063] Continuing to calculate the state values ​​in the finite state entropy code table, in order to disperse the state values ​​and ensure they have nonlinear and non-overlapping characteristics, a prime number diffusion method is used, and the diffusion interval is calculated using the following formula:

[0064]

[0065] Where step is the diffusion interval, ts is the total number of states, and n is a preset prime number; in this embodiment, ts is 128, and taking n as 5 as an example, step = 16 + 32 + 64 + 5 = 117. The sequence is diffused using step = 117, i.e., 117 * [1, 2, ..., 128]. The resulting data, after removing the portion exceeding 128, yields the following sequence:

[0066] [117, 106, 95, 84, 73, 62, 51, 40, 29, 18, 7, 124, 113, 102, 91, 80, 69, 58, 47, 36, 25, 14, 3, 120, 109, 98, 87, 76, 65, 54, 43, 32, 21, 10, 127, 116, 105, 94, 83, 72, 61, 50, 39, 28, 17, 6, 123, 112, 101, 90, 79, 68, 57, 46, 35, 24, 13, 2, 119, 108, 97, 86, 75] 64, 53, 42, 31, 20, 9, 126, 115, 104, 93, 82, 71, 60, 49, 38, 27, 16, 5, 122, 111, 100, 89, 78, 67, 56, 45, 34, 23, 12, 1, 118, 107, 96, 85, 74, 63, 52, 41, 30, 19, 8, 125, 114, 103, 92, 81, 70, 59, 48, 37, 26, 15, 4, 121, 110, 99, 88, 77, 66, 55, 44, 33, 22, 11, 0]

[0067] After removing the state values ​​that are -1 (i.e., the state values ​​occupied by characters 25 to 44), we get the following array:

[0068] [106, 95, 84, 73, 62, 51, 40, 29, 18, 7, 102, 91, 80, 69, 58, 47, 36, 25, 14, 3, 98, 87, 76, 65, 54, 43, 32, 21, 10, 105, 94, 83, 72, 61, 50, 39, 28, 17, 6, 101, 90, 79, 68, 57, 46, 35, 24, 13, 2, 97, 86, 75, 64, 5 3, 42, 31, 20, 9, 104, 93, 82, 71, 60, 49, 38, 27, 16, 5, 100, 89, 78, 67, 56, 45, 34, 23, 12, 1, 107, 96, 85, 74, 63, 52, 41, 30, 19, 8, 103, 92, 81, 70, 59, 48, 37, 26, 15, 4, 99, 88, 77, 66, 55, 44, 33, 22, 11, 0]

[0069] Extract the state numbers from the above sequence in ascending order of row number. The number of states extracted is determined by the column number of each row. Sort the extracted states in ascending order and fill them into their respective positions in each row. A schematic diagram of the final result of the finite state entropy code table is shown below. Figure 5 As shown.

[0070] It should be noted that the steps in each embodiment of the above-mentioned multi-character finite entropy encoding method can be interleaved, substituted, added, or deleted. Therefore, these reasonable permutations and combinations of the multi-character finite entropy encoding method should also fall within the protection scope of this invention, and the protection scope of this invention should not be limited to the embodiments.

[0071] Based on the above objectives, a second aspect of the present invention proposes a multi-character finite entropy encoding device. Figure 2 This diagram illustrates an embodiment of the multi-character finite entropy encoding device provided by the present invention. Figure 2 As shown, the multi-character finite entropy encoding device of this invention includes the following modules: a first module 011, configured to count the occurrence frequency of all characters in the data block to be compressed and encoded, and sort them from largest to smallest; a second module 012, configured to calculate the normalized state number of all characters based on their occurrence frequency, and set a preset mark for the normalized state number within a preset range; a third module 013, configured to create a finite state entropy code table based on the normalized state number and sorting result of all characters, wherein the number of columns in each row of the finite state entropy code table is set to the absolute value of the normalized state number of the corresponding character in each row; a fourth module 014, configured to fill in the state number of the characters with preset marks in reverse order at the corresponding positions in the finite state entropy code table, and calculate the state number of other characters at the corresponding positions in the finite state entropy code table through prime number expansion; and a fifth module 015, configured to perform finite entropy encoding based on the finite state entropy code table.

[0072] In view of the above objectives, a third aspect of the present invention provides a computer device. Figure 3 The diagram shown is a schematic representation of an embodiment of the computer device provided by the present invention. Figure 3 As shown, the computer device of this embodiment includes the following means: at least one processor 021; and a memory 022, the memory 022 storing computer instructions 023 that can be executed on the processor, the instructions implementing the steps of the above method when executed by the processor.

[0073] The present invention also provides a computer-readable storage medium. Figure 4 The diagram shown is a schematic representation of an embodiment of the computer-readable storage medium provided by the present invention. Figure 4 As shown, computer-readable storage medium 031 stores a computer program 032 that, when executed by a processor, performs the methods described above.

[0074] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program for the multi-character finite entropy encoding method can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The above computer program embodiments can achieve the same or similar effects as any of the corresponding foregoing method embodiments.

[0075] Furthermore, the method disclosed in the embodiments of the present invention can also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. When the computer program is executed by the processor, it performs the functions defined in the method disclosed in the embodiments of the present invention.

[0076] Furthermore, the above-described method steps and system units can also be implemented using a controller and a computer-readable storage medium for storing a computer program that enables the controller to perform the functions of the above-described steps or units.

[0077] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.

[0078] In one or more exemplary designs, functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on or transmitted via a computer-readable medium. Computer-readable media include computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. Storage media may be any available medium accessible to a general-purpose or special-purpose computer. By way of example, and not limitation, computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, or any other medium that may be used to carry or store the required program code in the form of instructions or data structures and is accessible to a general-purpose or special-purpose computer or a general-purpose or special-purpose processor. Furthermore, any connection may be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DOL), or wireless technologies such as infrared, radio, and microwave, then the aforementioned coaxial cable, fiber optic cable, twisted pair, DOL, or wireless technologies such as infrared, radio, and microwave are all included in the definition of media. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0079] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0080] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0081] The embodiment numbers disclosed in the above embodiments of the present invention are merely for description and do not represent the superiority or inferiority of the embodiments.

[0082] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0083] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A multi-character finite entropy encoding method, characterized in that, Includes the following steps: The frequency of occurrence of all characters in the data block to be compressed is counted and sorted from largest to smallest. The total number of states is calculated based on the total number of all characters. The ratio of the frequency of each character to the frequency of all characters is multiplied by the total number of states to obtain the normalized state number corresponding to each character. A preset flag is set for the normalized state number with a value less than 1. A finite state entropy code table is created based on the normalized state number and sorting result of all the characters, wherein the number of columns in each row of the finite state entropy code table is set to the absolute value of the normalized state number of the corresponding character in each row. The characters marked with the preset mark are filled with state numbers in reverse order at their corresponding positions in the finite state entropy code table, and the state numbers of other characters at their corresponding positions in the finite state entropy code table are calculated by prime number expansion; wherein, the calculation of the state numbers of other characters at their corresponding positions in the finite state entropy code table by prime number expansion includes: calculating the total number of states based on the number of all characters, and calculating the diffusion interval based on the total number of states using formula (1); spreading the state sequence based on the diffusion interval to obtain a first sequence in which each value is less than the total number of states; deleting the state numbers already filled in reverse order in the first sequence to generate a second sequence; taking state numbers from the second sequence in ascending order of row number, and deleting the taken state numbers from the second sequence; determining the number of state numbers to be taken based on the number of columns in each row, and sorting the taken state numbers in ascending order, and filling them into the positions of each row in turn; （1） in, For diffusion interval, The total number of states. Preset prime numbers; Finite entropy encoding is performed based on the aforementioned finite state entropy code table.

2. The multi-character finite entropy encoding method according to claim 1, characterized in that, The total number of states is calculated based on the number of all the characters, including: The total number of states is calculated using the following formula: ， in, The total number of states. The number of all the characters.

3. The multi-character finite entropy encoding method according to claim 1, characterized in that, Preset flags are set for the number of normalized states with values ​​less than 1, including: The normalized state number with a value less than 1 is marked as -1.

4. The multi-character finite entropy encoding method according to claim 1, characterized in that, The characters with the preset markers are used to fill in the state numbers in reverse order at the corresponding positions in the finite state entropy code table, including: The total number of states is calculated based on the total number of all characters, and the characters with the preset mark are filled in the state number in reverse order at the corresponding positions in the finite state entropy code table based on the total number of states.

5. A multi-character finite entropy encoding device, characterized in that, include: The first module is configured to count the frequency of occurrence of all characters in the data block to be compressed and encode, and sort them from largest to smallest. The second module is configured to calculate the total number of states based on the number of all characters, multiply the ratio of the frequency of each character to the frequency of all characters by the total number of states to obtain the normalized state number corresponding to each character, and set a preset flag for the normalized state number with a value less than 1. The third module is configured to create a finite state entropy code table based on the normalized state number and sorting result of all the characters, wherein the number of columns in each row of the finite state entropy code table is set to the absolute value of the normalized state number of the corresponding character in each row. The fourth module is configured to fill in the state numbers of the characters marked with the preset mark in reverse order at the corresponding positions in the finite state entropy code table, and calculate the state numbers of other characters at the corresponding positions in the finite state entropy code table by prime number expansion; wherein, the calculation of the state numbers of other characters at the corresponding positions in the finite state entropy code table by prime number expansion includes: calculating the total number of states based on the number of all characters, and calculating the diffusion interval based on the total number of states using formula (1); diffusing the state sequence based on the diffusion interval to obtain a first sequence in which each value is less than the total number of states; deleting the state numbers already filled in reverse order in the first sequence to generate a second sequence; taking out state numbers from the second sequence in ascending order of row number, and deleting the taken state numbers from the second sequence; determining the number of state numbers to be taken out based on the number of columns in each row, and sorting the taken state numbers in ascending order, and filling them into the positions of each row in turn; （1） in, For diffusion interval, The total number of states. Preset prime numbers; The fifth module is configured to perform finite entropy encoding based on the finite state entropy code table.

6. A computer device, characterized in that, include: At least one processor; as well as A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of the method according to any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-4.

Images