Data compression method, data decompression method, computing device, storage medium and program product

Abstract

The embodiment of the invention provides a data compression method, a data decompression method, computing equipment, a computer storage medium and a computer program product. The data compression method comprises the following steps: determining to-be-compressed data subjected to character coding according to a unified code coding format; reading character data from the to-be-compressed data according to a byte sequence, and determining a first dictionary or a second dictionary hit by the character data; searching coding subspaces respectively corresponding to the first dictionary or the second dictionary, and determining coding numerical values of dictionary data hit by the character data in the corresponding coding subspaces to obtain a coding numerical value group corresponding to the data to be compressed; and obtaining compressed data based on the coding numerical value group and the first dictionary. According to the technical scheme provided by the embodiment of the invention, the compression rate and the compression speed of data compression are ensured, and the compression performance is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a data compression method, a data decompression method, a computing device, a computer storage medium, and a computer program product. Background Art

[0002] With the rapid development of information technology and communication technology, the amount of data generated is increasing day by day. It costs a lot to store or transmit such a large amount of data.

[0003] Data compression is an effective way to reduce data storage costs and improve data transmission performance. Data compression converts compressed data into smaller data. By decompressing the compressed data, you can recover the original data.

[0004] Therefore, how to provide a data compression method with high compression performance has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The embodiments of the present invention provide a data compression method, a data decompression method, a computing device, a computer storage medium, and a computer program product, which improve data compression performance.

[0006] In a first aspect, an embodiment of the present invention provides a data compression method, comprising:

[0007] Determining data to be compressed that is character-encoded according to a Unicode encoding format;

[0008] Reading character data from the data to be compressed in byte order and determining a first dictionary or a second dictionary in which the character data matches; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus;

[0009] Searching the encoding subspace corresponding to the first dictionary or the second dictionary, respectively, and determining the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace, so as to obtain a set of encoding value groups corresponding to the data to be compressed;

[0010] Compressed data is obtained based on the encoding value group and the first dictionary.

[0011] In a second aspect, an embodiment of the present invention provides a data decompression method, comprising:

[0012] Get compressed data;

[0013] A first dictionary and a set of encoding values ​​are obtained from the compressed data; wherein the set of encoding values ​​is constructed by searching the encoding subspace corresponding to the first dictionary or the second dictionary, and according to the encoding values ​​of the dictionary data hit by the character data in the compressed data in the corresponding encoding subspace; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus;

[0014] The encoding subspace corresponding to the first dictionary or the second dictionary is searched, and the character data in the corresponding encoding subspace of the dictionary data hit by the encoding value group is determined to obtain the decompressed data corresponding to the encoding value group.

[0015] In a third aspect, an embodiment of the present invention provides a data compression device, including:

[0016] A data determination module, configured to determine the data to be compressed that is character-encoded according to the Unicode encoding format;

[0017] a data reading module, configured to read character data from the data to be compressed in byte order and determine a first dictionary or a second dictionary in which the character data matches; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus;

[0018] a first search module configured to search the encoding subspace corresponding to the first dictionary or the second dictionary, and determine the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace, so as to obtain a set of encoding value groups corresponding to the data to be compressed;

[0019] The first processing module is configured to obtain compressed data based on the encoding value group and the first dictionary.

[0020] In a fourth aspect, an embodiment of the present invention provides a data decompression device, comprising:

[0021] A data acquisition module, used for acquiring compressed data;

[0022] A second processing module is configured to obtain a first dictionary and a set of encoding values ​​from the compressed data; wherein the set of encoding values ​​is constructed by searching the encoding subspace corresponding to the first dictionary or the second dictionary, respectively, and according to the encoding values ​​of the dictionary data hit by the character data in the compressed data in the corresponding encoding subspace; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus;

[0023] The second search module is used to search the encoding subspace corresponding to the first dictionary or the second dictionary, determine the character data in the corresponding encoding subspace of the dictionary data hit by the encoding value group, and obtain the decompressed data corresponding to the encoding value group.

[0024] In a fifth aspect, an embodiment of the present invention provides a computing device, including a processing component and a storage component;

[0025] The storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the data compression method provided by the embodiment of the present invention, or to implement the data decompression method provided by the embodiment of the present invention.

[0026] In a sixth aspect, an embodiment of the present invention provides a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements the data compression method provided by an embodiment of the present invention, or implements the data decompression method provided by an embodiment of the present invention.

[0027] In the seventh aspect, an embodiment of the present invention provides a computer program product, which includes a computer program code. When the computer program code is executed by a computer, it implements the data compression method provided by the embodiment of the present invention, or implements the data decompression method provided by the embodiment of the present invention.

[0028] The embodiment of the present invention is directed to data to be compressed that is character-encoded according to the Unicode encoding format; character data is read from the data to be compressed in byte order, and a first dictionary or a second dictionary in which the character data hits is determined; wherein the first dictionary is constructed based on the data to be compressed; the second dictionary is constructed based on a text corpus; the encoding subspace corresponding to the first dictionary or the second dictionary is searched, and the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace is determined to obtain a group of encoding value groups corresponding to the data to be compressed; the encoding value group and the first dictionary are serialized to obtain compressed data. The technical solution of the embodiment of the present invention realizes lossless compression processing of data to be compressed that is character-encoded according to the Unicode encoding format, and combines the first dictionary constructed based on the data to be compressed and the second dictionary constructed based on the text corpus to ensure the compression rate and compression speed of data compression, thereby improving compression performance.

[0029] These and other aspects of the present invention will become more readily apparent from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 A flowchart of a data compression method provided by an embodiment of the present invention is schematically shown;

[0032] Figure 2 The following is a schematic diagram showing a flow chart of a data compression method provided in a practical application according to an embodiment of the present invention;

[0033] Figure 3 A flowchart of an embodiment of a data decompression method provided by an embodiment of the present invention is schematically shown;

[0034] Figure 4 The following is a schematic diagram showing a flow chart of a data compression / decompression process in a practical application of an embodiment of the present invention;

[0035] Figure 5 A schematic structural diagram of an embodiment of a data compression device provided by an embodiment of the present invention;

[0036] Figure 6 A schematic structural diagram of an embodiment of a data decompression device provided by an embodiment of the present invention;

[0037] Figure 7 A schematic structural diagram of an embodiment of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0039] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit "first" and "second" to be different types.

[0040] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0041] During the implementation of the present invention, the inventors discovered that traditional compression methods such as LZ4 (a lossless compression algorithm), Zstd (a fast lossless compression algorithm), and Gzip (a file compression algorithm) offer poor compression ratios and compression / decompression speeds for character data encoded in Unicode. Zlib and LZ4 are block compression technologies that compress data blocks of a set size. Because they are unaware of the characteristics of the compressed data (such as data type and value range), they are unable to achieve maximum compression.

[0042] Therefore, in order to improve the compression performance, the inventors proposed the technical solution of the embodiment of the present invention after a series of studies.

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] Figure 1 A flowchart of a data compression method provided by an embodiment of the present invention is schematically shown. Figure 1 As shown, the data compression method may specifically include the following steps:

[0045] 101: Determine the data to be compressed that is character-encoded according to the Unicode encoding format.

[0046] Unicode is an international standard that aims to provide a unified encoding scheme for almost all text in the world. The Unicode encoding format assigns a unique numerical value to each character in the world so that computers can understand and process these characters.

[0047] Unicode is usually represented by "U+" followed by the hexadecimal number of the character. For example, the Unicode code point of the Latin letter A is U+0041.

[0048] To facilitate transmission, Unicode characters can be encoded using different encoding formats, such as UTF-8, UTF-16, and UTF-32. These encoding formats encode characters in Unicode into byte sequences for easy storage and transmission.

[0049] UTF-8 is a variable-length encoding format used to represent characters in the Unicode character set. Each Unicode character can be encoded in UTF-8 using 1 to 4 bytes.

[0050] UTF-16 is a variable-length encoding format that uses 2 or 4 bytes to represent Unicode characters. It is suitable for processing a large number of non-ASCII (American Standard Code for Information Interchange, a character encoding standard based on the Latin alphabet) characters.

[0051] UTF-32 is a fixed-length encoding format. Each Unicode character can be encoded as a fixed 4-byte representation in UTF-16, and each character occupies the same storage space.

[0052] 102: Read character data from the data to be compressed in byte order, and determine a first dictionary or a second dictionary in which the character data matches; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus.

[0053] When character data is read from the data to be compressed in byte order, characters may be read one by one according to the byte representation of the data.

[0054] For example, in the data to be compressed in the UTF-8 encoding format, one character may occupy 1 to 4 bytes.

[0055] In a possible implementation of the present invention, reading character data from the data to be compressed in byte order can be implemented as follows:

[0056] First, an index variable is initialized to 0 to mark the byte position currently being read. Then, the data to be compressed is read in a loop until the end of the data to be compressed is reached.

[0057] The following steps can be performed in each loop:

[0058] a. Read the byte at the current index position.

[0059] b. Determine whether the high bit of the byte is 0. If so, it indicates that the byte is the starting byte of a single-byte character. Decode the byte into a character.

[0060] c. If the high-order bits of the byte begin with 110 (i.e., 0b110xxxxx), this indicates that the byte is the starting byte of a double-byte character. This byte and the next byte are decoded together as one character.

[0061] d. If the high-order bits of the byte begin with 1110 (i.e., 0b1110xxxx), this indicates that the byte is the start byte of a three-byte character. This byte and the following two bytes are decoded together as one character.

[0062] If the high-order bits of the byte begin with 11110 (i.e., 0b11110xxx), this indicates that the byte is the start byte of a four-byte character. This byte and the following three bytes are decoded together as one character.

[0063] f. Process the character data according to the decoding result.

[0064] g. Update the index variable to point to the next byte position to be read.

[0065] After the above loop steps are completed, multiple character data are read in sequence from the data to be compressed according to the byte order.

[0066] For each character data read, it is determined whether the character data hits the first dictionary or the second dictionary.

[0067] In an embodiment of the present invention, the first dictionary can be pre-constructed based on the data to be compressed. When constructing the first dictionary, repeated patterns, common items, or other features in the data to be compressed can be analyzed and recorded in the first dictionary. By extracting repeated items or frequently occurring segments from the data to be compressed to construct the first dictionary, subsequent compression processing can be facilitated, achieving efficient data compression.

[0068] The second dictionary can be constructed based on a text corpus. A large-scale text corpus can be used as an input data source for constructing the second dictionary. The text in the text corpus is statistically analyzed and processed to extract common words and phrases, which are then recorded in the second dictionary.

[0069] For each character data read, the character data can be matched with the dictionary data pre-written in the first dictionary and the second dictionary, and the matching result can be used to determine whether the character data matches the first dictionary or the second dictionary. For example, for the first character data, if the first dictionary data identical to the first character data is retrieved in the first dictionary, then it can be determined that the first character data matches the first dictionary; for the second character data, if the second dictionary data identical to the second character data is retrieved in the second dictionary, then it can be determined that the second character data matches the second dictionary. Optionally, the dictionary data in the first dictionary and the second dictionary can be encoded using the same encoding format as the character data.

[0070] 103: Searching the encoding subspace corresponding to the first dictionary or the second dictionary, and determining the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace, so as to obtain a set of encoding value groups corresponding to the data to be compressed.

[0071] The coding subspaces corresponding to the first dictionary and the second dictionary can be obtained by segmenting the target coding space; optionally, the dictionary data in the first dictionary and the second dictionary can be sorted from large to small according to importance, and the coding values ​​corresponding to the dictionary data with large importance are small.

[0072] In an embodiment of the present invention, the target coding space can be an integer range consisting of zero to a specified value, which can be determined in combination with the number of characters in the first dictionary, the second dictionary, and ASII characters. In a practical application, the target coding space can be, for example, 0 to 16383. Of course, the present invention is not limited to this.

[0073] The first dictionary typically contains repeated patterns or common items in the data to be compressed, while the second dictionary typically contains a wide range of linguistic information. Therefore, the encoding subspaces corresponding to the first and second dictionaries can be constructed by selecting different encoding ranges from the target codespace. In this way, the encoding subspaces corresponding to the first and second dictionaries are subsets of the target codespace.

[0074] Optionally, the target coding space may be segmented according to different rules or features to obtain multiple different coding subspaces. The first dictionary may correspond to at least one coding subspace, and the second dictionary may correspond to at least one coding subspace.

[0075] Each dictionary data in the first dictionary and the second dictionary corresponds to a code value in the code subspace. After determining the code subspaces corresponding to the first and second dictionaries from the target code space, a mapping relationship between the dictionary data in the first and second dictionaries and the corresponding code values ​​can be constructed. The dictionary data in the first and second dictionaries correspond one-to-one to the code values ​​in their respective code subspaces.

[0076] In data compression, it is often desirable to map the compressed data to smaller code values ​​to reduce the data's representation size. Therefore, in the first and second dictionaries, the dictionary data can be sorted from most important to least important, with more important dictionary data corresponding to smaller code values. The importance of the dictionary data can be related to its frequency of occurrence, with more frequently occurring dictionary data having greater importance.

[0077] In another possible implementation of the present invention, the importance of the dictionary data may also be related to the number of bytes that can be saved by the dictionary data compared to the data to be compressed, wherein the more bytes saved, the greater the importance, and accordingly, the dictionary data with greater importance may correspond to a smaller encoding value; the number of bytes saved can be determined by the difference between the number of bytes of the dictionary data and the number of bytes of the character data corresponding to the dictionary data.

[0078] In one embodiment of the present invention, the importance of the dictionary data of the first dictionary may refer to the number of bytes saved by compression, that is, the difference between the number of bytes of the dictionary data and the number of bytes of the character data corresponding to the dictionary data; the importance of the dictionary data of the second dictionary may be the frequency of occurrence of the dictionary data.

[0079] Of course, the calculation method of the importance degree can also be implemented in other ways, which will be described in detail in the following embodiments.

[0080] After reading the character data of the data to be compressed, the encoding value corresponding to the dictionary data can be found in at least one sub-encoding space of the target encoding space through the first dictionary or the second dictionary hit by each character data, thereby obtaining a group of encoding value groups corresponding to the character data contained in the data to be compressed.

[0081] 104. Obtain compressed data based on the encoding value group and the first dictionary.

[0082] In order to facilitate storage or transmission, optionally, after obtaining a set of coded value groups corresponding to the character data contained in the data to be compressed, the coded value groups can be serialized to obtain compressed data. Among them, serialization can be a process of converting a data structure or object into a format that can be stored or transmitted. There are many serialization technologies for serialization, such as JSON (JavaScript Object Notation, a lightweight data exchange format) serialization, Java native serialization, Varint (a variable-length integer encoding) serialization, etc., which are not limited in the present invention. By serializing the coded array, a compact representation of the data can be achieved, the storage space occupied can be reduced, and the transmission and processing of the data can be facilitated.

[0083] In some embodiments of the present invention, after serializing the encoded value group and the first dictionary, the generated serialized data can be further compressed using traditional compression methods, such as gzip, zlib, Lz4, Zstd, etc., to further improve the compression effect.

[0084] The resulting compressed data can be stored in a storage medium and can also be transmitted to other locations via a network to achieve effective data management and transmission.

[0085] In some embodiments, the method may further include:

[0086] If the character data does not match the first dictionary or the second dictionary, determining the ASCII character that the character data matches;

[0087] Search the encoding subspace corresponding to the ASCII character and determine the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace; wherein the encoding subspace corresponding to the ASCII character is obtained by segmenting the target encoding space; and the encoding value corresponding to at least part of the dictionary data in the first dictionary and the second dictionary is smaller than the encoding value corresponding to the ASCII character.

[0088] In actual application, the dictionary data included in the first dictionary and the second dictionary may not include all the character data in the data to be compressed.

[0089] When the dictionary data of the first dictionary and the second dictionary cannot contain all the character data in the data to be compressed, that is, the character data cannot match any one of the first dictionary and the second dictionary, the ASCII characters matched by the character data can be determined.

[0090] ASCII characters are a basic character encoding standard used to represent English letters, numbers, punctuation marks, and some control characters. The ASCII character set contains 128 characters, each represented by a 7-bit binary value ranging from 0 to 127.

[0091] In an embodiment of the present invention, the encoding subspace corresponding to ASCII characters can be constructed by selecting a portion of the encoding range from the target encoding space.

[0092] After selecting a code range for ASCII characters from the target code space, a mapping relationship between ASCII characters and their corresponding code values ​​can be constructed. ASCII characters correspond one-to-one with their code values ​​in the corresponding code subspace.

[0093] In addition, as another possible implementation, the method may further include:

[0094] When the character data does not match the first dictionary and the second dictionary, the character data is used as the encoding value.

[0095] In the case that the character data does not match the first dictionary and the second dictionary, in order to ensure the integrity of the data to be compressed and to retain as much information as possible in subsequent processing, the character data may be directly used as the encoding value.

[0096] In a possible implementation of the present invention, if the character data does not match the first dictionary and the second dictionary, the character data can be directly used as the encoding value.

[0097] In some embodiments, the first dictionary may be generated as follows:

[0098] Split the data to be compressed;

[0099] A first dictionary is constructed based on the multiple character strings obtained by segmentation.

[0100] In a possible implementation of the present invention, the data to be compressed may be segmented according to a preset rule to obtain multiple character strings.

[0101] The preset rules may include, for example, segmentation according to spaces, punctuation marks, or other separators.

[0102] In some embodiments, constructing a first dictionary based on the multiple character strings obtained by segmentation may include:

[0103] Count the occurrence frequencies of multiple strings in the data to be compressed;

[0104] Calculate the importance of multiple character strings based on their respective occurrence frequencies and character lengths;

[0105] A first number of character strings are selected in descending order of importance to generate a first dictionary.

[0106] First, the data to be compressed can be traversed, and the number of times each string appears in the data to be compressed can be counted to obtain the frequency of occurrence of each string.

[0107] In a possible implementation of the present invention, the importance of each character string may be obtained by multiplying the occurrence frequency of the character string by the character length.

[0108] In addition, as another possible implementation, the importance of multiple character strings can be calculated based on the occurrence frequencies and character lengths of the multiple character strings, which can be specifically implemented as follows:

[0109] The importance of the plurality of character strings is calculated according to the occurrence frequencies, character lengths and predetermined compressed byte numbers respectively corresponding to the plurality of character strings.

[0110] The importance of a character string can be calculated using the following formula (1).

[0111] score=freq*(len-code_widh)-(len+code_widh); (1)

[0112] Wherein, score may indicate the importance of a character string, freq may indicate the frequency of occurrence, code_widh may indicate the predetermined number of compressed bytes, which may be 1, and len may indicate the character length.

[0113] In some embodiments, the method may further include:

[0114] Searching the encoding subspace corresponding to the first dictionary in descending order of importance to determine the first compressed byte numbers corresponding to the plurality of character strings;

[0115] Determining first compression ratios corresponding to the plurality of character strings based on first compressed byte numbers corresponding to the plurality of character strings;

[0116] Querying the encoding subspace corresponding to the second dictionary to determine the second compressed byte numbers corresponding to the plurality of character strings;

[0117] Determining second compression ratios corresponding to the plurality of character strings based on second compressed byte numbers corresponding to the plurality of character strings;

[0118] A character string having a first compression ratio greater than a second compression ratio is deleted from the plurality of character strings.

[0119] Among them, the first compressed byte number can refer to the storage space occupied by the encoding value corresponding to the character data in the first dictionary, that is, the character length; the second compressed byte number can refer to the storage space occupied by the encoding value corresponding to the character data in the second dictionary, that is, the character length.

[0120] The first compression ratio can be calculated based on the first compressed byte number and the storage space occupied by the character data itself, that is, the character length;

[0121] The second compression ratio can be calculated based on the second compressed byte count and the storage space occupied by the character data itself.

[0122] The first compression ratio and the second compression ratio can be calculated using the following formula (2).

[0123]

[0124] Wherein, CR may indicate a compression ratio, compressed size may indicate the number of compressed bytes, and uncompressed size may indicate the character length of the character data.

[0125] The first compression ratio is greater than the second compression ratio, indicating that the first compressed byte count is greater than the second compressed byte count, that is, the storage space occupied by the encoding value determined by the character string through the first dictionary is greater than the storage space occupied by the encoding value determined by the second dictionary, indicating that the compression performance of the character string compressed using the first dictionary is weaker than the compression performance using the second dictionary. Therefore, the character string with the first compression ratio greater than the second compression ratio can be deleted from multiple character strings without being added to the first dictionary.

[0126] In some embodiments, selecting a first number of character strings in descending order of importance to generate the first dictionary may include:

[0127] recalculating the importance of the multiple character strings according to the first compressed byte numbers corresponding to the multiple character strings, combined with the occurrence frequencies and character lengths corresponding to the multiple character strings;

[0128] A first number of character strings are selected in descending order of recalculated importance to generate a first dictionary.

[0129] Since the first compressed byte count is the actual compressed byte count of the character string, the importance levels of the multiple character strings can be recalculated based on the first compressed byte count.

[0130] The importance of the character string can be recalculated using the following formula (3).

[0131] score=freq*(len-code_1)-(len+code_1); (3)

[0132] Wherein, score_new may indicate the importance of the recalculated character string, freq may indicate the frequency of occurrence, and code_1 may indicate the number of first compressed bytes.

[0133] After sorting the multiple strings from largest to smallest according to the recalculated importance, a first number of strings with higher importance can be selected from the sorted queue to form a first dictionary. The specific value of the first number can be flexibly set by those skilled in the art based on actual application requirements and is not specifically limited in the embodiments of the present invention.

[0134] In some embodiments, the segmentation process of the data to be compressed may be specifically implemented as follows:

[0135] Read character data from the data to be compressed in byte order and save the character data as a string;

[0136] Query the number of occurrences of the current string;

[0137] If the number of occurrences is greater than 1, accumulate the number of occurrences; if the number of occurrences is less than or equal to 1, merge the current string with the next string and accumulate the number of occurrences of the merged string.

[0138] The following is a specific example to schematically illustrate a specific implementation method of splitting the compressed data. It should be noted that the following example is only used to help those skilled in the art understand the implementation method of the embodiment of the present invention, and does not make any improper limitation to the present invention.

[0139] Assume that the data to be compressed may be "aababc".

[0140] When the data to be compressed is segmented, character data in the data to be compressed may be read sequentially according to byte order. In this example, a, a, b, a, b, c may be read sequentially.

[0141] When reading character data in byte order, when character a is read, the occurrence count of the string is 0, which is less than 1. Since a is the first character, the occurrence count of a is accumulated, recorded as 1, and the next character string is read. The next character string is a, and the occurrence count of a is greater than or equal to 1. The occurrence count of a is accumulated, and the occurrence count of a is recorded as 2. The next character string b is read. The occurrence count of b is less than 1. The current character string b is merged with the previously saved character string a to generate a merged character string ab, and the occurrence count of the merged character string ab is accumulated. The occurrence count of the merged character string ab is now 1. The next character string is read. The next character string is a, and the occurrence count of a is accumulated, and the occurrence count of a is recorded as 3. The next character string b is read and merged with character a and character b to generate a merged character string ab. The occurrence count of the merged character string ab is accumulated, and the occurrence count of the merged character string ab is recorded as 2. The next character string c is read. The occurrence count of c is less than 1. The character string c is merged with the previously saved character string ab to generate a merged character string abc, and the occurrence count of the merged character string abc is accumulated.

[0142] After the above segmentation process, the data to be compressed aababc can be segmented into a3, ab2, and abc1. Among them, a3 means that the number of times a appears in the data to be compressed is 3, ab2 means that the number of times a appears in the data to be compressed is 2, and abc means that the number of times abc appears in the data to be compressed is 1.

[0143] In some embodiments, the method may further include:

[0144] If the currently read character data is a delimiter, at least one character data separated by the delimiter is saved as a character string.

[0145] In the embodiment of the present invention, the separator may include special symbols such as spaces and punctuation marks, or a specific character may be set as the separator.

[0146] Exemplarily, the data to be compressed can be abc / ab, where there is a special symbol " / " between c and a. The special symbol can be a separator. When the separator is read, the character data before and after the separator can be saved as strings respectively, or only the character data after the separator can be saved as a string, or the character data before the separator can be saved as a string, etc.

[0147] In the above example, the segmentation results of the data abc / ab to be compressed may be abc1 and ab1. Of course, the above two methods of obtaining character strings can be used in combination. For example, assuming the data abc / ab to be compressed, the segmentation results may be: a2, b1, c1, abc1, ab2.

[0148] In some embodiments, the second dictionary may be generated as follows:

[0149] Count the occurrence frequency of each character in the text corpus; sort each character in the text corpus in descending order of frequency;

[0150] Generate a second dictionary based on the sorting results.

[0151] In one possible implementation of the present invention, a text corpus can be traversed, and the number of times each character appears in the text corpus can be counted to obtain the frequency of occurrence of each character. The frequency of occurrence of each character obtained by counting can then be sorted in descending order. Based on the sorting result, a second number of characters can be selected in descending order of frequency of occurrence to generate a second dictionary. The frequency of occurrence represents the degree of importance. Specifically, the selected first number of characters can be added to the second dictionary, and their corresponding degree of importance can be recorded. For example, the characters and the frequency of occurrence can be stored as key-value pairs, or the characters and the frequency of occurrence can be saved in an ordered list to obtain the second dictionary.

[0152] A text corpus can refer to a collection of text data. A text data collection can contain text from different sources, such as news articles, novels, web content, social media posts, etc.

[0153] In one possible implementation of the present invention, a subtext corpus can be obtained by filtering from the text corpus, and then a second dictionary can be generated by counting the frequency of occurrence of each character in the subtext corpus. The filtering condition can be, for example, filtering text from the text corpus that is in the same field as the data to be compressed.

[0154] By screening the text corpus, texts related to the field of the data to be compressed can be obtained, thereby increasing the relevance of the generated second dictionary to the data to be compressed and improving compression efficiency. In addition, texts unrelated to the data to be compressed can be excluded, reducing the impact of noise on the generation of the second dictionary.

[0155] In addition, in order to further improve the compression effect, in some embodiments, obtaining compressed data based on the first dictionary and the encoding value group can be specifically implemented as follows:

[0156] Determine N consecutive code values ​​in the code value group that meet the merging condition;

[0157] Combining the N code values ​​and merging the header identification data to obtain a combined value; wherein the header identification data is used to indicate the number of bits corresponding to the N code values; the merging condition is that the number of bytes occupied by the combined code value is less than the sum of the number of bytes occupied by the N code values;

[0158] The merged value, the remaining coded values ​​in the coded value group that do not meet the merge condition, and the first dictionary are serialized to obtain compressed data.

[0159] The merged value is obtained by searching for N consecutive (N is 2 or 3) coded values ​​that meet the merging condition. Since the number of bytes occupied by the merged value is less than the sum of the number of bytes occupied by the N coded values, the compression performance can be further improved.

[0160] In some embodiments, when the Varint serialization method is used for serialization, the merging condition may be specifically as follows:

[0161] The number of bits of the N encoded values ​​is greater than or equal to 8, and the number of bits of the combined value is M times of 7, where M is greater than 2.

[0162] Varint is a variable-length integer encoding method. The basic rule of Varint encoding is that an integer value can be represented using 7 consecutive binary bits. Since 7 bits of binary data can represent a maximum value of 127, only one byte is required for integers less than 128. Integers between 128 and 16383 require two bytes; integers greater than 16383 require at least three bytes after Varint encoding. The most significant bit of the first byte is 1, and the remaining 7 bits plus the 7 bits of the second byte together form the binary representation of the integer.

[0163] Therefore, for an 8-bit binary data, it takes up 2 bytes after Varint encoding;

[0164] For 21-bit binary data, Varint encoding requires at least 3 bytes.

[0165] Similarly, for a 28-bit binary data, it takes up at least 4 bytes after using Varint encoding.

[0166] Therefore, when the number of bits of the N coded values ​​is greater than or equal to 8, each coded value will occupy 2 bytes after Varint encoding, and the N coded values ​​will occupy 2N bytes. After the merging process, the combined value obtained will occupy M bytes after Varint encoding, and 2N is greater than M. Optionally, N is greater than or equal to 2. When N is 2, M can be 3; when N is 3, M can be 4, and so on.

[0167] To facilitate the merging operation, in some embodiments, determining N consecutive coded values ​​in a coded value group that meet the merging condition may be specifically implemented as follows:

[0168] Query the combined coding table corresponding to the N coding values, and determine the N consecutive coding values ​​in the coding value group that hit any group of bit combinations in the combined coding table;

[0169] According to an embodiment of the present invention, combining N coded values ​​and merging the header identification data to obtain the merged value can be specifically implemented as follows:

[0170] Find the header identification data corresponding to the bit combination in the combined coding table;

[0171] According to the merging method in the merged coding table, the N coding values ​​are combined and the header identification data is merged to obtain a merged value.

[0172] In order to further reduce the storage space occupied by the coded value group, at least one group of values ​​in the coded value group may be merged.

[0173] The merging of the coding values ​​can be performed when the coding value group contains N consecutive coding values ​​that hit any group of bit position combinations in the merged coding table.

[0174] In an embodiment of the present invention, a merged code table can be maintained in advance, and the merged code table records at least one set of bit position combinations that meet the merge conditions. When determining N code value groups that meet the merge conditions in the code value group, the code value groups can be traversed in order, N code values ​​are selected in turn, and it is determined whether the N code values ​​hit any set of bit position combinations in the merged code table. Each set of bit position combinations in the merged code table provides the possible bit position combinations corresponding to N consecutive code values ​​according to their order of appearance in the data to be compressed.

[0175] For the N code values ​​that meet the merging condition, the N code values ​​may be merged, and then the next group of N code values ​​may be read until the code value group is traversed.

[0176] The merging coding table may also record a merging method corresponding to each arrangement order and header identification data.

[0177] When merging N coding values, the N coding values ​​can be merged according to the merging method recorded in the merged coding table to generate a merged coding value. In addition, the header identification data corresponding to the hit bit position combination can be obtained from the merged coding table, and the header identification data and the coding value generated by the merger can be merged to obtain a merged value.

[0178] In an example, for example, there are three consecutive coded values ​​code1, code2, and code3, where the value of code1 is 10, which is represented as 00001010 in binary, the value of code2 is 7, which is represented as 00000111 in binary, and the value of code3 is 15, which is represented as 00001111 in binary. Code1 occupies 8 bits, code2 occupies 8 bits, and code3 occupies 8 bits. The three coded values ​​are serialized using Varint serialization. Each coded value will occupy 2 bytes after Varint encoding. Assuming that the three consecutive coded values ​​meet the merging conditions, the three consecutive coded values ​​can be merged according to the merging method recorded in the merged coding table.

[0179] For example, if the merging method is code1<<16|code2<<8|code3 (| represents an OR operation), the result of shifting code left by 16 bits is added to the result of shifting code2 left by eight bits, and then the result of the addition is added to code3 to obtain the merged result.

[0180] Specifically:

[0181] Shift code1 left by 16 bits and we get 00001010 00000000 00000000.

[0182] Shift code2 left by 8 bits to get 00000000 00000111 00000000.

[0183] Directly use the original binary representation of code3 as 00000000 00000000 00001111.

[0184] Add the above three results to get the combined result combined_code 00001010 0000011100001111.

[0185] Furthermore, after obtaining the merging result, it is also necessary to merge the header identification data Header based on the merging result. For example, the header identification data may be

[0000] , and the final merging result may be 00000000 0000000000000000000001010 00000111 00001111.

[0186] In the above example, Code 1 is represented using 8 bits, Code 2 is represented using 8 bits, and Code 3 is represented using 8 bits. The combined code header consists of a fixed 4-bit sequence

[0000] , while the combined code, combined_code, consists of 24 bits: 00001010 00000111 00001111. The final combined code is 00000000 00000000 00000000 00001010 00000111 00001111, which is 28 bits and occupies 4 bytes. This further improves compression performance compared to the 6 bytes required to serialize the three code values ​​separately.

[0187] In a practical application, N can be 2, then M can be 3, and the combined value is 21 bits. Then the combined coding table corresponding to the two coding values ​​can be shown in Table 1 below:

[0188] Table 1:

[0189]

[0190] Among them, code 1 can represent the first code value of two consecutive code values, code 2 can represent the second code value, Header can represent header identification data, combined_code can represent the merging result of code 1 and code 2, that is, the merged value. In Table 2, "<<" represents a bit shift operation. For example, code1<<8 means shifting code 1 left by 16. The specific implementation of the merging method can be found in the above description and will not be repeated here.

[0191] In another practical application, N can be 3, M can be 4, and the combined value is 28 bits. Then the combined coding table corresponding to the three coding values ​​can be shown in Table 2 below:

[0192] Table 2:

[0193]

[0194] Among them, code one can represent the first code value among three consecutive code values, code two can represent the second code value, code three can represent the third code value, Header can represent header identification data, and combined_code can represent a combined value.

[0195] It should be noted that Table 1 and Table 2 merely provide possible implementations of the embodiments of the present invention, and the present invention is not limited thereto.

[0196] In some embodiments, the coding subspace may be generated as follows:

[0197] Divide the target encoding space into multiple encoding subspaces;

[0198] Determine a plurality of data segments obtained by segmenting the first dictionary and the second dictionary respectively;

[0199] According to the allocation principle that dictionary data with greater importance corresponds to smaller coding values, coding subspaces are respectively allocated to the multiple data segments from the multiple coding subspaces.

[0200] The target coding space may represent a usable coding range. By dividing the target coding space into multiple coding subspaces, a certain range of coding values ​​may be assigned to each coding subspace.

[0201] According to an embodiment of the present invention, the target coding space may be generated by the following operations:

[0202] Count the number of characters contained in the first dictionary, the second dictionary and the ASII character set;

[0203] Generate target encoding space based on the number of characters.

[0204] In the embodiment of the present invention, the entire coding space may be evenly divided into a plurality of coding subspaces as required, but the invention is not limited thereto. The plurality of coding subspaces may also be unevenly divided according to different requirements.

[0205] Since dictionary data is arranged according to importance in the dictionary, the first coding subspace can be optionally assigned to the first data segment in the first dictionary; the second coding subspace can be assigned to the first data segment in the second dictionary, etc. Each dictionary data in a data segment corresponds to a coding value, and a greater importance corresponds to a smaller coding value.

[0206] Optionally, the ASII characters may be divided into multiple data segments, and encoding subspaces may be allocated to the multiple data segments in multiple encoding subspaces.

[0207] In order to facilitate the search, a coding space mapping table can be established according to the above allocation results, and the coding subspace corresponding to the first dictionary or the second dictionary or the ASII character can be determined by searching the coding space mapping table.

[0208] Table 3 below schematically shows a coding space mapping table provided by an embodiment of the present invention in a practical application: Assuming the target coding space is 0-16383, the first dictionary can store 280 dictionary data and the second dictionary can store 8192 dictionary data.

[0209] Table 3

[0210]

[0211] As shown in Table 3, the target coding space can be 0 to 16383. Assume that the target coding space can be divided into 8 coding subspaces, and each coding subspace can be a subset of the target coding space.

[0212] In Table 3, the coding segment number is used to identify different coding subspaces. Taking the first coding subspace 0 to 7 as an example, it corresponds to 1 to 8 of the first dictionary, that is, the data segment composed of the 1st to 8th dictionary data. According to the principle that the greater the importance, the smaller the coding value, the coding value corresponding to the first dictionary data is 0, the coding value corresponding to the second dictionary data is 1... and so on, the coding value corresponding to the 8th dictionary data is 8. Similarly, for each coding subspace in Table 2, the coding value corresponding to the dictionary data or ASII character in the corresponding dictionary can be determined, and examples will not be repeated here.

[0213] In Table 3, the first coding subspace, that is, coding subspace 0-7 corresponding to coding segment number 0, is assigned to data segments 1-8 of the first dictionary; the second coding subspace, that is, coding subspace 8-31 corresponding to coding segment number 1, is assigned to data segments 1-24 of the second dictionary. Similarly, the eighth coding subspace, that is, coding segment number 7, is assigned to data segments 1-24 of the second dictionary.

[0214] It should be noted that the division of the coding subspace in Table 2 and the division of the dictionary data or ASII character data segments are only one possible implementation of the present invention, and the present invention is not limited thereto.

[0215] For ease of understanding, Figure 2 The flowchart schematically shows a data compression method provided by an embodiment of the present invention in a practical application.

[0216] like Figure 2 As shown, for the data to be compressed 201 , firstly, a statistical analysis can be performed on the data to be compressed to obtain a first dictionary 202 .

[0217] After the first dictionary 202 is generated, a second dictionary 203 may be obtained. The second dictionary 203 may be constructed based on a text corpus.

[0218] When compressing the data to be compressed 201, character data is first read from the data to be compressed 201 in sequence according to byte order, and the first dictionary 202 or the second dictionary 203 in which the character data hits is determined. Then, the encoding subspace corresponding to the first dictionary 202 or the second dictionary 203 is searched, and an encoding value group 204 corresponding to multiple character data is obtained.

[0219] After obtaining the code value group 204 , in order to further reduce the space occupied by the code value group 304 , at least one set of code values ​​in the code value group 204 may be merged to obtain a merged code value group 205 .

[0220] Then, the combined encoding value group 205 and the first dictionary 202 may be serialized to obtain compressed data 206 .

[0221] Figure 3 The flowchart of an embodiment of a data decompression method provided by an embodiment of the present invention is schematically shown. Figure 4 As shown, the data decompression method may specifically include the following steps:

[0222] 301: Get compressed data;

[0223] 302: Obtain a first dictionary and a set of encoded values ​​from the compressed data.

[0224] Among them, the encoding value group is constructed by searching the encoding subspace corresponding to the first dictionary or the second dictionary respectively, and the encoding value of the dictionary data hit by the character data in the data to be compressed in the corresponding encoding subspace; among them, the first dictionary is constructed based on the data to be compressed; the second dictionary is constructed based on the text corpus.

[0225] When the compressed data is obtained by serializing the first dictionary and the encoding value group, the compressed data may be first deserialized to obtain the first dictionary and the encoding value group.

[0226] 303: Searching the encoding subspace corresponding to the first dictionary or the second dictionary, and determining the character data of the dictionary data hit by the encoding value group in the corresponding encoding subspace to obtain the decompressed data corresponding to the encoding value group.

[0227] Among them, compressed data can be used Figure 1 The data compression method shown is to compress the data to be compressed, which will not be described in detail here.

[0228] In an embodiment of the present invention, the first dictionary can be pre-constructed based on the data to be compressed. When constructing the first dictionary, repeated patterns, common items, or other features in the data to be compressed can be analyzed and recorded in the first dictionary. By extracting repeated items or frequently occurring segments from the data to be compressed to construct the first dictionary, subsequent compression processing can be facilitated, achieving efficient data compression.

[0229] The second dictionary can be constructed based on a text corpus. A large-scale text corpus can be used as an input data source for constructing the second dictionary. The text in the text corpus is statistically analyzed and processed to extract common words and phrases, which are then recorded in the second dictionary.

[0230] The encoding value group can be obtained by matching the character data in the data to be compressed with the dictionary data pre-written in the first dictionary and the second dictionary respectively, and determining whether the character data hits the first dictionary or the second dictionary based on the matching results. Then, the encoding subspace corresponding to the first dictionary or the second dictionary is searched, and the encoding value of the dictionary data hit by the character data in the data to be compressed in the corresponding encoding subspace is constructed. For example, for the first character data, the first dictionary data identical to the first character data is retrieved in the first dictionary, then it can be determined that the first character data hits the first dictionary; for the second character data, the second dictionary data identical to the second character data is retrieved in the second dictionary, then it can be determined that the second character data hits the second dictionary. Optionally, the dictionary data in the first dictionary and the second dictionary can be encoded using the same encoding format as the character data.

[0231] Each dictionary data in the first dictionary and the second dictionary corresponds to a coding value in the coding subspace. Thus, the coding values ​​in the coding value group can be matched with the dictionary data in the first dictionary and the second dictionary in sequence to determine the dictionary data corresponding to each coding value.

[0232] According to an embodiment of the present invention, searching the encoding subspace corresponding to the first dictionary or the second dictionary and determining the character data in the encoding subspace corresponding to the dictionary data hit by the encoding value group can be specifically implemented as follows:

[0233] For any code value in the code value group, search the code subspace corresponding to the first dictionary or the second dictionary to determine whether the dictionary data is hit;

[0234] If so, determine the character data of the dictionary data in the corresponding encoding subspace;

[0235] If not, search the encoding subspace corresponding to the ASII character and determine the character data whose encoding value hits.

[0236] In a possible implementation of the present invention, in the process of searching the coding subspace corresponding to the first dictionary or the second dictionary and determining whether the dictionary data is hit, it can be implemented as first searching the first dictionary, and then searching the second dictionary if the dictionary data of the first dictionary is not hit.

[0237] According to an embodiment of the present invention, the data decompression method further includes:

[0238] Determine a merged value in the code value group; wherein the merged value is obtained by combining N consecutive code values ​​that meet the merging condition and merging the header identification data;

[0239] Split the combined value to obtain N encoded values.

[0240] According to an embodiment of the present invention, the N coded values ​​obtained by splitting the combined value can be specifically implemented as follows:

[0241] Determine the number of bits corresponding to each of the N coded values ​​based on the header identification data in the combined value;

[0242] According to the number of bits, N coded values ​​are obtained by splitting the combined value.

[0243] When merging N coding values, the N coding values ​​can be merged according to the merging method recorded in the merged coding table to generate a merged coding value. In addition, the header identification data corresponding to the hit bit position combination can be obtained from the merged coding table, and the header identification data and the coding value generated by the merger can be merged to obtain a merged value.

[0244] Therefore, the merged value can be split using the inverse process of the merge method recorded in the merge coding table.

[0245] The combined coding table can refer to Table 1 and Table 2 above, which will not be repeated here.

[0246] In one possible implementation, the data compression method and data decompression method provided by the embodiment of the present invention can be applied in a database scenario, and the data to be compressed can be a data table stored in the database, or a row of data or a column of data in the data table.

[0247] Data compression in a database can effectively reduce the storage space occupied by data, improve storage efficiency and transmission speed. Taking a column of data in a database table as an example, the original data can be compressed to obtain compressed data.

[0248] During the compression process, a first dictionary is first generated by statistically analyzing the data to be compressed. This dictionary records the characters and their frequency information that appear in the data to be compressed. Simultaneously, a second dictionary, derived from a text corpus, is used to map the characters in the data to encoding values ​​in the target encoding space. To ensure encoding uniqueness, at least one sub-encoding space is allocated from the target encoding space for each of the first and second dictionary and ASCII characters.

[0249] After mapping the data to be compressed to the code value groups, to further reduce the space occupied by the code value groups, a code merge operation can be performed on at least one set of code values. This can reduce the length of the compressed data without affecting data restoration, thereby reducing storage space and transmission bandwidth requirements.

[0250] The above compression method compresses a column of data in a table within a database, reducing storage space usage. This improves database performance and reduces storage costs. Furthermore, the compressed data can be decompressed and restored to its original format when needed, ensuring data integrity and availability.

[0251] For ease of understanding, Figure 4 The flowchart of the data compression / decompression process in a practical application of an embodiment of the present invention is schematically shown.

[0252] like Figure 4 As shown, the data compression method provided by the embodiment of the present invention can compress the original data, i.e., the data to be compressed, to obtain compressed data. After the compression process, the storage space occupied by the compressed data is smaller than the storage space occupied by the data to be compressed, thereby reducing the storage space or transmission bandwidth occupied by the data.

[0253] During the data compression process, character data in the data to be compressed is mapped to encoding value groups in a target encoding space using a first dictionary obtained through statistical analysis of the data to be compressed and a second dictionary obtained based on a text corpus. Furthermore, encoding space allocation is performed on the target encoding space to allocate at least one sub-encoding space to each of the first dictionary, the second dictionary, and ASCII characters.

[0254] After mapping the character data in the data to be compressed to the code value group, in order to further reduce the space occupied by the code value group, at least one set of code values ​​in the code value group can be encoded and merged, and serialized to obtain compressed data.

[0255] When the compressed data needs to be decompressed, the compressed data may be deserialized first. After the compressed data is deserialized, the merged code value group may be demerged first, so that the merged code value group is restored to multiple code values.

[0256] When demerging the merged coding value group, the merged coding table can be searched according to the number of bits corresponding to the N coding values ​​indicated by the header representation data, the merging method can be determined, and then the merging method can be inversely operated to achieve demerging of the merged coding values.

[0257] After decomposition and merging, a series of encoding values ​​can be obtained, and then the character data corresponding to each encoding value can be determined by searching the first dictionary or the second dictionary to achieve decompression of the compressed data. Figure 4 The specific implementation methods of the operations involved can be found in the corresponding embodiments described above and will not be repeated here.

[0258] Figure 5 A schematic structural diagram of an embodiment of a data compression device provided in an embodiment of the present invention may include:

[0259] The data determination module 501 is used to determine the data to be compressed that is character-encoded according to the Unicode encoding format;

[0260] The data reading module 502 is configured to sequentially read character data from the data to be compressed in byte order and determine a first dictionary or a second dictionary in which the character data matches; wherein the first dictionary is constructed based on the data to be compressed and the second dictionary is constructed based on a text corpus;

[0261] The first search module 503 is configured to search the code subspace corresponding to the first dictionary or the second dictionary, determine the code value of the dictionary data hit by the character data in the corresponding code subspace, and obtain a set of code value groups corresponding to the data to be compressed; the code subspaces corresponding to the first dictionary and the second dictionary are obtained by segmenting the target code space; the dictionary data in the first dictionary and the second dictionary are sorted from most important to least important, with the most important dictionary data corresponding to the smallest code value;

[0262] The first processing module 504 is configured to obtain compressed data based on the encoding value group and the first dictionary.

[0263] In some embodiments, the data compression apparatus 500 further includes:

[0264] A character determination module, configured to determine an ASCII character that the character data hits if the character data does not hit the first dictionary or the second dictionary;

[0265] A space search module is used to search the coding subspace corresponding to the ASCII character and determine the coding value of the dictionary data hit by the character data in the corresponding coding subspace; wherein the coding subspace corresponding to the ASCII character is obtained by segmenting the target coding space; and the coding value corresponding to at least part of the dictionary data in the first dictionary and the second dictionary is smaller than the coding value corresponding to the ASCII character.

[0266] In some embodiments, the data compression apparatus 500 further includes:

[0267] A segmentation module is used to segment the data to be compressed;

[0268] The first dictionary building module is used to build a first dictionary based on the multiple character strings obtained by segmentation.

[0269] In some embodiments, the first dictionary building module includes:

[0270] A first statistical submodule, configured to count the occurrence frequencies of multiple character strings in the data to be compressed;

[0271] A first calculation submodule is used to calculate the importance of multiple character strings according to the occurrence frequencies and character lengths corresponding to the multiple character strings;

[0272] The first sorting submodule is configured to select a first number of character strings in descending order of importance to generate a first dictionary.

[0273] In some embodiments, the data compression apparatus 500 further includes:

[0274] A first byte number determination submodule is configured to query the encoding subspace corresponding to the first dictionary in descending order of importance to determine first compressed byte numbers corresponding to the plurality of character strings;

[0275] A first compression ratio determination submodule, configured to determine first compression ratios corresponding to the plurality of character strings based on first compressed byte counts corresponding to the plurality of character strings;

[0276] A second byte number determination submodule, configured to query the encoding subspace corresponding to the second dictionary to determine second compressed byte numbers corresponding to the plurality of character strings;

[0277] A second compression ratio determination submodule, configured to determine second compression ratios corresponding to the plurality of character strings based on second compressed byte counts corresponding to the plurality of character strings;

[0278] The deletion submodule is used to delete, from the plurality of character strings, a character string whose first compression ratio is greater than a second compression ratio.

[0279] In some embodiments, the first computing submodule includes:

[0280] a first calculation unit, configured to calculate the importance of the plurality of character strings according to the occurrence frequencies, character lengths, and predetermined compressed byte counts corresponding to the plurality of character strings;

[0281] In some embodiments, the first sorting submodule includes:

[0282] a second calculation unit, configured to recalculate the importance of the plurality of character strings based on the first compressed byte counts respectively corresponding to the plurality of character strings, in combination with the occurrence frequencies and character lengths respectively corresponding to the plurality of character strings;

[0283] The character string selection unit is configured to select a first number of character strings in descending order of recalculated importance to generate a first dictionary.

[0284] In some embodiments, the segmentation module includes:

[0285] The character data reading submodule is used to read character data from the data to be compressed in byte order and save the character data as a string;

[0286] The number query submodule is used to query the number of occurrences of the current string;

[0287] The number of times counting submodule is used to accumulate the number of occurrences if the number of occurrences is greater than or equal to 1; if the number of occurrences is less than 1, merge the current string with the previously saved string and accumulate the number of occurrences of the merged string.

[0288] In some embodiments, the data compression apparatus 500 further includes:

[0289] The separation module is used to save at least one character data separated by the separator as a character string if the currently read character data is a separator.

[0290] In some embodiments, the data compression apparatus 500 further includes:

[0291] Statistics module, used to count the occurrence frequency of characters in the text corpus;

[0292] The sorting module is used to sort the characters in the text corpus in descending order of frequency of occurrence;

[0293] The second dictionary generating module is used to generate a second dictionary according to the sorting result.

[0294] In some embodiments, the first processing module includes:

[0295] A value determination module, used to determine N consecutive coded values ​​in the coded value group that meet the merging condition;

[0296] a merging module, configured to combine the N coded values ​​and merge the header identification data to obtain a merged value; wherein the header identification data is used to indicate the number of bits corresponding to the N coded values; and a merging condition is that the number of bytes occupied by the merged coded value is less than the sum of the number of bytes occupied by the N coded values;

[0297] The serialization module is used to serialize the merged value, the remaining coded values ​​in the coded value group that do not meet the merging condition, and the first dictionary to obtain compressed data.

[0298] In some embodiments, when serialization is performed using Varint serialization, the merging condition is specifically as follows: the number of bits of the N encoded values ​​is greater than or equal to 8, and the number of bits of the merged value is M times 7, where N is 2 and M is 3, or N is 3 and M is 4.

[0299] In some embodiments, the value determination module includes:

[0300] The first coding table query submodule is used to query the merged coding table corresponding to N coding values, and determine the consecutive N coding values ​​in the coding value group that hit any group of bit combinations in the merged coding table;

[0301] In some embodiments, the merge module includes:

[0302] The second coding table query submodule is used to search the combined coding table for the header identification data corresponding to the bit combination;

[0303] The merging submodule is used to combine N code values ​​and merge the header identification data according to the merging method in the merging code table to obtain a merged value.

[0304] In some embodiments, the data compression apparatus 500 further includes:

[0305] A partitioning module, used to divide the target coding space into multiple coding subspaces;

[0306] A segmentation module, configured to determine a plurality of data segments obtained by segmenting the first dictionary and the second dictionary respectively;

[0307] The allocation module is used to allocate coding subspaces to the multiple data segments respectively from the multiple coding subspaces according to the allocation principle that the coding value corresponding to the dictionary data with greater importance is smaller.

[0308] Figure 5 The data compression device can perform Figure 1 The implementation principle and technical effects of the data compression method of the embodiment shown are not described in detail. The specific manner in which each module and unit performs operations in the data compression device in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0309] Figure 6 A schematic structural diagram of an embodiment of a data decompression device provided in an embodiment of the present invention, the device may include:

[0310] Data acquisition module 601, used to acquire compressed data;

[0311] The second processing module 602 is configured to obtain a first dictionary and a set of encoding values ​​from the compressed data; wherein the set of encoding values ​​is constructed by searching the encoding subspace corresponding to the first dictionary or the second dictionary, and according to the encoding values ​​of the dictionary data hit by the character data in the compressed data in the corresponding encoding subspace; wherein the first dictionary is constructed based on the compressed data; and the second dictionary is constructed based on the text corpus.

[0312] The second search module 603 is used to search the encoding subspace corresponding to the first dictionary or the second dictionary, determine the character data in the corresponding encoding subspace of the dictionary data hit by the encoding value group, and obtain the decompressed data corresponding to the encoding value group.

[0313] In some embodiments, the second search module 603 includes:

[0314] A first determination submodule is configured to search the encoding subspace corresponding to the first dictionary or the second dictionary for any encoding value in the encoding value group to determine whether the dictionary data is matched;

[0315] The second determining submodule is used to determine the character data of the dictionary data in the corresponding encoding subspace when the dictionary data is hit;

[0316] The third determining submodule is used to search the encoding subspace corresponding to the ASII character when no dictionary data is matched, and determine the character data whose encoding value matches.

[0317] In an embodiment of the present invention, the data decompression device 600 further includes:

[0318] A first determining module is configured to determine a combined value in the code value group; wherein the combined value is obtained by combining N consecutive code values ​​that meet a combining condition and combining the combined value with the header identification data;

[0319] The splitting module is used to split the merged value to obtain N encoded values.

[0320] In some embodiments, the split module includes:

[0321] A bit determination submodule is used to determine the number of bits corresponding to the N encoded values ​​according to the header identification data in the combined value;

[0322] The splitting submodule is used to split the combined value to obtain N coded values ​​according to the number of bits.

[0323] Figure 6 The data decompression device can execute Figure 3 The implementation principle and technical effects of the data decompression method described in the embodiment are not described in detail. The specific manner in which each module and unit performs operations in the data decompression device in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0324] In one possible design, the data compression device and the data decompression device provided by the embodiment of the present invention can be implemented as a computing device, such as Figure 7 As shown, the computing device may include a storage component 701 and a processing component 702;

[0325] The storage component 701 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 702 to implement the data compression method and the data decompression method provided by the embodiment of the present invention.

[0326] Of course, a computing device may also include other components, such as input / output interfaces and communication components. The input / output interface provides an interface between the processing component and peripheral interface modules, such as output devices and input devices. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0327] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0328] When the computing device is a physical device, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device.

[0329] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a computer, the data compression method and the data decompression method provided in the embodiment of the present invention can be implemented.

[0330] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by a computer, the data compression method and the data decompression method provided by the embodiment of the present invention can be implemented.

[0331] The processing components in the above embodiments may include one or more processors to execute computer instructions to complete all or part of the steps in the above methods. Of course, the processing components may also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0332] The memory component is configured to store various types of data to support operations in the device. The memory component can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0333] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0334] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0335] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0336] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data compression method, characterized in that: include: Determining data to be compressed that is character-encoded according to a Unicode encoding format; Reading character data from the data to be compressed in byte order and determining a first dictionary or a second dictionary in which the character data matches; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus; Searching the encoding subspace corresponding to the first dictionary or the second dictionary, respectively, and determining the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace, so as to obtain a set of encoding value groups corresponding to the data to be compressed; Compressed data is obtained based on the encoding value group and the first dictionary.

2. The method according to claim 1, characterized in that The method further comprises: If the character data does not match the first dictionary or the second dictionary, determining the ASCII character that the character data matches; Search the encoding subspace corresponding to the ASCII character and determine the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace; wherein the encoding subspace corresponding to the ASCII character is obtained by segmenting the target encoding space; and the encoding value corresponding to at least part of the dictionary data in the first dictionary and the second dictionary is smaller than the encoding value corresponding to the ASCII character.

3. The method according to claim 1, characterized in that The first dictionary is generated as follows: Segmenting the data to be compressed; The first dictionary is constructed based on the multiple character strings obtained by segmentation.

4. The method according to claim 3, characterized in that The constructing the first dictionary based on the multiple character strings obtained by segmentation includes: Counting the occurrence frequencies of the multiple character strings in the data to be compressed; Calculating the importance of the multiple character strings according to the occurrence frequencies and character lengths respectively corresponding to the multiple character strings; A first number of character strings are selected in descending order of importance to generate the first dictionary; wherein the dictionary data in the first dictionary are sorted in descending order of importance, and the code value corresponding to the dictionary data with greater importance is smaller.

5. The method according to claim 4, characterized in that The method further comprises: Searching the encoding subspace corresponding to the first dictionary in descending order of importance to determine first compressed byte numbers corresponding to the multiple character strings; Determining first compression ratios corresponding to the plurality of character strings based on first compressed byte numbers corresponding to the plurality of character strings; querying the encoding subspace corresponding to the second dictionary to determine the second compressed byte numbers corresponding to the multiple character strings respectively; determining second compression ratios corresponding to the plurality of character strings respectively based on second compressed byte numbers corresponding to the plurality of character strings respectively; Character strings having a first compression ratio greater than a second compression ratio are deleted from the plurality of character strings.

6. The method according to claim 4, characterized in that Calculating the importance of the plurality of character strings according to the occurrence frequencies and character lengths respectively corresponding to the plurality of character strings includes: Calculating the importance of the multiple character strings according to the occurrence frequencies, character lengths, and predetermined compressed byte counts corresponding to the multiple character strings; The selecting a first number of character strings in descending order of importance to generate the first dictionary includes: recalculating the importance of the multiple character strings according to the first compressed byte numbers respectively corresponding to the multiple character strings, combined with the occurrence frequencies and character lengths respectively corresponding to the multiple character strings; A first number of character strings are selected in descending order of recalculated importance to generate the first dictionary.

7. The method according to claim 3, characterized in that The segmenting of the data to be compressed includes: Reading character data from the data to be compressed in byte order, and saving the character data as a character string; Query the number of occurrences of the current string; If the number of occurrences is greater than or equal to 1, the number of occurrences is accumulated; if the number of occurrences is less than 1, the current string is merged with the previously saved string, and the number of occurrences of the merged string is accumulated.

8. The method according to claim 7, characterized in that The method further comprises: If the currently read character data is a separator, at least one character data separated by the separator is saved as a character string.

9. The method according to claim 1, characterized in that The second dictionary is generated as follows: Counting the occurrence frequencies of characters in the text corpus; Sorting the characters in the text corpus in descending order of frequency of occurrence; The second dictionary is generated according to the sorting result; wherein the dictionary data in the second dictionary are sorted from large to small according to importance, and the dictionary data with large importance corresponds to small coding values.

10. The method according to claim 1, characterized in that The obtaining of compressed data based on the encoding value group and the first dictionary includes: Determining N consecutive code values ​​in the code value group that meet a merging condition; Combining the N code values ​​and merging them with header identification data to obtain a combined value; wherein the header identification data is used to indicate the number of bits corresponding to the N code values; and the merging condition is that the number of bytes occupied by the combined code value is less than the sum of the number of bytes occupied by the N code values; The merged value, the remaining coded values ​​in the coded value group that do not meet the merging condition, and the first dictionary are serialized to obtain compressed data.

11. The method according to claim 10, characterized in that When serialization is performed using the variable-length integer encoding Varint serialization method, the merging conditions are specifically as follows: The number of bits of the N encoded values ​​is respectively greater than or equal to 8, and the number of bits of the combined value is M times 7, where N is 2 and M is 3, or N is 3 and M is 4.

12. The method according to claim 10, characterized in that Determining the N consecutive code values ​​in the code value group that meet the merging condition includes: Querying a combined coding table corresponding to N coding values, and determining N consecutive coding values ​​in the coding value group that hit any group of bit combinations in the combined coding table; The combining of the N encoded values ​​and merging the header identification data to obtain a combined value includes: Searching the combined coding table for header identification data corresponding to the bit combination; According to the merging method in the merged coding table, the N coding values ​​are combined and the header identification data is merged to obtain a merged value.

13. The method according to claim 1, wherein The coding subspace is generated as follows: Divide the target encoding space into multiple encoding subspaces; Determine a plurality of data segments obtained by segmenting the first dictionary and the second dictionary respectively; According to the allocation principle that dictionary data with greater importance corresponds to smaller coding values, coding subspaces are respectively allocated to the multiple data segments from the multiple coding subspaces.

14. A data decompression method, characterized in that: include: Get compressed data; A first dictionary and a set of encoding values ​​are obtained from the compressed data; wherein the set of encoding values ​​is constructed by searching the encoding subspace corresponding to the first dictionary or the second dictionary, and according to the encoding values ​​of the dictionary data hit by the character data in the compressed data in the corresponding encoding subspace; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus; The encoding subspace corresponding to the first dictionary or the second dictionary is searched, and the character data in the corresponding encoding subspace of the dictionary data hit by the encoding value group is determined to obtain the decompressed data corresponding to the encoding value group.

15. The method according to claim 14, characterized in that The searching of the encoding subspaces corresponding to the first dictionary or the second dictionary, and determining the character data in the corresponding encoding subspace of the dictionary data hit by the encoding value group includes: For any code value in the code value group, searching the code subspace corresponding to the first dictionary or the second dictionary to determine whether dictionary data is hit; If so, determining the character data of the dictionary data in the corresponding encoding subspace; If not, search the encoding subspace corresponding to the ASII character to determine the character data that the encoding value hits.

16. The method according to claim 14, characterized in that The obtaining of a first dictionary and a set of encoded values ​​from the compressed data includes: Deserialize the compressed data to obtain a first dictionary and an encoded value; The method further comprises: Determining a combined value in the encoding value group; wherein the combined value is obtained by combining N consecutive encoding values ​​that meet a combining condition and combining the header identification data; The N coded values ​​are obtained by splitting the combined value.

17. The method according to claim 16, characterized in that The step of splitting the combined value to obtain the N coded values ​​includes: Determining the number of bits corresponding to each of the N coded values ​​according to the header identification data in the combined value; The N coded values ​​are obtained by splitting the combined value according to the number of bits.

18. A computing device, characterized in that including processing components and storage components; The storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the data compression method as described in any one of claims 1 to 13, or to implement the data decompression method as described in any one of claims 14 to 17.

19. A computer storage medium, characterized in that A computer program is stored, and when the computer program is executed by a computer, the data compression method according to any one of claims 1 to 13 is implemented, or the data decompression method according to any one of claims 14 to 17 is implemented.

20. A computer program product, characterized in that The computer program product includes computer program code, and when the computer program code is executed by a computer, it implements the data compression method according to any one of claims 1 to 13, or implements the data decompression method according to any one of claims 14 to 17.

Images